* [PATCH v7 00/11] complete deferred page initialization
@ 2017-08-29 2:02 ` Pavel Tatashin
0 siblings, 0 replies; 79+ messages in thread
From: Pavel Tatashin @ 2017-08-29 2:02 UTC (permalink / raw)
To: linux-kernel, sparclinux, linux-mm, linuxppc-dev, linux-s390,
linux-arm-kernel, x86, kasan-dev, borntraeger, heiko.carstens,
davem, willy, mhocko, ard.biesheuvel, will.deacon,
catalin.marinas, sam, mgorman, Steven.Sistare, daniel.m.jordan,
bob.picco
Changelog:
v7 - v6
- Addressed comments from Michal Hocko
- memblock_discard() patch was removed from this series and integrated
separately
- Fixed bug reported by kbuild test robot new patch:
mm: zero reserved and unavailable struct pages
- Removed patch
x86/mm: reserve only exiting low pages
As, it is not needed anymore, because of the previous fix
- Re-wrote deferred_init_memmap(), found and fixed an existing bug, where
page variable is not reset when zone holes present.
- Merged several patches together per Michal request
- Added performance data including raw logs
v6 - v5
- Fixed ARM64 + kasan code, as reported by Ard Biesheuvel
- Tested ARM64 code in qemu and found few more issues, that I fixed in this
iteration
- Added page roundup/rounddown to x86 and arm zeroing routines to zero the
whole allocated range, instead of only provided address range.
- Addressed SPARC related comment from Sam Ravnborg
- Fixed section mismatch warnings related to memblock_discard().
v5 - v4
- Fixed build issues reported by kbuild on various configurations
v4 - v3
- Rewrote code to zero sturct pages in __init_single_page() as
suggested by Michal Hocko
- Added code to handle issues related to accessing struct page
memory before they are initialized.
v3 - v2
- Addressed David Miller comments about one change per patch:
* Splited changes to platforms into 4 patches
* Made "do not zero vmemmap_buf" as a separate patch
v2 - v1
- Per request, added s390 to deferred "struct page" zeroing
- Collected performance data on x86 which proofs the importance to
keep memset() as prefetch (see below).
SMP machines can benefit from the DEFERRED_STRUCT_PAGE_INIT config option,
which defers initializing struct pages until all cpus have been started so
it can be done in parallel.
However, this feature is sub-optimal, because the deferred page
initialization code expects that the struct pages have already been zeroed,
and the zeroing is done early in boot with a single thread only. Also, we
access that memory and set flags before struct pages are initialized. All
of this is fixed in this patchset.
In this work we do the following:
- Never read access struct page until it was initialized
- Never set any fields in struct pages before they are initialized
- Zero struct page at the beginning of struct page initialization
==========================================================================
Performance improvements on x86 machine with 8 nodes:
Intel(R) Xeon(R) CPU E7-8895 v3 @ 2.60GHz and 1T of memory:
TIME SPEED UP
base no deferred: 95.796233s
fix no deferred: 79.978956s 19.77%
base deferred: 77.254713s
fix deferred: 55.050509s 40.34%
==========================================================================
SPARC M6 3600 MHz with 15T of memory
TIME SPEED UP
base no deferred: 358.335727s
fix no deferred: 302.320936s 18.52%
base deferred: 237.534603s
fix deferred: 182.103003s 30.44%
==========================================================================
Raw dmesg output with timestamps:
x86 base no deferred: https://hastebin.com/ofunepurit.scala
x86 base deferred: https://hastebin.com/ifazegeyas.scala
x86 fix no deferred: https://hastebin.com/pegocohevo.scala
x86 fix deferred: https://hastebin.com/ofupevikuk.scala
sparc base no deferred: https://hastebin.com/ibobeteken.go
sparc base deferred: https://hastebin.com/fariqimiyu.go
sparc fix no deferred: https://hastebin.com/muhegoheyi.go
sparc fix deferred: https://hastebin.com/xadinobutu.go
Pavel Tatashin (11):
x86/mm: setting fields in deferred pages
sparc64/mm: setting fields in deferred pages
mm: deferred_init_memmap improvements
sparc64: simplify vmemmap_populate
mm: defining memblock_virt_alloc_try_nid_raw
mm: zero struct pages during initialization
sparc64: optimized struct page zeroing
mm: zero reserved and unavailable struct pages
x86/kasan: explicitly zero kasan shadow memory
arm64/kasan: explicitly zero kasan shadow memory
mm: stop zeroing memory during allocation in vmemmap
arch/arm64/mm/kasan_init.c | 42 ++++++++
arch/sparc/include/asm/pgtable_64.h | 30 ++++++
arch/sparc/mm/init_64.c | 31 +++---
arch/x86/mm/init_64.c | 9 +-
arch/x86/mm/kasan_init_64.c | 66 ++++++++++++
include/linux/bootmem.h | 27 +++++
include/linux/memblock.h | 16 +++
include/linux/mm.h | 26 +++++
mm/memblock.c | 60 +++++++++--
mm/page_alloc.c | 207 ++++++++++++++++++++----------------
mm/sparse-vmemmap.c | 14 +--
mm/sparse.c | 6 +-
12 files changed, 406 insertions(+), 128 deletions(-)
--
2.14.1
^ permalink raw reply [flat|nested] 79+ messages in thread
* [PATCH v7 00/11] complete deferred page initialization
@ 2017-08-29 2:02 ` Pavel Tatashin
0 siblings, 0 replies; 79+ messages in thread
From: Pavel Tatashin @ 2017-08-29 2:02 UTC (permalink / raw)
To: linux-arm-kernel
Changelog:
v7 - v6
- Addressed comments from Michal Hocko
- memblock_discard() patch was removed from this series and integrated
separately
- Fixed bug reported by kbuild test robot new patch:
mm: zero reserved and unavailable struct pages
- Removed patch
x86/mm: reserve only exiting low pages
As, it is not needed anymore, because of the previous fix
- Re-wrote deferred_init_memmap(), found and fixed an existing bug, where
page variable is not reset when zone holes present.
- Merged several patches together per Michal request
- Added performance data including raw logs
v6 - v5
- Fixed ARM64 + kasan code, as reported by Ard Biesheuvel
- Tested ARM64 code in qemu and found few more issues, that I fixed in this
iteration
- Added page roundup/rounddown to x86 and arm zeroing routines to zero the
whole allocated range, instead of only provided address range.
- Addressed SPARC related comment from Sam Ravnborg
- Fixed section mismatch warnings related to memblock_discard().
v5 - v4
- Fixed build issues reported by kbuild on various configurations
v4 - v3
- Rewrote code to zero sturct pages in __init_single_page() as
suggested by Michal Hocko
- Added code to handle issues related to accessing struct page
memory before they are initialized.
v3 - v2
- Addressed David Miller comments about one change per patch:
* Splited changes to platforms into 4 patches
* Made "do not zero vmemmap_buf" as a separate patch
v2 - v1
- Per request, added s390 to deferred "struct page" zeroing
- Collected performance data on x86 which proofs the importance to
keep memset() as prefetch (see below).
SMP machines can benefit from the DEFERRED_STRUCT_PAGE_INIT config option,
which defers initializing struct pages until all cpus have been started so
it can be done in parallel.
However, this feature is sub-optimal, because the deferred page
initialization code expects that the struct pages have already been zeroed,
and the zeroing is done early in boot with a single thread only. Also, we
access that memory and set flags before struct pages are initialized. All
of this is fixed in this patchset.
In this work we do the following:
- Never read access struct page until it was initialized
- Never set any fields in struct pages before they are initialized
- Zero struct page at the beginning of struct page initialization
=====================================
Performance improvements on x86 machine with 8 nodes:
Intel(R) Xeon(R) CPU E7-8895 v3 @ 2.60GHz and 1T of memory:
TIME SPEED UP
base no deferred: 95.796233s
fix no deferred: 79.978956s 19.77%
base deferred: 77.254713s
fix deferred: 55.050509s 40.34%
=====================================
SPARC M6 3600 MHz with 15T of memory
TIME SPEED UP
base no deferred: 358.335727s
fix no deferred: 302.320936s 18.52%
base deferred: 237.534603s
fix deferred: 182.103003s 30.44%
=====================================
Raw dmesg output with timestamps:
x86 base no deferred: https://hastebin.com/ofunepurit.scala
x86 base deferred: https://hastebin.com/ifazegeyas.scala
x86 fix no deferred: https://hastebin.com/pegocohevo.scala
x86 fix deferred: https://hastebin.com/ofupevikuk.scala
sparc base no deferred: https://hastebin.com/ibobeteken.go
sparc base deferred: https://hastebin.com/fariqimiyu.go
sparc fix no deferred: https://hastebin.com/muhegoheyi.go
sparc fix deferred: https://hastebin.com/xadinobutu.go
Pavel Tatashin (11):
x86/mm: setting fields in deferred pages
sparc64/mm: setting fields in deferred pages
mm: deferred_init_memmap improvements
sparc64: simplify vmemmap_populate
mm: defining memblock_virt_alloc_try_nid_raw
mm: zero struct pages during initialization
sparc64: optimized struct page zeroing
mm: zero reserved and unavailable struct pages
x86/kasan: explicitly zero kasan shadow memory
arm64/kasan: explicitly zero kasan shadow memory
mm: stop zeroing memory during allocation in vmemmap
arch/arm64/mm/kasan_init.c | 42 ++++++++
arch/sparc/include/asm/pgtable_64.h | 30 ++++++
arch/sparc/mm/init_64.c | 31 +++---
arch/x86/mm/init_64.c | 9 +-
arch/x86/mm/kasan_init_64.c | 66 ++++++++++++
include/linux/bootmem.h | 27 +++++
include/linux/memblock.h | 16 +++
include/linux/mm.h | 26 +++++
mm/memblock.c | 60 +++++++++--
mm/page_alloc.c | 207 ++++++++++++++++++++----------------
mm/sparse-vmemmap.c | 14 +--
mm/sparse.c | 6 +-
12 files changed, 406 insertions(+), 128 deletions(-)
--
2.14.1
^ permalink raw reply [flat|nested] 79+ messages in thread
* [PATCH v7 00/11] complete deferred page initialization
@ 2017-08-29 2:02 ` Pavel Tatashin
0 siblings, 0 replies; 79+ messages in thread
From: Pavel Tatashin @ 2017-08-29 2:02 UTC (permalink / raw)
To: linux-kernel, sparclinux, linux-mm, linuxppc-dev, linux-s390,
linux-arm-kernel, x86, kasan-dev, borntraeger, heiko.carstens,
davem, willy, mhocko, ard.biesheuvel, will.deacon,
catalin.marinas, sam, mgorman, Steven.Sistare, daniel.m.jordan,
bob.picco
Changelog:
v7 - v6
- Addressed comments from Michal Hocko
- memblock_discard() patch was removed from this series and integrated
separately
- Fixed bug reported by kbuild test robot new patch:
mm: zero reserved and unavailable struct pages
- Removed patch
x86/mm: reserve only exiting low pages
As, it is not needed anymore, because of the previous fix
- Re-wrote deferred_init_memmap(), found and fixed an existing bug, where
page variable is not reset when zone holes present.
- Merged several patches together per Michal request
- Added performance data including raw logs
v6 - v5
- Fixed ARM64 + kasan code, as reported by Ard Biesheuvel
- Tested ARM64 code in qemu and found few more issues, that I fixed in this
iteration
- Added page roundup/rounddown to x86 and arm zeroing routines to zero the
whole allocated range, instead of only provided address range.
- Addressed SPARC related comment from Sam Ravnborg
- Fixed section mismatch warnings related to memblock_discard().
v5 - v4
- Fixed build issues reported by kbuild on various configurations
v4 - v3
- Rewrote code to zero sturct pages in __init_single_page() as
suggested by Michal Hocko
- Added code to handle issues related to accessing struct page
memory before they are initialized.
v3 - v2
- Addressed David Miller comments about one change per patch:
* Splited changes to platforms into 4 patches
* Made "do not zero vmemmap_buf" as a separate patch
v2 - v1
- Per request, added s390 to deferred "struct page" zeroing
- Collected performance data on x86 which proofs the importance to
keep memset() as prefetch (see below).
SMP machines can benefit from the DEFERRED_STRUCT_PAGE_INIT config option,
which defers initializing struct pages until all cpus have been started so
it can be done in parallel.
However, this feature is sub-optimal, because the deferred page
initialization code expects that the struct pages have already been zeroed,
and the zeroing is done early in boot with a single thread only. Also, we
access that memory and set flags before struct pages are initialized. All
of this is fixed in this patchset.
In this work we do the following:
- Never read access struct page until it was initialized
- Never set any fields in struct pages before they are initialized
- Zero struct page at the beginning of struct page initialization
==========================================================================
Performance improvements on x86 machine with 8 nodes:
Intel(R) Xeon(R) CPU E7-8895 v3 @ 2.60GHz and 1T of memory:
TIME SPEED UP
base no deferred: 95.796233s
fix no deferred: 79.978956s 19.77%
base deferred: 77.254713s
fix deferred: 55.050509s 40.34%
==========================================================================
SPARC M6 3600 MHz with 15T of memory
TIME SPEED UP
base no deferred: 358.335727s
fix no deferred: 302.320936s 18.52%
base deferred: 237.534603s
fix deferred: 182.103003s 30.44%
==========================================================================
Raw dmesg output with timestamps:
x86 base no deferred: https://hastebin.com/ofunepurit.scala
x86 base deferred: https://hastebin.com/ifazegeyas.scala
x86 fix no deferred: https://hastebin.com/pegocohevo.scala
x86 fix deferred: https://hastebin.com/ofupevikuk.scala
sparc base no deferred: https://hastebin.com/ibobeteken.go
sparc base deferred: https://hastebin.com/fariqimiyu.go
sparc fix no deferred: https://hastebin.com/muhegoheyi.go
sparc fix deferred: https://hastebin.com/xadinobutu.go
Pavel Tatashin (11):
x86/mm: setting fields in deferred pages
sparc64/mm: setting fields in deferred pages
mm: deferred_init_memmap improvements
sparc64: simplify vmemmap_populate
mm: defining memblock_virt_alloc_try_nid_raw
mm: zero struct pages during initialization
sparc64: optimized struct page zeroing
mm: zero reserved and unavailable struct pages
x86/kasan: explicitly zero kasan shadow memory
arm64/kasan: explicitly zero kasan shadow memory
mm: stop zeroing memory during allocation in vmemmap
arch/arm64/mm/kasan_init.c | 42 ++++++++
arch/sparc/include/asm/pgtable_64.h | 30 ++++++
arch/sparc/mm/init_64.c | 31 +++---
arch/x86/mm/init_64.c | 9 +-
arch/x86/mm/kasan_init_64.c | 66 ++++++++++++
include/linux/bootmem.h | 27 +++++
include/linux/memblock.h | 16 +++
include/linux/mm.h | 26 +++++
mm/memblock.c | 60 +++++++++--
mm/page_alloc.c | 207 ++++++++++++++++++++----------------
mm/sparse-vmemmap.c | 14 +--
mm/sparse.c | 6 +-
12 files changed, 406 insertions(+), 128 deletions(-)
--
2.14.1
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 79+ messages in thread
* [PATCH v7 00/11] complete deferred page initialization
@ 2017-08-29 2:02 ` Pavel Tatashin
0 siblings, 0 replies; 79+ messages in thread
From: Pavel Tatashin @ 2017-08-29 2:02 UTC (permalink / raw)
To: linux-arm-kernel
Changelog:
v7 - v6
- Addressed comments from Michal Hocko
- memblock_discard() patch was removed from this series and integrated
separately
- Fixed bug reported by kbuild test robot new patch:
mm: zero reserved and unavailable struct pages
- Removed patch
x86/mm: reserve only exiting low pages
As, it is not needed anymore, because of the previous fix
- Re-wrote deferred_init_memmap(), found and fixed an existing bug, where
page variable is not reset when zone holes present.
- Merged several patches together per Michal request
- Added performance data including raw logs
v6 - v5
- Fixed ARM64 + kasan code, as reported by Ard Biesheuvel
- Tested ARM64 code in qemu and found few more issues, that I fixed in this
iteration
- Added page roundup/rounddown to x86 and arm zeroing routines to zero the
whole allocated range, instead of only provided address range.
- Addressed SPARC related comment from Sam Ravnborg
- Fixed section mismatch warnings related to memblock_discard().
v5 - v4
- Fixed build issues reported by kbuild on various configurations
v4 - v3
- Rewrote code to zero sturct pages in __init_single_page() as
suggested by Michal Hocko
- Added code to handle issues related to accessing struct page
memory before they are initialized.
v3 - v2
- Addressed David Miller comments about one change per patch:
* Splited changes to platforms into 4 patches
* Made "do not zero vmemmap_buf" as a separate patch
v2 - v1
- Per request, added s390 to deferred "struct page" zeroing
- Collected performance data on x86 which proofs the importance to
keep memset() as prefetch (see below).
SMP machines can benefit from the DEFERRED_STRUCT_PAGE_INIT config option,
which defers initializing struct pages until all cpus have been started so
it can be done in parallel.
However, this feature is sub-optimal, because the deferred page
initialization code expects that the struct pages have already been zeroed,
and the zeroing is done early in boot with a single thread only. Also, we
access that memory and set flags before struct pages are initialized. All
of this is fixed in this patchset.
In this work we do the following:
- Never read access struct page until it was initialized
- Never set any fields in struct pages before they are initialized
- Zero struct page at the beginning of struct page initialization
==========================================================================
Performance improvements on x86 machine with 8 nodes:
Intel(R) Xeon(R) CPU E7-8895 v3 @ 2.60GHz and 1T of memory:
TIME SPEED UP
base no deferred: 95.796233s
fix no deferred: 79.978956s 19.77%
base deferred: 77.254713s
fix deferred: 55.050509s 40.34%
==========================================================================
SPARC M6 3600 MHz with 15T of memory
TIME SPEED UP
base no deferred: 358.335727s
fix no deferred: 302.320936s 18.52%
base deferred: 237.534603s
fix deferred: 182.103003s 30.44%
==========================================================================
Raw dmesg output with timestamps:
x86 base no deferred: https://hastebin.com/ofunepurit.scala
x86 base deferred: https://hastebin.com/ifazegeyas.scala
x86 fix no deferred: https://hastebin.com/pegocohevo.scala
x86 fix deferred: https://hastebin.com/ofupevikuk.scala
sparc base no deferred: https://hastebin.com/ibobeteken.go
sparc base deferred: https://hastebin.com/fariqimiyu.go
sparc fix no deferred: https://hastebin.com/muhegoheyi.go
sparc fix deferred: https://hastebin.com/xadinobutu.go
Pavel Tatashin (11):
x86/mm: setting fields in deferred pages
sparc64/mm: setting fields in deferred pages
mm: deferred_init_memmap improvements
sparc64: simplify vmemmap_populate
mm: defining memblock_virt_alloc_try_nid_raw
mm: zero struct pages during initialization
sparc64: optimized struct page zeroing
mm: zero reserved and unavailable struct pages
x86/kasan: explicitly zero kasan shadow memory
arm64/kasan: explicitly zero kasan shadow memory
mm: stop zeroing memory during allocation in vmemmap
arch/arm64/mm/kasan_init.c | 42 ++++++++
arch/sparc/include/asm/pgtable_64.h | 30 ++++++
arch/sparc/mm/init_64.c | 31 +++---
arch/x86/mm/init_64.c | 9 +-
arch/x86/mm/kasan_init_64.c | 66 ++++++++++++
include/linux/bootmem.h | 27 +++++
include/linux/memblock.h | 16 +++
include/linux/mm.h | 26 +++++
mm/memblock.c | 60 +++++++++--
mm/page_alloc.c | 207 ++++++++++++++++++++----------------
mm/sparse-vmemmap.c | 14 +--
mm/sparse.c | 6 +-
12 files changed, 406 insertions(+), 128 deletions(-)
--
2.14.1
^ permalink raw reply [flat|nested] 79+ messages in thread
* [PATCH v7 01/11] x86/mm: setting fields in deferred pages
2017-08-29 2:02 ` Pavel Tatashin
(?)
(?)
@ 2017-08-29 2:02 ` Pavel Tatashin
-1 siblings, 0 replies; 79+ messages in thread
From: Pavel Tatashin @ 2017-08-29 2:02 UTC (permalink / raw)
To: linux-kernel, sparclinux, linux-mm, linuxppc-dev, linux-s390,
linux-arm-kernel, x86, kasan-dev, borntraeger, heiko.carstens,
davem, willy, mhocko, ard.biesheuvel, will.deacon,
catalin.marinas, sam, mgorman, Steven.Sistare, daniel.m.jordan,
bob.picco
Without deferred struct page feature (CONFIG_DEFERRED_STRUCT_PAGE_INIT),
flags and other fields in "struct page"es are never changed prior to first
initializing struct pages by going through __init_single_page().
With deferred struct page feature enabled, however, we set fields in
register_page_bootmem_info that are subsequently clobbered right after in
free_all_bootmem:
mem_init() {
register_page_bootmem_info();
free_all_bootmem();
...
}
When register_page_bootmem_info() is called only non-deferred struct pages
are initialized. But, this function goes through some reserved pages which
might be part of the deferred, and thus are not yet initialized.
mem_init
register_page_bootmem_info
register_page_bootmem_info_node
get_page_bootmem
.. setting fields here ..
such as: page->freelist = (void *)type;
free_all_bootmem()
free_low_memory_core_early()
for_each_reserved_mem_region()
reserve_bootmem_region()
init_reserved_page() <- Only if this is deferred reserved page
__init_single_pfn()
__init_single_page()
memset(0) <-- Loose the set fields here
We end-up with issue where, currently we do not observe problem as memory
is explicitly zeroed. But, if flag asserts are changed we can start hitting
issues.
Also, because in this patch series we will stop zeroing struct page memory
during allocation, we must make sure that struct pages are properly
initialized prior to using them.
The deferred-reserved pages are initialized in free_all_bootmem().
Therefore, the fix is to switch the above calls.
Signed-off-by: Pavel Tatashin <pasha.tatashin@oracle.com>
Reviewed-by: Steven Sistare <steven.sistare@oracle.com>
Reviewed-by: Daniel Jordan <daniel.m.jordan@oracle.com>
Reviewed-by: Bob Picco <bob.picco@oracle.com>
---
arch/x86/mm/init_64.c | 9 +++++++--
1 file changed, 7 insertions(+), 2 deletions(-)
diff --git a/arch/x86/mm/init_64.c b/arch/x86/mm/init_64.c
index 62a91e6b1237..3a997352a992 100644
--- a/arch/x86/mm/init_64.c
+++ b/arch/x86/mm/init_64.c
@@ -1174,12 +1174,17 @@ void __init mem_init(void)
/* clear_bss() already clear the empty_zero_page */
- register_page_bootmem_info();
-
/* this will put all memory onto the freelists */
free_all_bootmem();
after_bootmem = 1;
+ /* Must be done after boot memory is put on freelist, because here we
+ * might set fields in deferred struct pages that have not yet been
+ * initialized, and free_all_bootmem() initializes all the reserved
+ * deferred pages for us.
+ */
+ register_page_bootmem_info();
+
/* Register memory areas for /proc/kcore */
kclist_add(&kcore_vsyscall, (void *)VSYSCALL_ADDR,
PAGE_SIZE, KCORE_OTHER);
--
2.14.1
^ permalink raw reply related [flat|nested] 79+ messages in thread
* [PATCH v7 01/11] x86/mm: setting fields in deferred pages
@ 2017-08-29 2:02 ` Pavel Tatashin
0 siblings, 0 replies; 79+ messages in thread
From: Pavel Tatashin @ 2017-08-29 2:02 UTC (permalink / raw)
To: linux-arm-kernel
Without deferred struct page feature (CONFIG_DEFERRED_STRUCT_PAGE_INIT),
flags and other fields in "struct page"es are never changed prior to first
initializing struct pages by going through __init_single_page().
With deferred struct page feature enabled, however, we set fields in
register_page_bootmem_info that are subsequently clobbered right after in
free_all_bootmem:
mem_init() {
register_page_bootmem_info();
free_all_bootmem();
...
}
When register_page_bootmem_info() is called only non-deferred struct pages
are initialized. But, this function goes through some reserved pages which
might be part of the deferred, and thus are not yet initialized.
mem_init
register_page_bootmem_info
register_page_bootmem_info_node
get_page_bootmem
.. setting fields here ..
such as: page->freelist = (void *)type;
free_all_bootmem()
free_low_memory_core_early()
for_each_reserved_mem_region()
reserve_bootmem_region()
init_reserved_page() <- Only if this is deferred reserved page
__init_single_pfn()
__init_single_page()
memset(0) <-- Loose the set fields here
We end-up with issue where, currently we do not observe problem as memory
is explicitly zeroed. But, if flag asserts are changed we can start hitting
issues.
Also, because in this patch series we will stop zeroing struct page memory
during allocation, we must make sure that struct pages are properly
initialized prior to using them.
The deferred-reserved pages are initialized in free_all_bootmem().
Therefore, the fix is to switch the above calls.
Signed-off-by: Pavel Tatashin <pasha.tatashin@oracle.com>
Reviewed-by: Steven Sistare <steven.sistare@oracle.com>
Reviewed-by: Daniel Jordan <daniel.m.jordan@oracle.com>
Reviewed-by: Bob Picco <bob.picco@oracle.com>
---
arch/x86/mm/init_64.c | 9 +++++++--
1 file changed, 7 insertions(+), 2 deletions(-)
diff --git a/arch/x86/mm/init_64.c b/arch/x86/mm/init_64.c
index 62a91e6b1237..3a997352a992 100644
--- a/arch/x86/mm/init_64.c
+++ b/arch/x86/mm/init_64.c
@@ -1174,12 +1174,17 @@ void __init mem_init(void)
/* clear_bss() already clear the empty_zero_page */
- register_page_bootmem_info();
-
/* this will put all memory onto the freelists */
free_all_bootmem();
after_bootmem = 1;
+ /* Must be done after boot memory is put on freelist, because here we
+ * might set fields in deferred struct pages that have not yet been
+ * initialized, and free_all_bootmem() initializes all the reserved
+ * deferred pages for us.
+ */
+ register_page_bootmem_info();
+
/* Register memory areas for /proc/kcore */
kclist_add(&kcore_vsyscall, (void *)VSYSCALL_ADDR,
PAGE_SIZE, KCORE_OTHER);
--
2.14.1
^ permalink raw reply related [flat|nested] 79+ messages in thread
* [PATCH v7 01/11] x86/mm: setting fields in deferred pages
@ 2017-08-29 2:02 ` Pavel Tatashin
0 siblings, 0 replies; 79+ messages in thread
From: Pavel Tatashin @ 2017-08-29 2:02 UTC (permalink / raw)
To: linux-kernel, sparclinux, linux-mm, linuxppc-dev, linux-s390,
linux-arm-kernel, x86, kasan-dev, borntraeger, heiko.carstens,
davem, willy, mhocko, ard.biesheuvel, will.deacon,
catalin.marinas, sam, mgorman, Steven.Sistare, daniel.m.jordan,
bob.picco
Without deferred struct page feature (CONFIG_DEFERRED_STRUCT_PAGE_INIT),
flags and other fields in "struct page"es are never changed prior to first
initializing struct pages by going through __init_single_page().
With deferred struct page feature enabled, however, we set fields in
register_page_bootmem_info that are subsequently clobbered right after in
free_all_bootmem:
mem_init() {
register_page_bootmem_info();
free_all_bootmem();
...
}
When register_page_bootmem_info() is called only non-deferred struct pages
are initialized. But, this function goes through some reserved pages which
might be part of the deferred, and thus are not yet initialized.
mem_init
register_page_bootmem_info
register_page_bootmem_info_node
get_page_bootmem
.. setting fields here ..
such as: page->freelist = (void *)type;
free_all_bootmem()
free_low_memory_core_early()
for_each_reserved_mem_region()
reserve_bootmem_region()
init_reserved_page() <- Only if this is deferred reserved page
__init_single_pfn()
__init_single_page()
memset(0) <-- Loose the set fields here
We end-up with issue where, currently we do not observe problem as memory
is explicitly zeroed. But, if flag asserts are changed we can start hitting
issues.
Also, because in this patch series we will stop zeroing struct page memory
during allocation, we must make sure that struct pages are properly
initialized prior to using them.
The deferred-reserved pages are initialized in free_all_bootmem().
Therefore, the fix is to switch the above calls.
Signed-off-by: Pavel Tatashin <pasha.tatashin@oracle.com>
Reviewed-by: Steven Sistare <steven.sistare@oracle.com>
Reviewed-by: Daniel Jordan <daniel.m.jordan@oracle.com>
Reviewed-by: Bob Picco <bob.picco@oracle.com>
---
arch/x86/mm/init_64.c | 9 +++++++--
1 file changed, 7 insertions(+), 2 deletions(-)
diff --git a/arch/x86/mm/init_64.c b/arch/x86/mm/init_64.c
index 62a91e6b1237..3a997352a992 100644
--- a/arch/x86/mm/init_64.c
+++ b/arch/x86/mm/init_64.c
@@ -1174,12 +1174,17 @@ void __init mem_init(void)
/* clear_bss() already clear the empty_zero_page */
- register_page_bootmem_info();
-
/* this will put all memory onto the freelists */
free_all_bootmem();
after_bootmem = 1;
+ /* Must be done after boot memory is put on freelist, because here we
+ * might set fields in deferred struct pages that have not yet been
+ * initialized, and free_all_bootmem() initializes all the reserved
+ * deferred pages for us.
+ */
+ register_page_bootmem_info();
+
/* Register memory areas for /proc/kcore */
kclist_add(&kcore_vsyscall, (void *)VSYSCALL_ADDR,
PAGE_SIZE, KCORE_OTHER);
--
2.14.1
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply related [flat|nested] 79+ messages in thread
* [PATCH v7 01/11] x86/mm: setting fields in deferred pages
@ 2017-08-29 2:02 ` Pavel Tatashin
0 siblings, 0 replies; 79+ messages in thread
From: Pavel Tatashin @ 2017-08-29 2:02 UTC (permalink / raw)
To: linux-arm-kernel
Without deferred struct page feature (CONFIG_DEFERRED_STRUCT_PAGE_INIT),
flags and other fields in "struct page"es are never changed prior to first
initializing struct pages by going through __init_single_page().
With deferred struct page feature enabled, however, we set fields in
register_page_bootmem_info that are subsequently clobbered right after in
free_all_bootmem:
mem_init() {
register_page_bootmem_info();
free_all_bootmem();
...
}
When register_page_bootmem_info() is called only non-deferred struct pages
are initialized. But, this function goes through some reserved pages which
might be part of the deferred, and thus are not yet initialized.
mem_init
register_page_bootmem_info
register_page_bootmem_info_node
get_page_bootmem
.. setting fields here ..
such as: page->freelist = (void *)type;
free_all_bootmem()
free_low_memory_core_early()
for_each_reserved_mem_region()
reserve_bootmem_region()
init_reserved_page() <- Only if this is deferred reserved page
__init_single_pfn()
__init_single_page()
memset(0) <-- Loose the set fields here
We end-up with issue where, currently we do not observe problem as memory
is explicitly zeroed. But, if flag asserts are changed we can start hitting
issues.
Also, because in this patch series we will stop zeroing struct page memory
during allocation, we must make sure that struct pages are properly
initialized prior to using them.
The deferred-reserved pages are initialized in free_all_bootmem().
Therefore, the fix is to switch the above calls.
Signed-off-by: Pavel Tatashin <pasha.tatashin@oracle.com>
Reviewed-by: Steven Sistare <steven.sistare@oracle.com>
Reviewed-by: Daniel Jordan <daniel.m.jordan@oracle.com>
Reviewed-by: Bob Picco <bob.picco@oracle.com>
---
arch/x86/mm/init_64.c | 9 +++++++--
1 file changed, 7 insertions(+), 2 deletions(-)
diff --git a/arch/x86/mm/init_64.c b/arch/x86/mm/init_64.c
index 62a91e6b1237..3a997352a992 100644
--- a/arch/x86/mm/init_64.c
+++ b/arch/x86/mm/init_64.c
@@ -1174,12 +1174,17 @@ void __init mem_init(void)
/* clear_bss() already clear the empty_zero_page */
- register_page_bootmem_info();
-
/* this will put all memory onto the freelists */
free_all_bootmem();
after_bootmem = 1;
+ /* Must be done after boot memory is put on freelist, because here we
+ * might set fields in deferred struct pages that have not yet been
+ * initialized, and free_all_bootmem() initializes all the reserved
+ * deferred pages for us.
+ */
+ register_page_bootmem_info();
+
/* Register memory areas for /proc/kcore */
kclist_add(&kcore_vsyscall, (void *)VSYSCALL_ADDR,
PAGE_SIZE, KCORE_OTHER);
--
2.14.1
^ permalink raw reply related [flat|nested] 79+ messages in thread
* [PATCH v7 02/11] sparc64/mm: setting fields in deferred pages
2017-08-29 2:02 ` Pavel Tatashin
(?)
(?)
@ 2017-08-29 2:02 ` Pavel Tatashin
-1 siblings, 0 replies; 79+ messages in thread
From: Pavel Tatashin @ 2017-08-29 2:02 UTC (permalink / raw)
To: linux-kernel, sparclinux, linux-mm, linuxppc-dev, linux-s390,
linux-arm-kernel, x86, kasan-dev, borntraeger, heiko.carstens,
davem, willy, mhocko, ard.biesheuvel, will.deacon,
catalin.marinas, sam, mgorman, Steven.Sistare, daniel.m.jordan,
bob.picco
Without deferred struct page feature (CONFIG_DEFERRED_STRUCT_PAGE_INIT),
flags and other fields in "struct page"es are never changed prior to first
initializing struct pages by going through __init_single_page().
With deferred struct page feature enabled there is a case where we set some
fields prior to initializing:
mem_init() {
register_page_bootmem_info();
free_all_bootmem();
...
}
When register_page_bootmem_info() is called only non-deferred struct pages
are initialized. But, this function goes through some reserved pages which
might be part of the deferred, and thus are not yet initialized.
mem_init
register_page_bootmem_info
register_page_bootmem_info_node
get_page_bootmem
.. setting fields here ..
such as: page->freelist = (void *)type;
free_all_bootmem()
free_low_memory_core_early()
for_each_reserved_mem_region()
reserve_bootmem_region()
init_reserved_page() <- Only if this is deferred reserved page
__init_single_pfn()
__init_single_page()
memset(0) <-- Loose the set fields here
We end-up with similar issue as in the previous patch, where currently we
do not observe problem as memory is zeroed. But, if flag asserts are
changed we can start hitting issues.
Also, because in this patch series we will stop zeroing struct page memory
during allocation, we must make sure that struct pages are properly
initialized prior to using them.
The deferred-reserved pages are initialized in free_all_bootmem().
Therefore, the fix is to switch the above calls.
Signed-off-by: Pavel Tatashin <pasha.tatashin@oracle.com>
Reviewed-by: Steven Sistare <steven.sistare@oracle.com>
Reviewed-by: Daniel Jordan <daniel.m.jordan@oracle.com>
Reviewed-by: Bob Picco <bob.picco@oracle.com>
---
arch/sparc/mm/init_64.c | 8 +++++++-
1 file changed, 7 insertions(+), 1 deletion(-)
diff --git a/arch/sparc/mm/init_64.c b/arch/sparc/mm/init_64.c
index b3020a956b87..12dbba85a2e2 100644
--- a/arch/sparc/mm/init_64.c
+++ b/arch/sparc/mm/init_64.c
@@ -2508,9 +2508,15 @@ void __init mem_init(void)
{
high_memory = __va(last_valid_pfn << PAGE_SHIFT);
- register_page_bootmem_info();
free_all_bootmem();
+ /* Must be done after boot memory is put on freelist, because here we
+ * might set fields in deferred struct pages that have not yet been
+ * initialized, and free_all_bootmem() initializes all the reserved
+ * deferred pages for us.
+ */
+ register_page_bootmem_info();
+
/*
* Set up the zero page, mark it reserved, so that page count
* is not manipulated when freeing the page from user ptes.
--
2.14.1
^ permalink raw reply related [flat|nested] 79+ messages in thread
* [PATCH v7 02/11] sparc64/mm: setting fields in deferred pages
@ 2017-08-29 2:02 ` Pavel Tatashin
0 siblings, 0 replies; 79+ messages in thread
From: Pavel Tatashin @ 2017-08-29 2:02 UTC (permalink / raw)
To: linux-arm-kernel
Without deferred struct page feature (CONFIG_DEFERRED_STRUCT_PAGE_INIT),
flags and other fields in "struct page"es are never changed prior to first
initializing struct pages by going through __init_single_page().
With deferred struct page feature enabled there is a case where we set some
fields prior to initializing:
mem_init() {
register_page_bootmem_info();
free_all_bootmem();
...
}
When register_page_bootmem_info() is called only non-deferred struct pages
are initialized. But, this function goes through some reserved pages which
might be part of the deferred, and thus are not yet initialized.
mem_init
register_page_bootmem_info
register_page_bootmem_info_node
get_page_bootmem
.. setting fields here ..
such as: page->freelist = (void *)type;
free_all_bootmem()
free_low_memory_core_early()
for_each_reserved_mem_region()
reserve_bootmem_region()
init_reserved_page() <- Only if this is deferred reserved page
__init_single_pfn()
__init_single_page()
memset(0) <-- Loose the set fields here
We end-up with similar issue as in the previous patch, where currently we
do not observe problem as memory is zeroed. But, if flag asserts are
changed we can start hitting issues.
Also, because in this patch series we will stop zeroing struct page memory
during allocation, we must make sure that struct pages are properly
initialized prior to using them.
The deferred-reserved pages are initialized in free_all_bootmem().
Therefore, the fix is to switch the above calls.
Signed-off-by: Pavel Tatashin <pasha.tatashin@oracle.com>
Reviewed-by: Steven Sistare <steven.sistare@oracle.com>
Reviewed-by: Daniel Jordan <daniel.m.jordan@oracle.com>
Reviewed-by: Bob Picco <bob.picco@oracle.com>
---
arch/sparc/mm/init_64.c | 8 +++++++-
1 file changed, 7 insertions(+), 1 deletion(-)
diff --git a/arch/sparc/mm/init_64.c b/arch/sparc/mm/init_64.c
index b3020a956b87..12dbba85a2e2 100644
--- a/arch/sparc/mm/init_64.c
+++ b/arch/sparc/mm/init_64.c
@@ -2508,9 +2508,15 @@ void __init mem_init(void)
{
high_memory = __va(last_valid_pfn << PAGE_SHIFT);
- register_page_bootmem_info();
free_all_bootmem();
+ /* Must be done after boot memory is put on freelist, because here we
+ * might set fields in deferred struct pages that have not yet been
+ * initialized, and free_all_bootmem() initializes all the reserved
+ * deferred pages for us.
+ */
+ register_page_bootmem_info();
+
/*
* Set up the zero page, mark it reserved, so that page count
* is not manipulated when freeing the page from user ptes.
--
2.14.1
^ permalink raw reply related [flat|nested] 79+ messages in thread
* [PATCH v7 02/11] sparc64/mm: setting fields in deferred pages
@ 2017-08-29 2:02 ` Pavel Tatashin
0 siblings, 0 replies; 79+ messages in thread
From: Pavel Tatashin @ 2017-08-29 2:02 UTC (permalink / raw)
To: linux-kernel, sparclinux, linux-mm, linuxppc-dev, linux-s390,
linux-arm-kernel, x86, kasan-dev, borntraeger, heiko.carstens,
davem, willy, mhocko, ard.biesheuvel, will.deacon,
catalin.marinas, sam, mgorman, Steven.Sistare, daniel.m.jordan,
bob.picco
Without deferred struct page feature (CONFIG_DEFERRED_STRUCT_PAGE_INIT),
flags and other fields in "struct page"es are never changed prior to first
initializing struct pages by going through __init_single_page().
With deferred struct page feature enabled there is a case where we set some
fields prior to initializing:
mem_init() {
register_page_bootmem_info();
free_all_bootmem();
...
}
When register_page_bootmem_info() is called only non-deferred struct pages
are initialized. But, this function goes through some reserved pages which
might be part of the deferred, and thus are not yet initialized.
mem_init
register_page_bootmem_info
register_page_bootmem_info_node
get_page_bootmem
.. setting fields here ..
such as: page->freelist = (void *)type;
free_all_bootmem()
free_low_memory_core_early()
for_each_reserved_mem_region()
reserve_bootmem_region()
init_reserved_page() <- Only if this is deferred reserved page
__init_single_pfn()
__init_single_page()
memset(0) <-- Loose the set fields here
We end-up with similar issue as in the previous patch, where currently we
do not observe problem as memory is zeroed. But, if flag asserts are
changed we can start hitting issues.
Also, because in this patch series we will stop zeroing struct page memory
during allocation, we must make sure that struct pages are properly
initialized prior to using them.
The deferred-reserved pages are initialized in free_all_bootmem().
Therefore, the fix is to switch the above calls.
Signed-off-by: Pavel Tatashin <pasha.tatashin@oracle.com>
Reviewed-by: Steven Sistare <steven.sistare@oracle.com>
Reviewed-by: Daniel Jordan <daniel.m.jordan@oracle.com>
Reviewed-by: Bob Picco <bob.picco@oracle.com>
---
arch/sparc/mm/init_64.c | 8 +++++++-
1 file changed, 7 insertions(+), 1 deletion(-)
diff --git a/arch/sparc/mm/init_64.c b/arch/sparc/mm/init_64.c
index b3020a956b87..12dbba85a2e2 100644
--- a/arch/sparc/mm/init_64.c
+++ b/arch/sparc/mm/init_64.c
@@ -2508,9 +2508,15 @@ void __init mem_init(void)
{
high_memory = __va(last_valid_pfn << PAGE_SHIFT);
- register_page_bootmem_info();
free_all_bootmem();
+ /* Must be done after boot memory is put on freelist, because here we
+ * might set fields in deferred struct pages that have not yet been
+ * initialized, and free_all_bootmem() initializes all the reserved
+ * deferred pages for us.
+ */
+ register_page_bootmem_info();
+
/*
* Set up the zero page, mark it reserved, so that page count
* is not manipulated when freeing the page from user ptes.
--
2.14.1
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply related [flat|nested] 79+ messages in thread
* [PATCH v7 02/11] sparc64/mm: setting fields in deferred pages
@ 2017-08-29 2:02 ` Pavel Tatashin
0 siblings, 0 replies; 79+ messages in thread
From: Pavel Tatashin @ 2017-08-29 2:02 UTC (permalink / raw)
To: linux-arm-kernel
Without deferred struct page feature (CONFIG_DEFERRED_STRUCT_PAGE_INIT),
flags and other fields in "struct page"es are never changed prior to first
initializing struct pages by going through __init_single_page().
With deferred struct page feature enabled there is a case where we set some
fields prior to initializing:
mem_init() {
register_page_bootmem_info();
free_all_bootmem();
...
}
When register_page_bootmem_info() is called only non-deferred struct pages
are initialized. But, this function goes through some reserved pages which
might be part of the deferred, and thus are not yet initialized.
mem_init
register_page_bootmem_info
register_page_bootmem_info_node
get_page_bootmem
.. setting fields here ..
such as: page->freelist = (void *)type;
free_all_bootmem()
free_low_memory_core_early()
for_each_reserved_mem_region()
reserve_bootmem_region()
init_reserved_page() <- Only if this is deferred reserved page
__init_single_pfn()
__init_single_page()
memset(0) <-- Loose the set fields here
We end-up with similar issue as in the previous patch, where currently we
do not observe problem as memory is zeroed. But, if flag asserts are
changed we can start hitting issues.
Also, because in this patch series we will stop zeroing struct page memory
during allocation, we must make sure that struct pages are properly
initialized prior to using them.
The deferred-reserved pages are initialized in free_all_bootmem().
Therefore, the fix is to switch the above calls.
Signed-off-by: Pavel Tatashin <pasha.tatashin@oracle.com>
Reviewed-by: Steven Sistare <steven.sistare@oracle.com>
Reviewed-by: Daniel Jordan <daniel.m.jordan@oracle.com>
Reviewed-by: Bob Picco <bob.picco@oracle.com>
---
arch/sparc/mm/init_64.c | 8 +++++++-
1 file changed, 7 insertions(+), 1 deletion(-)
diff --git a/arch/sparc/mm/init_64.c b/arch/sparc/mm/init_64.c
index b3020a956b87..12dbba85a2e2 100644
--- a/arch/sparc/mm/init_64.c
+++ b/arch/sparc/mm/init_64.c
@@ -2508,9 +2508,15 @@ void __init mem_init(void)
{
high_memory = __va(last_valid_pfn << PAGE_SHIFT);
- register_page_bootmem_info();
free_all_bootmem();
+ /* Must be done after boot memory is put on freelist, because here we
+ * might set fields in deferred struct pages that have not yet been
+ * initialized, and free_all_bootmem() initializes all the reserved
+ * deferred pages for us.
+ */
+ register_page_bootmem_info();
+
/*
* Set up the zero page, mark it reserved, so that page count
* is not manipulated when freeing the page from user ptes.
--
2.14.1
^ permalink raw reply related [flat|nested] 79+ messages in thread
* [PATCH v7 03/11] mm: deferred_init_memmap improvements
2017-08-29 2:02 ` Pavel Tatashin
(?)
(?)
@ 2017-08-29 2:02 ` Pavel Tatashin
-1 siblings, 0 replies; 79+ messages in thread
From: Pavel Tatashin @ 2017-08-29 2:02 UTC (permalink / raw)
To: linux-kernel, sparclinux, linux-mm, linuxppc-dev, linux-s390,
linux-arm-kernel, x86, kasan-dev, borntraeger, heiko.carstens,
davem, willy, mhocko, ard.biesheuvel, will.deacon,
catalin.marinas, sam, mgorman, Steven.Sistare, daniel.m.jordan,
bob.picco
This patch fixes two issues in deferred_init_memmap
=====
In deferred_init_memmap() where all deferred struct pages are initialized
we have a check like this:
if (page->flags) {
VM_BUG_ON(page_zone(page) != zone);
goto free_range;
}
This way we are checking if the current deferred page has already been
initialized. It works, because memory for struct pages has been zeroed, and
the only way flags are not zero if it went through __init_single_page()
before. But, once we change the current behavior and won't zero the memory
in memblock allocator, we cannot trust anything inside "struct page"es
until they are initialized. This patch fixes this.
The deferred_init_memmap() is re-written to loop through only free memory
ranges provided by memblock.
=====
This patch fixes another existing issue on systems that have holes in
zones i.e CONFIG_HOLES_IN_ZONE is defined.
In for_each_mem_pfn_range() we have code like this:
if (!pfn_valid_within(pfn)
goto free_range;
Note: 'page' is not set to NULL and is not incremented but 'pfn' advances.
Thus means if deferred struct pages are enabled on systems with these kind
of holes, linux would get memory corruptions. I have fixed this issue by
defining a new macro that performs all the necessary operations when we
free the current set of pages.
Signed-off-by: Pavel Tatashin <pasha.tatashin@oracle.com>
Reviewed-by: Steven Sistare <steven.sistare@oracle.com>
Reviewed-by: Daniel Jordan <daniel.m.jordan@oracle.com>
Reviewed-by: Bob Picco <bob.picco@oracle.com>
---
mm/page_alloc.c | 161 +++++++++++++++++++++++++++-----------------------------
1 file changed, 78 insertions(+), 83 deletions(-)
diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index 7a58eb5757e3..c170ac569aec 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -1409,14 +1409,17 @@ void clear_zone_contiguous(struct zone *zone)
}
#ifdef CONFIG_DEFERRED_STRUCT_PAGE_INIT
-static void __init deferred_free_range(struct page *page,
- unsigned long pfn, int nr_pages)
+static void __init deferred_free_range(unsigned long pfn,
+ unsigned long nr_pages)
{
- int i;
+ struct page *page;
+ unsigned long i;
- if (!page)
+ if (!nr_pages)
return;
+ page = pfn_to_page(pfn);
+
/* Free a large naturally-aligned chunk if possible */
if (nr_pages == pageblock_nr_pages &&
(pfn & (pageblock_nr_pages - 1)) == 0) {
@@ -1442,19 +1445,82 @@ static inline void __init pgdat_init_report_one_done(void)
complete(&pgdat_init_all_done_comp);
}
+#define DEFERRED_FREE(nr_free, free_base_pfn, page) \
+({ \
+ unsigned long nr = (nr_free); \
+ \
+ deferred_free_range((free_base_pfn), (nr)); \
+ (free_base_pfn) = 0; \
+ (nr_free) = 0; \
+ page = NULL; \
+ nr; \
+})
+
+static unsigned long deferred_init_range(int nid, int zid, unsigned long pfn,
+ unsigned long end_pfn)
+{
+ struct mminit_pfnnid_cache nid_init_state = { };
+ unsigned long nr_pgmask = pageblock_nr_pages - 1;
+ unsigned long free_base_pfn = 0;
+ unsigned long nr_pages = 0;
+ unsigned long nr_free = 0;
+ struct page *page = NULL;
+
+ for (; pfn < end_pfn; pfn++) {
+ /*
+ * First we check if pfn is valid on architectures where it is
+ * possible to have holes within pageblock_nr_pages. On systems
+ * where it is not possible, this function is optimized out.
+ *
+ * Then, we check if a current large page is valid by only
+ * checking the validity of the head pfn.
+ *
+ * meminit_pfn_in_nid is checked on systems where pfns can
+ * interleave within a node: a pfn is between start and end
+ * of a node, but does not belong to this memory node.
+ *
+ * Finally, we minimize pfn page lookups and scheduler checks by
+ * performing it only once every pageblock_nr_pages.
+ */
+ if (!pfn_valid_within(pfn)) {
+ nr_pages += DEFERRED_FREE(nr_free, free_base_pfn, page);
+ } else if (!(pfn & nr_pgmask) && !pfn_valid(pfn)) {
+ nr_pages += DEFERRED_FREE(nr_free, free_base_pfn, page);
+ } else if (!meminit_pfn_in_nid(pfn, nid, &nid_init_state)) {
+ nr_pages += DEFERRED_FREE(nr_free, free_base_pfn, page);
+ } else if (page && (pfn & nr_pgmask)) {
+ page++;
+ __init_single_page(page, pfn, zid, nid);
+ nr_free++;
+ } else {
+ nr_pages += DEFERRED_FREE(nr_free, free_base_pfn, page);
+ page = pfn_to_page(pfn);
+ __init_single_page(page, pfn, zid, nid);
+ free_base_pfn = pfn;
+ nr_free = 1;
+ cond_resched();
+ }
+ }
+ /* Free the last block of pages to allocator */
+ nr_pages += DEFERRED_FREE(nr_free, free_base_pfn, page);
+
+ return nr_pages;
+}
+
/* Initialise remaining memory on a node */
static int __init deferred_init_memmap(void *data)
{
pg_data_t *pgdat = data;
int nid = pgdat->node_id;
- struct mminit_pfnnid_cache nid_init_state = { };
unsigned long start = jiffies;
unsigned long nr_pages = 0;
- unsigned long walk_start, walk_end;
- int i, zid;
+ unsigned long spfn, epfn;
+ phys_addr_t spa, epa;
+ int zid;
struct zone *zone;
unsigned long first_init_pfn = pgdat->first_deferred_pfn;
const struct cpumask *cpumask = cpumask_of_node(pgdat->node_id);
+ u64 i;
if (first_init_pfn == ULONG_MAX) {
pgdat_init_report_one_done();
@@ -1476,83 +1542,12 @@ static int __init deferred_init_memmap(void *data)
if (first_init_pfn < zone_end_pfn(zone))
break;
}
+ first_init_pfn = max(zone->zone_start_pfn, first_init_pfn);
- for_each_mem_pfn_range(i, nid, &walk_start, &walk_end, NULL) {
- unsigned long pfn, end_pfn;
- struct page *page = NULL;
- struct page *free_base_page = NULL;
- unsigned long free_base_pfn = 0;
- int nr_to_free = 0;
-
- end_pfn = min(walk_end, zone_end_pfn(zone));
- pfn = first_init_pfn;
- if (pfn < walk_start)
- pfn = walk_start;
- if (pfn < zone->zone_start_pfn)
- pfn = zone->zone_start_pfn;
-
- for (; pfn < end_pfn; pfn++) {
- if (!pfn_valid_within(pfn))
- goto free_range;
-
- /*
- * Ensure pfn_valid is checked every
- * pageblock_nr_pages for memory holes
- */
- if ((pfn & (pageblock_nr_pages - 1)) == 0) {
- if (!pfn_valid(pfn)) {
- page = NULL;
- goto free_range;
- }
- }
-
- if (!meminit_pfn_in_nid(pfn, nid, &nid_init_state)) {
- page = NULL;
- goto free_range;
- }
-
- /* Minimise pfn page lookups and scheduler checks */
- if (page && (pfn & (pageblock_nr_pages - 1)) != 0) {
- page++;
- } else {
- nr_pages += nr_to_free;
- deferred_free_range(free_base_page,
- free_base_pfn, nr_to_free);
- free_base_page = NULL;
- free_base_pfn = nr_to_free = 0;
-
- page = pfn_to_page(pfn);
- cond_resched();
- }
-
- if (page->flags) {
- VM_BUG_ON(page_zone(page) != zone);
- goto free_range;
- }
-
- __init_single_page(page, pfn, zid, nid);
- if (!free_base_page) {
- free_base_page = page;
- free_base_pfn = pfn;
- nr_to_free = 0;
- }
- nr_to_free++;
-
- /* Where possible, batch up pages for a single free */
- continue;
-free_range:
- /* Free the current block of pages to allocator */
- nr_pages += nr_to_free;
- deferred_free_range(free_base_page, free_base_pfn,
- nr_to_free);
- free_base_page = NULL;
- free_base_pfn = nr_to_free = 0;
- }
- /* Free the last block of pages to allocator */
- nr_pages += nr_to_free;
- deferred_free_range(free_base_page, free_base_pfn, nr_to_free);
-
- first_init_pfn = max(end_pfn, first_init_pfn);
+ for_each_free_mem_range(i, nid, MEMBLOCK_NONE, &spa, &epa, NULL) {
+ spfn = max_t(unsigned long, first_init_pfn, PFN_UP(spa));
+ epfn = min_t(unsigned long, zone_end_pfn(zone), PFN_DOWN(epa));
+ nr_pages += deferred_init_range(nid, zid, spfn, epfn);
}
/* Sanity check that the next zone really is unpopulated */
--
2.14.1
^ permalink raw reply related [flat|nested] 79+ messages in thread
* [PATCH v7 03/11] mm: deferred_init_memmap improvements
@ 2017-08-29 2:02 ` Pavel Tatashin
0 siblings, 0 replies; 79+ messages in thread
From: Pavel Tatashin @ 2017-08-29 2:02 UTC (permalink / raw)
To: linux-arm-kernel
This patch fixes two issues in deferred_init_memmap
==In deferred_init_memmap() where all deferred struct pages are initialized
we have a check like this:
if (page->flags) {
VM_BUG_ON(page_zone(page) != zone);
goto free_range;
}
This way we are checking if the current deferred page has already been
initialized. It works, because memory for struct pages has been zeroed, and
the only way flags are not zero if it went through __init_single_page()
before. But, once we change the current behavior and won't zero the memory
in memblock allocator, we cannot trust anything inside "struct page"es
until they are initialized. This patch fixes this.
The deferred_init_memmap() is re-written to loop through only free memory
ranges provided by memblock.
==This patch fixes another existing issue on systems that have holes in
zones i.e CONFIG_HOLES_IN_ZONE is defined.
In for_each_mem_pfn_range() we have code like this:
if (!pfn_valid_within(pfn)
goto free_range;
Note: 'page' is not set to NULL and is not incremented but 'pfn' advances.
Thus means if deferred struct pages are enabled on systems with these kind
of holes, linux would get memory corruptions. I have fixed this issue by
defining a new macro that performs all the necessary operations when we
free the current set of pages.
Signed-off-by: Pavel Tatashin <pasha.tatashin@oracle.com>
Reviewed-by: Steven Sistare <steven.sistare@oracle.com>
Reviewed-by: Daniel Jordan <daniel.m.jordan@oracle.com>
Reviewed-by: Bob Picco <bob.picco@oracle.com>
---
mm/page_alloc.c | 161 +++++++++++++++++++++++++++-----------------------------
1 file changed, 78 insertions(+), 83 deletions(-)
diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index 7a58eb5757e3..c170ac569aec 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -1409,14 +1409,17 @@ void clear_zone_contiguous(struct zone *zone)
}
#ifdef CONFIG_DEFERRED_STRUCT_PAGE_INIT
-static void __init deferred_free_range(struct page *page,
- unsigned long pfn, int nr_pages)
+static void __init deferred_free_range(unsigned long pfn,
+ unsigned long nr_pages)
{
- int i;
+ struct page *page;
+ unsigned long i;
- if (!page)
+ if (!nr_pages)
return;
+ page = pfn_to_page(pfn);
+
/* Free a large naturally-aligned chunk if possible */
if (nr_pages = pageblock_nr_pages &&
(pfn & (pageblock_nr_pages - 1)) = 0) {
@@ -1442,19 +1445,82 @@ static inline void __init pgdat_init_report_one_done(void)
complete(&pgdat_init_all_done_comp);
}
+#define DEFERRED_FREE(nr_free, free_base_pfn, page) \
+({ \
+ unsigned long nr = (nr_free); \
+ \
+ deferred_free_range((free_base_pfn), (nr)); \
+ (free_base_pfn) = 0; \
+ (nr_free) = 0; \
+ page = NULL; \
+ nr; \
+})
+
+static unsigned long deferred_init_range(int nid, int zid, unsigned long pfn,
+ unsigned long end_pfn)
+{
+ struct mminit_pfnnid_cache nid_init_state = { };
+ unsigned long nr_pgmask = pageblock_nr_pages - 1;
+ unsigned long free_base_pfn = 0;
+ unsigned long nr_pages = 0;
+ unsigned long nr_free = 0;
+ struct page *page = NULL;
+
+ for (; pfn < end_pfn; pfn++) {
+ /*
+ * First we check if pfn is valid on architectures where it is
+ * possible to have holes within pageblock_nr_pages. On systems
+ * where it is not possible, this function is optimized out.
+ *
+ * Then, we check if a current large page is valid by only
+ * checking the validity of the head pfn.
+ *
+ * meminit_pfn_in_nid is checked on systems where pfns can
+ * interleave within a node: a pfn is between start and end
+ * of a node, but does not belong to this memory node.
+ *
+ * Finally, we minimize pfn page lookups and scheduler checks by
+ * performing it only once every pageblock_nr_pages.
+ */
+ if (!pfn_valid_within(pfn)) {
+ nr_pages += DEFERRED_FREE(nr_free, free_base_pfn, page);
+ } else if (!(pfn & nr_pgmask) && !pfn_valid(pfn)) {
+ nr_pages += DEFERRED_FREE(nr_free, free_base_pfn, page);
+ } else if (!meminit_pfn_in_nid(pfn, nid, &nid_init_state)) {
+ nr_pages += DEFERRED_FREE(nr_free, free_base_pfn, page);
+ } else if (page && (pfn & nr_pgmask)) {
+ page++;
+ __init_single_page(page, pfn, zid, nid);
+ nr_free++;
+ } else {
+ nr_pages += DEFERRED_FREE(nr_free, free_base_pfn, page);
+ page = pfn_to_page(pfn);
+ __init_single_page(page, pfn, zid, nid);
+ free_base_pfn = pfn;
+ nr_free = 1;
+ cond_resched();
+ }
+ }
+ /* Free the last block of pages to allocator */
+ nr_pages += DEFERRED_FREE(nr_free, free_base_pfn, page);
+
+ return nr_pages;
+}
+
/* Initialise remaining memory on a node */
static int __init deferred_init_memmap(void *data)
{
pg_data_t *pgdat = data;
int nid = pgdat->node_id;
- struct mminit_pfnnid_cache nid_init_state = { };
unsigned long start = jiffies;
unsigned long nr_pages = 0;
- unsigned long walk_start, walk_end;
- int i, zid;
+ unsigned long spfn, epfn;
+ phys_addr_t spa, epa;
+ int zid;
struct zone *zone;
unsigned long first_init_pfn = pgdat->first_deferred_pfn;
const struct cpumask *cpumask = cpumask_of_node(pgdat->node_id);
+ u64 i;
if (first_init_pfn = ULONG_MAX) {
pgdat_init_report_one_done();
@@ -1476,83 +1542,12 @@ static int __init deferred_init_memmap(void *data)
if (first_init_pfn < zone_end_pfn(zone))
break;
}
+ first_init_pfn = max(zone->zone_start_pfn, first_init_pfn);
- for_each_mem_pfn_range(i, nid, &walk_start, &walk_end, NULL) {
- unsigned long pfn, end_pfn;
- struct page *page = NULL;
- struct page *free_base_page = NULL;
- unsigned long free_base_pfn = 0;
- int nr_to_free = 0;
-
- end_pfn = min(walk_end, zone_end_pfn(zone));
- pfn = first_init_pfn;
- if (pfn < walk_start)
- pfn = walk_start;
- if (pfn < zone->zone_start_pfn)
- pfn = zone->zone_start_pfn;
-
- for (; pfn < end_pfn; pfn++) {
- if (!pfn_valid_within(pfn))
- goto free_range;
-
- /*
- * Ensure pfn_valid is checked every
- * pageblock_nr_pages for memory holes
- */
- if ((pfn & (pageblock_nr_pages - 1)) = 0) {
- if (!pfn_valid(pfn)) {
- page = NULL;
- goto free_range;
- }
- }
-
- if (!meminit_pfn_in_nid(pfn, nid, &nid_init_state)) {
- page = NULL;
- goto free_range;
- }
-
- /* Minimise pfn page lookups and scheduler checks */
- if (page && (pfn & (pageblock_nr_pages - 1)) != 0) {
- page++;
- } else {
- nr_pages += nr_to_free;
- deferred_free_range(free_base_page,
- free_base_pfn, nr_to_free);
- free_base_page = NULL;
- free_base_pfn = nr_to_free = 0;
-
- page = pfn_to_page(pfn);
- cond_resched();
- }
-
- if (page->flags) {
- VM_BUG_ON(page_zone(page) != zone);
- goto free_range;
- }
-
- __init_single_page(page, pfn, zid, nid);
- if (!free_base_page) {
- free_base_page = page;
- free_base_pfn = pfn;
- nr_to_free = 0;
- }
- nr_to_free++;
-
- /* Where possible, batch up pages for a single free */
- continue;
-free_range:
- /* Free the current block of pages to allocator */
- nr_pages += nr_to_free;
- deferred_free_range(free_base_page, free_base_pfn,
- nr_to_free);
- free_base_page = NULL;
- free_base_pfn = nr_to_free = 0;
- }
- /* Free the last block of pages to allocator */
- nr_pages += nr_to_free;
- deferred_free_range(free_base_page, free_base_pfn, nr_to_free);
-
- first_init_pfn = max(end_pfn, first_init_pfn);
+ for_each_free_mem_range(i, nid, MEMBLOCK_NONE, &spa, &epa, NULL) {
+ spfn = max_t(unsigned long, first_init_pfn, PFN_UP(spa));
+ epfn = min_t(unsigned long, zone_end_pfn(zone), PFN_DOWN(epa));
+ nr_pages += deferred_init_range(nid, zid, spfn, epfn);
}
/* Sanity check that the next zone really is unpopulated */
--
2.14.1
^ permalink raw reply related [flat|nested] 79+ messages in thread
* [PATCH v7 03/11] mm: deferred_init_memmap improvements
@ 2017-08-29 2:02 ` Pavel Tatashin
0 siblings, 0 replies; 79+ messages in thread
From: Pavel Tatashin @ 2017-08-29 2:02 UTC (permalink / raw)
To: linux-kernel, sparclinux, linux-mm, linuxppc-dev, linux-s390,
linux-arm-kernel, x86, kasan-dev, borntraeger, heiko.carstens,
davem, willy, mhocko, ard.biesheuvel, will.deacon,
catalin.marinas, sam, mgorman, Steven.Sistare, daniel.m.jordan,
bob.picco
This patch fixes two issues in deferred_init_memmap
=====
In deferred_init_memmap() where all deferred struct pages are initialized
we have a check like this:
if (page->flags) {
VM_BUG_ON(page_zone(page) != zone);
goto free_range;
}
This way we are checking if the current deferred page has already been
initialized. It works, because memory for struct pages has been zeroed, and
the only way flags are not zero if it went through __init_single_page()
before. But, once we change the current behavior and won't zero the memory
in memblock allocator, we cannot trust anything inside "struct page"es
until they are initialized. This patch fixes this.
The deferred_init_memmap() is re-written to loop through only free memory
ranges provided by memblock.
=====
This patch fixes another existing issue on systems that have holes in
zones i.e CONFIG_HOLES_IN_ZONE is defined.
In for_each_mem_pfn_range() we have code like this:
if (!pfn_valid_within(pfn)
goto free_range;
Note: 'page' is not set to NULL and is not incremented but 'pfn' advances.
Thus means if deferred struct pages are enabled on systems with these kind
of holes, linux would get memory corruptions. I have fixed this issue by
defining a new macro that performs all the necessary operations when we
free the current set of pages.
Signed-off-by: Pavel Tatashin <pasha.tatashin@oracle.com>
Reviewed-by: Steven Sistare <steven.sistare@oracle.com>
Reviewed-by: Daniel Jordan <daniel.m.jordan@oracle.com>
Reviewed-by: Bob Picco <bob.picco@oracle.com>
---
mm/page_alloc.c | 161 +++++++++++++++++++++++++++-----------------------------
1 file changed, 78 insertions(+), 83 deletions(-)
diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index 7a58eb5757e3..c170ac569aec 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -1409,14 +1409,17 @@ void clear_zone_contiguous(struct zone *zone)
}
#ifdef CONFIG_DEFERRED_STRUCT_PAGE_INIT
-static void __init deferred_free_range(struct page *page,
- unsigned long pfn, int nr_pages)
+static void __init deferred_free_range(unsigned long pfn,
+ unsigned long nr_pages)
{
- int i;
+ struct page *page;
+ unsigned long i;
- if (!page)
+ if (!nr_pages)
return;
+ page = pfn_to_page(pfn);
+
/* Free a large naturally-aligned chunk if possible */
if (nr_pages == pageblock_nr_pages &&
(pfn & (pageblock_nr_pages - 1)) == 0) {
@@ -1442,19 +1445,82 @@ static inline void __init pgdat_init_report_one_done(void)
complete(&pgdat_init_all_done_comp);
}
+#define DEFERRED_FREE(nr_free, free_base_pfn, page) \
+({ \
+ unsigned long nr = (nr_free); \
+ \
+ deferred_free_range((free_base_pfn), (nr)); \
+ (free_base_pfn) = 0; \
+ (nr_free) = 0; \
+ page = NULL; \
+ nr; \
+})
+
+static unsigned long deferred_init_range(int nid, int zid, unsigned long pfn,
+ unsigned long end_pfn)
+{
+ struct mminit_pfnnid_cache nid_init_state = { };
+ unsigned long nr_pgmask = pageblock_nr_pages - 1;
+ unsigned long free_base_pfn = 0;
+ unsigned long nr_pages = 0;
+ unsigned long nr_free = 0;
+ struct page *page = NULL;
+
+ for (; pfn < end_pfn; pfn++) {
+ /*
+ * First we check if pfn is valid on architectures where it is
+ * possible to have holes within pageblock_nr_pages. On systems
+ * where it is not possible, this function is optimized out.
+ *
+ * Then, we check if a current large page is valid by only
+ * checking the validity of the head pfn.
+ *
+ * meminit_pfn_in_nid is checked on systems where pfns can
+ * interleave within a node: a pfn is between start and end
+ * of a node, but does not belong to this memory node.
+ *
+ * Finally, we minimize pfn page lookups and scheduler checks by
+ * performing it only once every pageblock_nr_pages.
+ */
+ if (!pfn_valid_within(pfn)) {
+ nr_pages += DEFERRED_FREE(nr_free, free_base_pfn, page);
+ } else if (!(pfn & nr_pgmask) && !pfn_valid(pfn)) {
+ nr_pages += DEFERRED_FREE(nr_free, free_base_pfn, page);
+ } else if (!meminit_pfn_in_nid(pfn, nid, &nid_init_state)) {
+ nr_pages += DEFERRED_FREE(nr_free, free_base_pfn, page);
+ } else if (page && (pfn & nr_pgmask)) {
+ page++;
+ __init_single_page(page, pfn, zid, nid);
+ nr_free++;
+ } else {
+ nr_pages += DEFERRED_FREE(nr_free, free_base_pfn, page);
+ page = pfn_to_page(pfn);
+ __init_single_page(page, pfn, zid, nid);
+ free_base_pfn = pfn;
+ nr_free = 1;
+ cond_resched();
+ }
+ }
+ /* Free the last block of pages to allocator */
+ nr_pages += DEFERRED_FREE(nr_free, free_base_pfn, page);
+
+ return nr_pages;
+}
+
/* Initialise remaining memory on a node */
static int __init deferred_init_memmap(void *data)
{
pg_data_t *pgdat = data;
int nid = pgdat->node_id;
- struct mminit_pfnnid_cache nid_init_state = { };
unsigned long start = jiffies;
unsigned long nr_pages = 0;
- unsigned long walk_start, walk_end;
- int i, zid;
+ unsigned long spfn, epfn;
+ phys_addr_t spa, epa;
+ int zid;
struct zone *zone;
unsigned long first_init_pfn = pgdat->first_deferred_pfn;
const struct cpumask *cpumask = cpumask_of_node(pgdat->node_id);
+ u64 i;
if (first_init_pfn == ULONG_MAX) {
pgdat_init_report_one_done();
@@ -1476,83 +1542,12 @@ static int __init deferred_init_memmap(void *data)
if (first_init_pfn < zone_end_pfn(zone))
break;
}
+ first_init_pfn = max(zone->zone_start_pfn, first_init_pfn);
- for_each_mem_pfn_range(i, nid, &walk_start, &walk_end, NULL) {
- unsigned long pfn, end_pfn;
- struct page *page = NULL;
- struct page *free_base_page = NULL;
- unsigned long free_base_pfn = 0;
- int nr_to_free = 0;
-
- end_pfn = min(walk_end, zone_end_pfn(zone));
- pfn = first_init_pfn;
- if (pfn < walk_start)
- pfn = walk_start;
- if (pfn < zone->zone_start_pfn)
- pfn = zone->zone_start_pfn;
-
- for (; pfn < end_pfn; pfn++) {
- if (!pfn_valid_within(pfn))
- goto free_range;
-
- /*
- * Ensure pfn_valid is checked every
- * pageblock_nr_pages for memory holes
- */
- if ((pfn & (pageblock_nr_pages - 1)) == 0) {
- if (!pfn_valid(pfn)) {
- page = NULL;
- goto free_range;
- }
- }
-
- if (!meminit_pfn_in_nid(pfn, nid, &nid_init_state)) {
- page = NULL;
- goto free_range;
- }
-
- /* Minimise pfn page lookups and scheduler checks */
- if (page && (pfn & (pageblock_nr_pages - 1)) != 0) {
- page++;
- } else {
- nr_pages += nr_to_free;
- deferred_free_range(free_base_page,
- free_base_pfn, nr_to_free);
- free_base_page = NULL;
- free_base_pfn = nr_to_free = 0;
-
- page = pfn_to_page(pfn);
- cond_resched();
- }
-
- if (page->flags) {
- VM_BUG_ON(page_zone(page) != zone);
- goto free_range;
- }
-
- __init_single_page(page, pfn, zid, nid);
- if (!free_base_page) {
- free_base_page = page;
- free_base_pfn = pfn;
- nr_to_free = 0;
- }
- nr_to_free++;
-
- /* Where possible, batch up pages for a single free */
- continue;
-free_range:
- /* Free the current block of pages to allocator */
- nr_pages += nr_to_free;
- deferred_free_range(free_base_page, free_base_pfn,
- nr_to_free);
- free_base_page = NULL;
- free_base_pfn = nr_to_free = 0;
- }
- /* Free the last block of pages to allocator */
- nr_pages += nr_to_free;
- deferred_free_range(free_base_page, free_base_pfn, nr_to_free);
-
- first_init_pfn = max(end_pfn, first_init_pfn);
+ for_each_free_mem_range(i, nid, MEMBLOCK_NONE, &spa, &epa, NULL) {
+ spfn = max_t(unsigned long, first_init_pfn, PFN_UP(spa));
+ epfn = min_t(unsigned long, zone_end_pfn(zone), PFN_DOWN(epa));
+ nr_pages += deferred_init_range(nid, zid, spfn, epfn);
}
/* Sanity check that the next zone really is unpopulated */
--
2.14.1
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply related [flat|nested] 79+ messages in thread
* [PATCH v7 03/11] mm: deferred_init_memmap improvements
@ 2017-08-29 2:02 ` Pavel Tatashin
0 siblings, 0 replies; 79+ messages in thread
From: Pavel Tatashin @ 2017-08-29 2:02 UTC (permalink / raw)
To: linux-arm-kernel
This patch fixes two issues in deferred_init_memmap
=====
In deferred_init_memmap() where all deferred struct pages are initialized
we have a check like this:
if (page->flags) {
VM_BUG_ON(page_zone(page) != zone);
goto free_range;
}
This way we are checking if the current deferred page has already been
initialized. It works, because memory for struct pages has been zeroed, and
the only way flags are not zero if it went through __init_single_page()
before. But, once we change the current behavior and won't zero the memory
in memblock allocator, we cannot trust anything inside "struct page"es
until they are initialized. This patch fixes this.
The deferred_init_memmap() is re-written to loop through only free memory
ranges provided by memblock.
=====
This patch fixes another existing issue on systems that have holes in
zones i.e CONFIG_HOLES_IN_ZONE is defined.
In for_each_mem_pfn_range() we have code like this:
if (!pfn_valid_within(pfn)
goto free_range;
Note: 'page' is not set to NULL and is not incremented but 'pfn' advances.
Thus means if deferred struct pages are enabled on systems with these kind
of holes, linux would get memory corruptions. I have fixed this issue by
defining a new macro that performs all the necessary operations when we
free the current set of pages.
Signed-off-by: Pavel Tatashin <pasha.tatashin@oracle.com>
Reviewed-by: Steven Sistare <steven.sistare@oracle.com>
Reviewed-by: Daniel Jordan <daniel.m.jordan@oracle.com>
Reviewed-by: Bob Picco <bob.picco@oracle.com>
---
mm/page_alloc.c | 161 +++++++++++++++++++++++++++-----------------------------
1 file changed, 78 insertions(+), 83 deletions(-)
diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index 7a58eb5757e3..c170ac569aec 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -1409,14 +1409,17 @@ void clear_zone_contiguous(struct zone *zone)
}
#ifdef CONFIG_DEFERRED_STRUCT_PAGE_INIT
-static void __init deferred_free_range(struct page *page,
- unsigned long pfn, int nr_pages)
+static void __init deferred_free_range(unsigned long pfn,
+ unsigned long nr_pages)
{
- int i;
+ struct page *page;
+ unsigned long i;
- if (!page)
+ if (!nr_pages)
return;
+ page = pfn_to_page(pfn);
+
/* Free a large naturally-aligned chunk if possible */
if (nr_pages == pageblock_nr_pages &&
(pfn & (pageblock_nr_pages - 1)) == 0) {
@@ -1442,19 +1445,82 @@ static inline void __init pgdat_init_report_one_done(void)
complete(&pgdat_init_all_done_comp);
}
+#define DEFERRED_FREE(nr_free, free_base_pfn, page) \
+({ \
+ unsigned long nr = (nr_free); \
+ \
+ deferred_free_range((free_base_pfn), (nr)); \
+ (free_base_pfn) = 0; \
+ (nr_free) = 0; \
+ page = NULL; \
+ nr; \
+})
+
+static unsigned long deferred_init_range(int nid, int zid, unsigned long pfn,
+ unsigned long end_pfn)
+{
+ struct mminit_pfnnid_cache nid_init_state = { };
+ unsigned long nr_pgmask = pageblock_nr_pages - 1;
+ unsigned long free_base_pfn = 0;
+ unsigned long nr_pages = 0;
+ unsigned long nr_free = 0;
+ struct page *page = NULL;
+
+ for (; pfn < end_pfn; pfn++) {
+ /*
+ * First we check if pfn is valid on architectures where it is
+ * possible to have holes within pageblock_nr_pages. On systems
+ * where it is not possible, this function is optimized out.
+ *
+ * Then, we check if a current large page is valid by only
+ * checking the validity of the head pfn.
+ *
+ * meminit_pfn_in_nid is checked on systems where pfns can
+ * interleave within a node: a pfn is between start and end
+ * of a node, but does not belong to this memory node.
+ *
+ * Finally, we minimize pfn page lookups and scheduler checks by
+ * performing it only once every pageblock_nr_pages.
+ */
+ if (!pfn_valid_within(pfn)) {
+ nr_pages += DEFERRED_FREE(nr_free, free_base_pfn, page);
+ } else if (!(pfn & nr_pgmask) && !pfn_valid(pfn)) {
+ nr_pages += DEFERRED_FREE(nr_free, free_base_pfn, page);
+ } else if (!meminit_pfn_in_nid(pfn, nid, &nid_init_state)) {
+ nr_pages += DEFERRED_FREE(nr_free, free_base_pfn, page);
+ } else if (page && (pfn & nr_pgmask)) {
+ page++;
+ __init_single_page(page, pfn, zid, nid);
+ nr_free++;
+ } else {
+ nr_pages += DEFERRED_FREE(nr_free, free_base_pfn, page);
+ page = pfn_to_page(pfn);
+ __init_single_page(page, pfn, zid, nid);
+ free_base_pfn = pfn;
+ nr_free = 1;
+ cond_resched();
+ }
+ }
+ /* Free the last block of pages to allocator */
+ nr_pages += DEFERRED_FREE(nr_free, free_base_pfn, page);
+
+ return nr_pages;
+}
+
/* Initialise remaining memory on a node */
static int __init deferred_init_memmap(void *data)
{
pg_data_t *pgdat = data;
int nid = pgdat->node_id;
- struct mminit_pfnnid_cache nid_init_state = { };
unsigned long start = jiffies;
unsigned long nr_pages = 0;
- unsigned long walk_start, walk_end;
- int i, zid;
+ unsigned long spfn, epfn;
+ phys_addr_t spa, epa;
+ int zid;
struct zone *zone;
unsigned long first_init_pfn = pgdat->first_deferred_pfn;
const struct cpumask *cpumask = cpumask_of_node(pgdat->node_id);
+ u64 i;
if (first_init_pfn == ULONG_MAX) {
pgdat_init_report_one_done();
@@ -1476,83 +1542,12 @@ static int __init deferred_init_memmap(void *data)
if (first_init_pfn < zone_end_pfn(zone))
break;
}
+ first_init_pfn = max(zone->zone_start_pfn, first_init_pfn);
- for_each_mem_pfn_range(i, nid, &walk_start, &walk_end, NULL) {
- unsigned long pfn, end_pfn;
- struct page *page = NULL;
- struct page *free_base_page = NULL;
- unsigned long free_base_pfn = 0;
- int nr_to_free = 0;
-
- end_pfn = min(walk_end, zone_end_pfn(zone));
- pfn = first_init_pfn;
- if (pfn < walk_start)
- pfn = walk_start;
- if (pfn < zone->zone_start_pfn)
- pfn = zone->zone_start_pfn;
-
- for (; pfn < end_pfn; pfn++) {
- if (!pfn_valid_within(pfn))
- goto free_range;
-
- /*
- * Ensure pfn_valid is checked every
- * pageblock_nr_pages for memory holes
- */
- if ((pfn & (pageblock_nr_pages - 1)) == 0) {
- if (!pfn_valid(pfn)) {
- page = NULL;
- goto free_range;
- }
- }
-
- if (!meminit_pfn_in_nid(pfn, nid, &nid_init_state)) {
- page = NULL;
- goto free_range;
- }
-
- /* Minimise pfn page lookups and scheduler checks */
- if (page && (pfn & (pageblock_nr_pages - 1)) != 0) {
- page++;
- } else {
- nr_pages += nr_to_free;
- deferred_free_range(free_base_page,
- free_base_pfn, nr_to_free);
- free_base_page = NULL;
- free_base_pfn = nr_to_free = 0;
-
- page = pfn_to_page(pfn);
- cond_resched();
- }
-
- if (page->flags) {
- VM_BUG_ON(page_zone(page) != zone);
- goto free_range;
- }
-
- __init_single_page(page, pfn, zid, nid);
- if (!free_base_page) {
- free_base_page = page;
- free_base_pfn = pfn;
- nr_to_free = 0;
- }
- nr_to_free++;
-
- /* Where possible, batch up pages for a single free */
- continue;
-free_range:
- /* Free the current block of pages to allocator */
- nr_pages += nr_to_free;
- deferred_free_range(free_base_page, free_base_pfn,
- nr_to_free);
- free_base_page = NULL;
- free_base_pfn = nr_to_free = 0;
- }
- /* Free the last block of pages to allocator */
- nr_pages += nr_to_free;
- deferred_free_range(free_base_page, free_base_pfn, nr_to_free);
-
- first_init_pfn = max(end_pfn, first_init_pfn);
+ for_each_free_mem_range(i, nid, MEMBLOCK_NONE, &spa, &epa, NULL) {
+ spfn = max_t(unsigned long, first_init_pfn, PFN_UP(spa));
+ epfn = min_t(unsigned long, zone_end_pfn(zone), PFN_DOWN(epa));
+ nr_pages += deferred_init_range(nid, zid, spfn, epfn);
}
/* Sanity check that the next zone really is unpopulated */
--
2.14.1
^ permalink raw reply related [flat|nested] 79+ messages in thread
* [PATCH v7 04/11] sparc64: simplify vmemmap_populate
2017-08-29 2:02 ` Pavel Tatashin
(?)
(?)
@ 2017-08-29 2:02 ` Pavel Tatashin
-1 siblings, 0 replies; 79+ messages in thread
From: Pavel Tatashin @ 2017-08-29 2:02 UTC (permalink / raw)
To: linux-kernel, sparclinux, linux-mm, linuxppc-dev, linux-s390,
linux-arm-kernel, x86, kasan-dev, borntraeger, heiko.carstens,
davem, willy, mhocko, ard.biesheuvel, will.deacon,
catalin.marinas, sam, mgorman, Steven.Sistare, daniel.m.jordan,
bob.picco
Remove duplicating code by using common functions
vmemmap_pud_populate and vmemmap_pgd_populate.
Signed-off-by: Pavel Tatashin <pasha.tatashin@oracle.com>
Reviewed-by: Steven Sistare <steven.sistare@oracle.com>
Reviewed-by: Daniel Jordan <daniel.m.jordan@oracle.com>
Reviewed-by: Bob Picco <bob.picco@oracle.com>
---
arch/sparc/mm/init_64.c | 23 ++++++-----------------
1 file changed, 6 insertions(+), 17 deletions(-)
diff --git a/arch/sparc/mm/init_64.c b/arch/sparc/mm/init_64.c
index 12dbba85a2e2..a603d2c9087d 100644
--- a/arch/sparc/mm/init_64.c
+++ b/arch/sparc/mm/init_64.c
@@ -2611,30 +2611,19 @@ int __meminit vmemmap_populate(unsigned long vstart, unsigned long vend,
vstart = vstart & PMD_MASK;
vend = ALIGN(vend, PMD_SIZE);
for (; vstart < vend; vstart += PMD_SIZE) {
- pgd_t *pgd = pgd_offset_k(vstart);
+ pgd_t *pgd = vmemmap_pgd_populate(vstart, node);
unsigned long pte;
pud_t *pud;
pmd_t *pmd;
- if (pgd_none(*pgd)) {
- pud_t *new = vmemmap_alloc_block(PAGE_SIZE, node);
+ if (!pgd)
+ return -ENOMEM;
- if (!new)
- return -ENOMEM;
- pgd_populate(&init_mm, pgd, new);
- }
-
- pud = pud_offset(pgd, vstart);
- if (pud_none(*pud)) {
- pmd_t *new = vmemmap_alloc_block(PAGE_SIZE, node);
-
- if (!new)
- return -ENOMEM;
- pud_populate(&init_mm, pud, new);
- }
+ pud = vmemmap_pud_populate(pgd, vstart, node);
+ if (!pud)
+ return -ENOMEM;
pmd = pmd_offset(pud, vstart);
-
pte = pmd_val(*pmd);
if (!(pte & _PAGE_VALID)) {
void *block = vmemmap_alloc_block(PMD_SIZE, node);
--
2.14.1
^ permalink raw reply related [flat|nested] 79+ messages in thread
* [PATCH v7 04/11] sparc64: simplify vmemmap_populate
@ 2017-08-29 2:02 ` Pavel Tatashin
0 siblings, 0 replies; 79+ messages in thread
From: Pavel Tatashin @ 2017-08-29 2:02 UTC (permalink / raw)
To: linux-arm-kernel
Remove duplicating code by using common functions
vmemmap_pud_populate and vmemmap_pgd_populate.
Signed-off-by: Pavel Tatashin <pasha.tatashin@oracle.com>
Reviewed-by: Steven Sistare <steven.sistare@oracle.com>
Reviewed-by: Daniel Jordan <daniel.m.jordan@oracle.com>
Reviewed-by: Bob Picco <bob.picco@oracle.com>
---
arch/sparc/mm/init_64.c | 23 ++++++-----------------
1 file changed, 6 insertions(+), 17 deletions(-)
diff --git a/arch/sparc/mm/init_64.c b/arch/sparc/mm/init_64.c
index 12dbba85a2e2..a603d2c9087d 100644
--- a/arch/sparc/mm/init_64.c
+++ b/arch/sparc/mm/init_64.c
@@ -2611,30 +2611,19 @@ int __meminit vmemmap_populate(unsigned long vstart, unsigned long vend,
vstart = vstart & PMD_MASK;
vend = ALIGN(vend, PMD_SIZE);
for (; vstart < vend; vstart += PMD_SIZE) {
- pgd_t *pgd = pgd_offset_k(vstart);
+ pgd_t *pgd = vmemmap_pgd_populate(vstart, node);
unsigned long pte;
pud_t *pud;
pmd_t *pmd;
- if (pgd_none(*pgd)) {
- pud_t *new = vmemmap_alloc_block(PAGE_SIZE, node);
+ if (!pgd)
+ return -ENOMEM;
- if (!new)
- return -ENOMEM;
- pgd_populate(&init_mm, pgd, new);
- }
-
- pud = pud_offset(pgd, vstart);
- if (pud_none(*pud)) {
- pmd_t *new = vmemmap_alloc_block(PAGE_SIZE, node);
-
- if (!new)
- return -ENOMEM;
- pud_populate(&init_mm, pud, new);
- }
+ pud = vmemmap_pud_populate(pgd, vstart, node);
+ if (!pud)
+ return -ENOMEM;
pmd = pmd_offset(pud, vstart);
-
pte = pmd_val(*pmd);
if (!(pte & _PAGE_VALID)) {
void *block = vmemmap_alloc_block(PMD_SIZE, node);
--
2.14.1
^ permalink raw reply related [flat|nested] 79+ messages in thread
* [PATCH v7 04/11] sparc64: simplify vmemmap_populate
@ 2017-08-29 2:02 ` Pavel Tatashin
0 siblings, 0 replies; 79+ messages in thread
From: Pavel Tatashin @ 2017-08-29 2:02 UTC (permalink / raw)
To: linux-kernel, sparclinux, linux-mm, linuxppc-dev, linux-s390,
linux-arm-kernel, x86, kasan-dev, borntraeger, heiko.carstens,
davem, willy, mhocko, ard.biesheuvel, will.deacon,
catalin.marinas, sam, mgorman, Steven.Sistare, daniel.m.jordan,
bob.picco
Remove duplicating code by using common functions
vmemmap_pud_populate and vmemmap_pgd_populate.
Signed-off-by: Pavel Tatashin <pasha.tatashin@oracle.com>
Reviewed-by: Steven Sistare <steven.sistare@oracle.com>
Reviewed-by: Daniel Jordan <daniel.m.jordan@oracle.com>
Reviewed-by: Bob Picco <bob.picco@oracle.com>
---
arch/sparc/mm/init_64.c | 23 ++++++-----------------
1 file changed, 6 insertions(+), 17 deletions(-)
diff --git a/arch/sparc/mm/init_64.c b/arch/sparc/mm/init_64.c
index 12dbba85a2e2..a603d2c9087d 100644
--- a/arch/sparc/mm/init_64.c
+++ b/arch/sparc/mm/init_64.c
@@ -2611,30 +2611,19 @@ int __meminit vmemmap_populate(unsigned long vstart, unsigned long vend,
vstart = vstart & PMD_MASK;
vend = ALIGN(vend, PMD_SIZE);
for (; vstart < vend; vstart += PMD_SIZE) {
- pgd_t *pgd = pgd_offset_k(vstart);
+ pgd_t *pgd = vmemmap_pgd_populate(vstart, node);
unsigned long pte;
pud_t *pud;
pmd_t *pmd;
- if (pgd_none(*pgd)) {
- pud_t *new = vmemmap_alloc_block(PAGE_SIZE, node);
+ if (!pgd)
+ return -ENOMEM;
- if (!new)
- return -ENOMEM;
- pgd_populate(&init_mm, pgd, new);
- }
-
- pud = pud_offset(pgd, vstart);
- if (pud_none(*pud)) {
- pmd_t *new = vmemmap_alloc_block(PAGE_SIZE, node);
-
- if (!new)
- return -ENOMEM;
- pud_populate(&init_mm, pud, new);
- }
+ pud = vmemmap_pud_populate(pgd, vstart, node);
+ if (!pud)
+ return -ENOMEM;
pmd = pmd_offset(pud, vstart);
-
pte = pmd_val(*pmd);
if (!(pte & _PAGE_VALID)) {
void *block = vmemmap_alloc_block(PMD_SIZE, node);
--
2.14.1
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply related [flat|nested] 79+ messages in thread
* [PATCH v7 04/11] sparc64: simplify vmemmap_populate
@ 2017-08-29 2:02 ` Pavel Tatashin
0 siblings, 0 replies; 79+ messages in thread
From: Pavel Tatashin @ 2017-08-29 2:02 UTC (permalink / raw)
To: linux-arm-kernel
Remove duplicating code by using common functions
vmemmap_pud_populate and vmemmap_pgd_populate.
Signed-off-by: Pavel Tatashin <pasha.tatashin@oracle.com>
Reviewed-by: Steven Sistare <steven.sistare@oracle.com>
Reviewed-by: Daniel Jordan <daniel.m.jordan@oracle.com>
Reviewed-by: Bob Picco <bob.picco@oracle.com>
---
arch/sparc/mm/init_64.c | 23 ++++++-----------------
1 file changed, 6 insertions(+), 17 deletions(-)
diff --git a/arch/sparc/mm/init_64.c b/arch/sparc/mm/init_64.c
index 12dbba85a2e2..a603d2c9087d 100644
--- a/arch/sparc/mm/init_64.c
+++ b/arch/sparc/mm/init_64.c
@@ -2611,30 +2611,19 @@ int __meminit vmemmap_populate(unsigned long vstart, unsigned long vend,
vstart = vstart & PMD_MASK;
vend = ALIGN(vend, PMD_SIZE);
for (; vstart < vend; vstart += PMD_SIZE) {
- pgd_t *pgd = pgd_offset_k(vstart);
+ pgd_t *pgd = vmemmap_pgd_populate(vstart, node);
unsigned long pte;
pud_t *pud;
pmd_t *pmd;
- if (pgd_none(*pgd)) {
- pud_t *new = vmemmap_alloc_block(PAGE_SIZE, node);
+ if (!pgd)
+ return -ENOMEM;
- if (!new)
- return -ENOMEM;
- pgd_populate(&init_mm, pgd, new);
- }
-
- pud = pud_offset(pgd, vstart);
- if (pud_none(*pud)) {
- pmd_t *new = vmemmap_alloc_block(PAGE_SIZE, node);
-
- if (!new)
- return -ENOMEM;
- pud_populate(&init_mm, pud, new);
- }
+ pud = vmemmap_pud_populate(pgd, vstart, node);
+ if (!pud)
+ return -ENOMEM;
pmd = pmd_offset(pud, vstart);
-
pte = pmd_val(*pmd);
if (!(pte & _PAGE_VALID)) {
void *block = vmemmap_alloc_block(PMD_SIZE, node);
--
2.14.1
^ permalink raw reply related [flat|nested] 79+ messages in thread
* [PATCH v7 05/11] mm: defining memblock_virt_alloc_try_nid_raw
2017-08-29 2:02 ` Pavel Tatashin
(?)
(?)
@ 2017-08-29 2:02 ` Pavel Tatashin
-1 siblings, 0 replies; 79+ messages in thread
From: Pavel Tatashin @ 2017-08-29 2:02 UTC (permalink / raw)
To: linux-kernel, sparclinux, linux-mm, linuxppc-dev, linux-s390,
linux-arm-kernel, x86, kasan-dev, borntraeger, heiko.carstens,
davem, willy, mhocko, ard.biesheuvel, will.deacon,
catalin.marinas, sam, mgorman, Steven.Sistare, daniel.m.jordan,
bob.picco
* A new variant of memblock_virt_alloc_* allocations:
memblock_virt_alloc_try_nid_raw()
- Does not zero the allocated memory
- Does not panic if request cannot be satisfied
* optimize early system hash allocations
Clients can call alloc_large_system_hash() with flag: HASH_ZERO to specify
that memory that was allocated for system hash needs to be zeroed,
otherwise the memory does not need to be zeroed, and client will initialize
it.
If memory does not need to be zero'd, call the new
memblock_virt_alloc_raw() interface, and thus improve the boot performance.
* debug for raw alloctor
When CONFIG_DEBUG_VM is enabled, this patch sets all the memory that is
returned by memblock_virt_alloc_try_nid_raw() to ones to ensure that no
places excpect zeroed memory.
Signed-off-by: Pavel Tatashin <pasha.tatashin@oracle.com>
Reviewed-by: Steven Sistare <steven.sistare@oracle.com>
Reviewed-by: Daniel Jordan <daniel.m.jordan@oracle.com>
Reviewed-by: Bob Picco <bob.picco@oracle.com>
Acked-by: Michal Hocko <mhocko@suse.com>
---
include/linux/bootmem.h | 27 ++++++++++++++++++++++
mm/memblock.c | 60 +++++++++++++++++++++++++++++++++++++++++++------
mm/page_alloc.c | 15 ++++++-------
3 files changed, 87 insertions(+), 15 deletions(-)
diff --git a/include/linux/bootmem.h b/include/linux/bootmem.h
index e223d91b6439..ea30b3987282 100644
--- a/include/linux/bootmem.h
+++ b/include/linux/bootmem.h
@@ -160,6 +160,9 @@ extern void *__alloc_bootmem_low_node(pg_data_t *pgdat,
#define BOOTMEM_ALLOC_ANYWHERE (~(phys_addr_t)0)
/* FIXME: Move to memblock.h at a point where we remove nobootmem.c */
+void *memblock_virt_alloc_try_nid_raw(phys_addr_t size, phys_addr_t align,
+ phys_addr_t min_addr,
+ phys_addr_t max_addr, int nid);
void *memblock_virt_alloc_try_nid_nopanic(phys_addr_t size,
phys_addr_t align, phys_addr_t min_addr,
phys_addr_t max_addr, int nid);
@@ -176,6 +179,14 @@ static inline void * __init memblock_virt_alloc(
NUMA_NO_NODE);
}
+static inline void * __init memblock_virt_alloc_raw(
+ phys_addr_t size, phys_addr_t align)
+{
+ return memblock_virt_alloc_try_nid_raw(size, align, BOOTMEM_LOW_LIMIT,
+ BOOTMEM_ALLOC_ACCESSIBLE,
+ NUMA_NO_NODE);
+}
+
static inline void * __init memblock_virt_alloc_nopanic(
phys_addr_t size, phys_addr_t align)
{
@@ -257,6 +268,14 @@ static inline void * __init memblock_virt_alloc(
return __alloc_bootmem(size, align, BOOTMEM_LOW_LIMIT);
}
+static inline void * __init memblock_virt_alloc_raw(
+ phys_addr_t size, phys_addr_t align)
+{
+ if (!align)
+ align = SMP_CACHE_BYTES;
+ return __alloc_bootmem_nopanic(size, align, BOOTMEM_LOW_LIMIT);
+}
+
static inline void * __init memblock_virt_alloc_nopanic(
phys_addr_t size, phys_addr_t align)
{
@@ -309,6 +328,14 @@ static inline void * __init memblock_virt_alloc_try_nid(phys_addr_t size,
min_addr);
}
+static inline void * __init memblock_virt_alloc_try_nid_raw(
+ phys_addr_t size, phys_addr_t align,
+ phys_addr_t min_addr, phys_addr_t max_addr, int nid)
+{
+ return ___alloc_bootmem_node_nopanic(NODE_DATA(nid), size, align,
+ min_addr, max_addr);
+}
+
static inline void * __init memblock_virt_alloc_try_nid_nopanic(
phys_addr_t size, phys_addr_t align,
phys_addr_t min_addr, phys_addr_t max_addr, int nid)
diff --git a/mm/memblock.c b/mm/memblock.c
index 91205780e6b1..1f299fb1eb08 100644
--- a/mm/memblock.c
+++ b/mm/memblock.c
@@ -1327,7 +1327,6 @@ static void * __init memblock_virt_alloc_internal(
return NULL;
done:
ptr = phys_to_virt(alloc);
- memset(ptr, 0, size);
/*
* The min_count is set to 0 so that bootmem allocated blocks
@@ -1340,6 +1339,45 @@ static void * __init memblock_virt_alloc_internal(
return ptr;
}
+/**
+ * memblock_virt_alloc_try_nid_raw - allocate boot memory block without zeroing
+ * memory and without panicking
+ * @size: size of memory block to be allocated in bytes
+ * @align: alignment of the region and block's size
+ * @min_addr: the lower bound of the memory region from where the allocation
+ * is preferred (phys address)
+ * @max_addr: the upper bound of the memory region from where the allocation
+ * is preferred (phys address), or %BOOTMEM_ALLOC_ACCESSIBLE to
+ * allocate only from memory limited by memblock.current_limit value
+ * @nid: nid of the free area to find, %NUMA_NO_NODE for any node
+ *
+ * Public function, provides additional debug information (including caller
+ * info), if enabled. Does not zero allocated memory, does not panic if request
+ * cannot be satisfied.
+ *
+ * RETURNS:
+ * Virtual address of allocated memory block on success, NULL on failure.
+ */
+void * __init memblock_virt_alloc_try_nid_raw(
+ phys_addr_t size, phys_addr_t align,
+ phys_addr_t min_addr, phys_addr_t max_addr,
+ int nid)
+{
+ void *ptr;
+
+ memblock_dbg("%s: %llu bytes align=0x%llx nid=%d from=0x%llx max_addr=0x%llx %pF\n",
+ __func__, (u64)size, (u64)align, nid, (u64)min_addr,
+ (u64)max_addr, (void *)_RET_IP_);
+
+ ptr = memblock_virt_alloc_internal(size, align,
+ min_addr, max_addr, nid);
+#ifdef CONFIG_DEBUG_VM
+ if (ptr && size > 0)
+ memset(ptr, 0xff, size);
+#endif
+ return ptr;
+}
+
/**
* memblock_virt_alloc_try_nid_nopanic - allocate boot memory block
* @size: size of memory block to be allocated in bytes
@@ -1351,8 +1389,8 @@ static void * __init memblock_virt_alloc_internal(
* allocate only from memory limited by memblock.current_limit value
* @nid: nid of the free area to find, %NUMA_NO_NODE for any node
*
- * Public version of _memblock_virt_alloc_try_nid_nopanic() which provides
- * additional debug information (including caller info), if enabled.
+ * Public function, provides additional debug information (including caller
+ * info), if enabled. This function zeroes the allocated memory.
*
* RETURNS:
* Virtual address of allocated memory block on success, NULL on failure.
@@ -1362,11 +1400,17 @@ void * __init memblock_virt_alloc_try_nid_nopanic(
phys_addr_t min_addr, phys_addr_t max_addr,
int nid)
{
+ void *ptr;
+
memblock_dbg("%s: %llu bytes align=0x%llx nid=%d from=0x%llx max_addr=0x%llx %pF\n",
__func__, (u64)size, (u64)align, nid, (u64)min_addr,
(u64)max_addr, (void *)_RET_IP_);
- return memblock_virt_alloc_internal(size, align, min_addr,
- max_addr, nid);
+
+ ptr = memblock_virt_alloc_internal(size, align,
+ min_addr, max_addr, nid);
+ if (ptr)
+ memset(ptr, 0, size);
+ return ptr;
}
/**
@@ -1380,7 +1424,7 @@ void * __init memblock_virt_alloc_try_nid_nopanic(
* allocate only from memory limited by memblock.current_limit value
* @nid: nid of the free area to find, %NUMA_NO_NODE for any node
*
- * Public panicking version of _memblock_virt_alloc_try_nid_nopanic()
+ * Public panicking version of memblock_virt_alloc_try_nid_nopanic()
* which provides debug information (including caller info), if enabled,
* and panics if the request can not be satisfied.
*
@@ -1399,8 +1443,10 @@ void * __init memblock_virt_alloc_try_nid(
(u64)max_addr, (void *)_RET_IP_);
ptr = memblock_virt_alloc_internal(size, align,
min_addr, max_addr, nid);
- if (ptr)
+ if (ptr) {
+ memset(ptr, 0, size);
return ptr;
+ }
panic("%s: Failed to allocate %llu bytes align=0x%llx nid=%d from=0x%llx max_addr=0x%llx\n",
__func__, (u64)size, (u64)align, nid, (u64)min_addr,
diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index c170ac569aec..8293815ca85d 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -7356,18 +7356,17 @@ void *__init alloc_large_system_hash(const char *tablename,
log2qty = ilog2(numentries);
- /*
- * memblock allocator returns zeroed memory already, so HASH_ZERO is
- * currently not used when HASH_EARLY is specified.
- */
gfp_flags = (flags & HASH_ZERO) ? GFP_ATOMIC | __GFP_ZERO : GFP_ATOMIC;
do {
size = bucketsize << log2qty;
- if (flags & HASH_EARLY)
- table = memblock_virt_alloc_nopanic(size, 0);
- else if (hashdist)
+ if (flags & HASH_EARLY) {
+ if (flags & HASH_ZERO)
+ table = memblock_virt_alloc_nopanic(size, 0);
+ else
+ table = memblock_virt_alloc_raw(size, 0);
+ } else if (hashdist) {
table = __vmalloc(size, gfp_flags, PAGE_KERNEL);
- else {
+ } else {
/*
* If bucketsize is not a power-of-two, we may free
* some pages at the end of hash table which
--
2.14.1
^ permalink raw reply related [flat|nested] 79+ messages in thread
* [PATCH v7 05/11] mm: defining memblock_virt_alloc_try_nid_raw
@ 2017-08-29 2:02 ` Pavel Tatashin
0 siblings, 0 replies; 79+ messages in thread
From: Pavel Tatashin @ 2017-08-29 2:02 UTC (permalink / raw)
To: linux-arm-kernel
* A new variant of memblock_virt_alloc_* allocations:
memblock_virt_alloc_try_nid_raw()
- Does not zero the allocated memory
- Does not panic if request cannot be satisfied
* optimize early system hash allocations
Clients can call alloc_large_system_hash() with flag: HASH_ZERO to specify
that memory that was allocated for system hash needs to be zeroed,
otherwise the memory does not need to be zeroed, and client will initialize
it.
If memory does not need to be zero'd, call the new
memblock_virt_alloc_raw() interface, and thus improve the boot performance.
* debug for raw alloctor
When CONFIG_DEBUG_VM is enabled, this patch sets all the memory that is
returned by memblock_virt_alloc_try_nid_raw() to ones to ensure that no
places excpect zeroed memory.
Signed-off-by: Pavel Tatashin <pasha.tatashin@oracle.com>
Reviewed-by: Steven Sistare <steven.sistare@oracle.com>
Reviewed-by: Daniel Jordan <daniel.m.jordan@oracle.com>
Reviewed-by: Bob Picco <bob.picco@oracle.com>
Acked-by: Michal Hocko <mhocko@suse.com>
---
include/linux/bootmem.h | 27 ++++++++++++++++++++++
mm/memblock.c | 60 +++++++++++++++++++++++++++++++++++++++++++------
mm/page_alloc.c | 15 ++++++-------
3 files changed, 87 insertions(+), 15 deletions(-)
diff --git a/include/linux/bootmem.h b/include/linux/bootmem.h
index e223d91b6439..ea30b3987282 100644
--- a/include/linux/bootmem.h
+++ b/include/linux/bootmem.h
@@ -160,6 +160,9 @@ extern void *__alloc_bootmem_low_node(pg_data_t *pgdat,
#define BOOTMEM_ALLOC_ANYWHERE (~(phys_addr_t)0)
/* FIXME: Move to memblock.h at a point where we remove nobootmem.c */
+void *memblock_virt_alloc_try_nid_raw(phys_addr_t size, phys_addr_t align,
+ phys_addr_t min_addr,
+ phys_addr_t max_addr, int nid);
void *memblock_virt_alloc_try_nid_nopanic(phys_addr_t size,
phys_addr_t align, phys_addr_t min_addr,
phys_addr_t max_addr, int nid);
@@ -176,6 +179,14 @@ static inline void * __init memblock_virt_alloc(
NUMA_NO_NODE);
}
+static inline void * __init memblock_virt_alloc_raw(
+ phys_addr_t size, phys_addr_t align)
+{
+ return memblock_virt_alloc_try_nid_raw(size, align, BOOTMEM_LOW_LIMIT,
+ BOOTMEM_ALLOC_ACCESSIBLE,
+ NUMA_NO_NODE);
+}
+
static inline void * __init memblock_virt_alloc_nopanic(
phys_addr_t size, phys_addr_t align)
{
@@ -257,6 +268,14 @@ static inline void * __init memblock_virt_alloc(
return __alloc_bootmem(size, align, BOOTMEM_LOW_LIMIT);
}
+static inline void * __init memblock_virt_alloc_raw(
+ phys_addr_t size, phys_addr_t align)
+{
+ if (!align)
+ align = SMP_CACHE_BYTES;
+ return __alloc_bootmem_nopanic(size, align, BOOTMEM_LOW_LIMIT);
+}
+
static inline void * __init memblock_virt_alloc_nopanic(
phys_addr_t size, phys_addr_t align)
{
@@ -309,6 +328,14 @@ static inline void * __init memblock_virt_alloc_try_nid(phys_addr_t size,
min_addr);
}
+static inline void * __init memblock_virt_alloc_try_nid_raw(
+ phys_addr_t size, phys_addr_t align,
+ phys_addr_t min_addr, phys_addr_t max_addr, int nid)
+{
+ return ___alloc_bootmem_node_nopanic(NODE_DATA(nid), size, align,
+ min_addr, max_addr);
+}
+
static inline void * __init memblock_virt_alloc_try_nid_nopanic(
phys_addr_t size, phys_addr_t align,
phys_addr_t min_addr, phys_addr_t max_addr, int nid)
diff --git a/mm/memblock.c b/mm/memblock.c
index 91205780e6b1..1f299fb1eb08 100644
--- a/mm/memblock.c
+++ b/mm/memblock.c
@@ -1327,7 +1327,6 @@ static void * __init memblock_virt_alloc_internal(
return NULL;
done:
ptr = phys_to_virt(alloc);
- memset(ptr, 0, size);
/*
* The min_count is set to 0 so that bootmem allocated blocks
@@ -1340,6 +1339,45 @@ static void * __init memblock_virt_alloc_internal(
return ptr;
}
+/**
+ * memblock_virt_alloc_try_nid_raw - allocate boot memory block without zeroing
+ * memory and without panicking
+ * @size: size of memory block to be allocated in bytes
+ * @align: alignment of the region and block's size
+ * @min_addr: the lower bound of the memory region from where the allocation
+ * is preferred (phys address)
+ * @max_addr: the upper bound of the memory region from where the allocation
+ * is preferred (phys address), or %BOOTMEM_ALLOC_ACCESSIBLE to
+ * allocate only from memory limited by memblock.current_limit value
+ * @nid: nid of the free area to find, %NUMA_NO_NODE for any node
+ *
+ * Public function, provides additional debug information (including caller
+ * info), if enabled. Does not zero allocated memory, does not panic if request
+ * cannot be satisfied.
+ *
+ * RETURNS:
+ * Virtual address of allocated memory block on success, NULL on failure.
+ */
+void * __init memblock_virt_alloc_try_nid_raw(
+ phys_addr_t size, phys_addr_t align,
+ phys_addr_t min_addr, phys_addr_t max_addr,
+ int nid)
+{
+ void *ptr;
+
+ memblock_dbg("%s: %llu bytes align=0x%llx nid=%d from=0x%llx max_addr=0x%llx %pF\n",
+ __func__, (u64)size, (u64)align, nid, (u64)min_addr,
+ (u64)max_addr, (void *)_RET_IP_);
+
+ ptr = memblock_virt_alloc_internal(size, align,
+ min_addr, max_addr, nid);
+#ifdef CONFIG_DEBUG_VM
+ if (ptr && size > 0)
+ memset(ptr, 0xff, size);
+#endif
+ return ptr;
+}
+
/**
* memblock_virt_alloc_try_nid_nopanic - allocate boot memory block
* @size: size of memory block to be allocated in bytes
@@ -1351,8 +1389,8 @@ static void * __init memblock_virt_alloc_internal(
* allocate only from memory limited by memblock.current_limit value
* @nid: nid of the free area to find, %NUMA_NO_NODE for any node
*
- * Public version of _memblock_virt_alloc_try_nid_nopanic() which provides
- * additional debug information (including caller info), if enabled.
+ * Public function, provides additional debug information (including caller
+ * info), if enabled. This function zeroes the allocated memory.
*
* RETURNS:
* Virtual address of allocated memory block on success, NULL on failure.
@@ -1362,11 +1400,17 @@ void * __init memblock_virt_alloc_try_nid_nopanic(
phys_addr_t min_addr, phys_addr_t max_addr,
int nid)
{
+ void *ptr;
+
memblock_dbg("%s: %llu bytes align=0x%llx nid=%d from=0x%llx max_addr=0x%llx %pF\n",
__func__, (u64)size, (u64)align, nid, (u64)min_addr,
(u64)max_addr, (void *)_RET_IP_);
- return memblock_virt_alloc_internal(size, align, min_addr,
- max_addr, nid);
+
+ ptr = memblock_virt_alloc_internal(size, align,
+ min_addr, max_addr, nid);
+ if (ptr)
+ memset(ptr, 0, size);
+ return ptr;
}
/**
@@ -1380,7 +1424,7 @@ void * __init memblock_virt_alloc_try_nid_nopanic(
* allocate only from memory limited by memblock.current_limit value
* @nid: nid of the free area to find, %NUMA_NO_NODE for any node
*
- * Public panicking version of _memblock_virt_alloc_try_nid_nopanic()
+ * Public panicking version of memblock_virt_alloc_try_nid_nopanic()
* which provides debug information (including caller info), if enabled,
* and panics if the request can not be satisfied.
*
@@ -1399,8 +1443,10 @@ void * __init memblock_virt_alloc_try_nid(
(u64)max_addr, (void *)_RET_IP_);
ptr = memblock_virt_alloc_internal(size, align,
min_addr, max_addr, nid);
- if (ptr)
+ if (ptr) {
+ memset(ptr, 0, size);
return ptr;
+ }
panic("%s: Failed to allocate %llu bytes align=0x%llx nid=%d from=0x%llx max_addr=0x%llx\n",
__func__, (u64)size, (u64)align, nid, (u64)min_addr,
diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index c170ac569aec..8293815ca85d 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -7356,18 +7356,17 @@ void *__init alloc_large_system_hash(const char *tablename,
log2qty = ilog2(numentries);
- /*
- * memblock allocator returns zeroed memory already, so HASH_ZERO is
- * currently not used when HASH_EARLY is specified.
- */
gfp_flags = (flags & HASH_ZERO) ? GFP_ATOMIC | __GFP_ZERO : GFP_ATOMIC;
do {
size = bucketsize << log2qty;
- if (flags & HASH_EARLY)
- table = memblock_virt_alloc_nopanic(size, 0);
- else if (hashdist)
+ if (flags & HASH_EARLY) {
+ if (flags & HASH_ZERO)
+ table = memblock_virt_alloc_nopanic(size, 0);
+ else
+ table = memblock_virt_alloc_raw(size, 0);
+ } else if (hashdist) {
table = __vmalloc(size, gfp_flags, PAGE_KERNEL);
- else {
+ } else {
/*
* If bucketsize is not a power-of-two, we may free
* some pages at the end of hash table which
--
2.14.1
^ permalink raw reply related [flat|nested] 79+ messages in thread
* [PATCH v7 05/11] mm: defining memblock_virt_alloc_try_nid_raw
@ 2017-08-29 2:02 ` Pavel Tatashin
0 siblings, 0 replies; 79+ messages in thread
From: Pavel Tatashin @ 2017-08-29 2:02 UTC (permalink / raw)
To: linux-kernel, sparclinux, linux-mm, linuxppc-dev, linux-s390,
linux-arm-kernel, x86, kasan-dev, borntraeger, heiko.carstens,
davem, willy, mhocko, ard.biesheuvel, will.deacon,
catalin.marinas, sam, mgorman, Steven.Sistare, daniel.m.jordan,
bob.picco
* A new variant of memblock_virt_alloc_* allocations:
memblock_virt_alloc_try_nid_raw()
- Does not zero the allocated memory
- Does not panic if request cannot be satisfied
* optimize early system hash allocations
Clients can call alloc_large_system_hash() with flag: HASH_ZERO to specify
that memory that was allocated for system hash needs to be zeroed,
otherwise the memory does not need to be zeroed, and client will initialize
it.
If memory does not need to be zero'd, call the new
memblock_virt_alloc_raw() interface, and thus improve the boot performance.
* debug for raw alloctor
When CONFIG_DEBUG_VM is enabled, this patch sets all the memory that is
returned by memblock_virt_alloc_try_nid_raw() to ones to ensure that no
places excpect zeroed memory.
Signed-off-by: Pavel Tatashin <pasha.tatashin@oracle.com>
Reviewed-by: Steven Sistare <steven.sistare@oracle.com>
Reviewed-by: Daniel Jordan <daniel.m.jordan@oracle.com>
Reviewed-by: Bob Picco <bob.picco@oracle.com>
Acked-by: Michal Hocko <mhocko@suse.com>
---
include/linux/bootmem.h | 27 ++++++++++++++++++++++
mm/memblock.c | 60 +++++++++++++++++++++++++++++++++++++++++++------
mm/page_alloc.c | 15 ++++++-------
3 files changed, 87 insertions(+), 15 deletions(-)
diff --git a/include/linux/bootmem.h b/include/linux/bootmem.h
index e223d91b6439..ea30b3987282 100644
--- a/include/linux/bootmem.h
+++ b/include/linux/bootmem.h
@@ -160,6 +160,9 @@ extern void *__alloc_bootmem_low_node(pg_data_t *pgdat,
#define BOOTMEM_ALLOC_ANYWHERE (~(phys_addr_t)0)
/* FIXME: Move to memblock.h at a point where we remove nobootmem.c */
+void *memblock_virt_alloc_try_nid_raw(phys_addr_t size, phys_addr_t align,
+ phys_addr_t min_addr,
+ phys_addr_t max_addr, int nid);
void *memblock_virt_alloc_try_nid_nopanic(phys_addr_t size,
phys_addr_t align, phys_addr_t min_addr,
phys_addr_t max_addr, int nid);
@@ -176,6 +179,14 @@ static inline void * __init memblock_virt_alloc(
NUMA_NO_NODE);
}
+static inline void * __init memblock_virt_alloc_raw(
+ phys_addr_t size, phys_addr_t align)
+{
+ return memblock_virt_alloc_try_nid_raw(size, align, BOOTMEM_LOW_LIMIT,
+ BOOTMEM_ALLOC_ACCESSIBLE,
+ NUMA_NO_NODE);
+}
+
static inline void * __init memblock_virt_alloc_nopanic(
phys_addr_t size, phys_addr_t align)
{
@@ -257,6 +268,14 @@ static inline void * __init memblock_virt_alloc(
return __alloc_bootmem(size, align, BOOTMEM_LOW_LIMIT);
}
+static inline void * __init memblock_virt_alloc_raw(
+ phys_addr_t size, phys_addr_t align)
+{
+ if (!align)
+ align = SMP_CACHE_BYTES;
+ return __alloc_bootmem_nopanic(size, align, BOOTMEM_LOW_LIMIT);
+}
+
static inline void * __init memblock_virt_alloc_nopanic(
phys_addr_t size, phys_addr_t align)
{
@@ -309,6 +328,14 @@ static inline void * __init memblock_virt_alloc_try_nid(phys_addr_t size,
min_addr);
}
+static inline void * __init memblock_virt_alloc_try_nid_raw(
+ phys_addr_t size, phys_addr_t align,
+ phys_addr_t min_addr, phys_addr_t max_addr, int nid)
+{
+ return ___alloc_bootmem_node_nopanic(NODE_DATA(nid), size, align,
+ min_addr, max_addr);
+}
+
static inline void * __init memblock_virt_alloc_try_nid_nopanic(
phys_addr_t size, phys_addr_t align,
phys_addr_t min_addr, phys_addr_t max_addr, int nid)
diff --git a/mm/memblock.c b/mm/memblock.c
index 91205780e6b1..1f299fb1eb08 100644
--- a/mm/memblock.c
+++ b/mm/memblock.c
@@ -1327,7 +1327,6 @@ static void * __init memblock_virt_alloc_internal(
return NULL;
done:
ptr = phys_to_virt(alloc);
- memset(ptr, 0, size);
/*
* The min_count is set to 0 so that bootmem allocated blocks
@@ -1340,6 +1339,45 @@ static void * __init memblock_virt_alloc_internal(
return ptr;
}
+/**
+ * memblock_virt_alloc_try_nid_raw - allocate boot memory block without zeroing
+ * memory and without panicking
+ * @size: size of memory block to be allocated in bytes
+ * @align: alignment of the region and block's size
+ * @min_addr: the lower bound of the memory region from where the allocation
+ * is preferred (phys address)
+ * @max_addr: the upper bound of the memory region from where the allocation
+ * is preferred (phys address), or %BOOTMEM_ALLOC_ACCESSIBLE to
+ * allocate only from memory limited by memblock.current_limit value
+ * @nid: nid of the free area to find, %NUMA_NO_NODE for any node
+ *
+ * Public function, provides additional debug information (including caller
+ * info), if enabled. Does not zero allocated memory, does not panic if request
+ * cannot be satisfied.
+ *
+ * RETURNS:
+ * Virtual address of allocated memory block on success, NULL on failure.
+ */
+void * __init memblock_virt_alloc_try_nid_raw(
+ phys_addr_t size, phys_addr_t align,
+ phys_addr_t min_addr, phys_addr_t max_addr,
+ int nid)
+{
+ void *ptr;
+
+ memblock_dbg("%s: %llu bytes align=0x%llx nid=%d from=0x%llx max_addr=0x%llx %pF\n",
+ __func__, (u64)size, (u64)align, nid, (u64)min_addr,
+ (u64)max_addr, (void *)_RET_IP_);
+
+ ptr = memblock_virt_alloc_internal(size, align,
+ min_addr, max_addr, nid);
+#ifdef CONFIG_DEBUG_VM
+ if (ptr && size > 0)
+ memset(ptr, 0xff, size);
+#endif
+ return ptr;
+}
+
/**
* memblock_virt_alloc_try_nid_nopanic - allocate boot memory block
* @size: size of memory block to be allocated in bytes
@@ -1351,8 +1389,8 @@ static void * __init memblock_virt_alloc_internal(
* allocate only from memory limited by memblock.current_limit value
* @nid: nid of the free area to find, %NUMA_NO_NODE for any node
*
- * Public version of _memblock_virt_alloc_try_nid_nopanic() which provides
- * additional debug information (including caller info), if enabled.
+ * Public function, provides additional debug information (including caller
+ * info), if enabled. This function zeroes the allocated memory.
*
* RETURNS:
* Virtual address of allocated memory block on success, NULL on failure.
@@ -1362,11 +1400,17 @@ void * __init memblock_virt_alloc_try_nid_nopanic(
phys_addr_t min_addr, phys_addr_t max_addr,
int nid)
{
+ void *ptr;
+
memblock_dbg("%s: %llu bytes align=0x%llx nid=%d from=0x%llx max_addr=0x%llx %pF\n",
__func__, (u64)size, (u64)align, nid, (u64)min_addr,
(u64)max_addr, (void *)_RET_IP_);
- return memblock_virt_alloc_internal(size, align, min_addr,
- max_addr, nid);
+
+ ptr = memblock_virt_alloc_internal(size, align,
+ min_addr, max_addr, nid);
+ if (ptr)
+ memset(ptr, 0, size);
+ return ptr;
}
/**
@@ -1380,7 +1424,7 @@ void * __init memblock_virt_alloc_try_nid_nopanic(
* allocate only from memory limited by memblock.current_limit value
* @nid: nid of the free area to find, %NUMA_NO_NODE for any node
*
- * Public panicking version of _memblock_virt_alloc_try_nid_nopanic()
+ * Public panicking version of memblock_virt_alloc_try_nid_nopanic()
* which provides debug information (including caller info), if enabled,
* and panics if the request can not be satisfied.
*
@@ -1399,8 +1443,10 @@ void * __init memblock_virt_alloc_try_nid(
(u64)max_addr, (void *)_RET_IP_);
ptr = memblock_virt_alloc_internal(size, align,
min_addr, max_addr, nid);
- if (ptr)
+ if (ptr) {
+ memset(ptr, 0, size);
return ptr;
+ }
panic("%s: Failed to allocate %llu bytes align=0x%llx nid=%d from=0x%llx max_addr=0x%llx\n",
__func__, (u64)size, (u64)align, nid, (u64)min_addr,
diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index c170ac569aec..8293815ca85d 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -7356,18 +7356,17 @@ void *__init alloc_large_system_hash(const char *tablename,
log2qty = ilog2(numentries);
- /*
- * memblock allocator returns zeroed memory already, so HASH_ZERO is
- * currently not used when HASH_EARLY is specified.
- */
gfp_flags = (flags & HASH_ZERO) ? GFP_ATOMIC | __GFP_ZERO : GFP_ATOMIC;
do {
size = bucketsize << log2qty;
- if (flags & HASH_EARLY)
- table = memblock_virt_alloc_nopanic(size, 0);
- else if (hashdist)
+ if (flags & HASH_EARLY) {
+ if (flags & HASH_ZERO)
+ table = memblock_virt_alloc_nopanic(size, 0);
+ else
+ table = memblock_virt_alloc_raw(size, 0);
+ } else if (hashdist) {
table = __vmalloc(size, gfp_flags, PAGE_KERNEL);
- else {
+ } else {
/*
* If bucketsize is not a power-of-two, we may free
* some pages at the end of hash table which
--
2.14.1
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply related [flat|nested] 79+ messages in thread
* [PATCH v7 05/11] mm: defining memblock_virt_alloc_try_nid_raw
@ 2017-08-29 2:02 ` Pavel Tatashin
0 siblings, 0 replies; 79+ messages in thread
From: Pavel Tatashin @ 2017-08-29 2:02 UTC (permalink / raw)
To: linux-arm-kernel
* A new variant of memblock_virt_alloc_* allocations:
memblock_virt_alloc_try_nid_raw()
- Does not zero the allocated memory
- Does not panic if request cannot be satisfied
* optimize early system hash allocations
Clients can call alloc_large_system_hash() with flag: HASH_ZERO to specify
that memory that was allocated for system hash needs to be zeroed,
otherwise the memory does not need to be zeroed, and client will initialize
it.
If memory does not need to be zero'd, call the new
memblock_virt_alloc_raw() interface, and thus improve the boot performance.
* debug for raw alloctor
When CONFIG_DEBUG_VM is enabled, this patch sets all the memory that is
returned by memblock_virt_alloc_try_nid_raw() to ones to ensure that no
places excpect zeroed memory.
Signed-off-by: Pavel Tatashin <pasha.tatashin@oracle.com>
Reviewed-by: Steven Sistare <steven.sistare@oracle.com>
Reviewed-by: Daniel Jordan <daniel.m.jordan@oracle.com>
Reviewed-by: Bob Picco <bob.picco@oracle.com>
Acked-by: Michal Hocko <mhocko@suse.com>
---
include/linux/bootmem.h | 27 ++++++++++++++++++++++
mm/memblock.c | 60 +++++++++++++++++++++++++++++++++++++++++++------
mm/page_alloc.c | 15 ++++++-------
3 files changed, 87 insertions(+), 15 deletions(-)
diff --git a/include/linux/bootmem.h b/include/linux/bootmem.h
index e223d91b6439..ea30b3987282 100644
--- a/include/linux/bootmem.h
+++ b/include/linux/bootmem.h
@@ -160,6 +160,9 @@ extern void *__alloc_bootmem_low_node(pg_data_t *pgdat,
#define BOOTMEM_ALLOC_ANYWHERE (~(phys_addr_t)0)
/* FIXME: Move to memblock.h at a point where we remove nobootmem.c */
+void *memblock_virt_alloc_try_nid_raw(phys_addr_t size, phys_addr_t align,
+ phys_addr_t min_addr,
+ phys_addr_t max_addr, int nid);
void *memblock_virt_alloc_try_nid_nopanic(phys_addr_t size,
phys_addr_t align, phys_addr_t min_addr,
phys_addr_t max_addr, int nid);
@@ -176,6 +179,14 @@ static inline void * __init memblock_virt_alloc(
NUMA_NO_NODE);
}
+static inline void * __init memblock_virt_alloc_raw(
+ phys_addr_t size, phys_addr_t align)
+{
+ return memblock_virt_alloc_try_nid_raw(size, align, BOOTMEM_LOW_LIMIT,
+ BOOTMEM_ALLOC_ACCESSIBLE,
+ NUMA_NO_NODE);
+}
+
static inline void * __init memblock_virt_alloc_nopanic(
phys_addr_t size, phys_addr_t align)
{
@@ -257,6 +268,14 @@ static inline void * __init memblock_virt_alloc(
return __alloc_bootmem(size, align, BOOTMEM_LOW_LIMIT);
}
+static inline void * __init memblock_virt_alloc_raw(
+ phys_addr_t size, phys_addr_t align)
+{
+ if (!align)
+ align = SMP_CACHE_BYTES;
+ return __alloc_bootmem_nopanic(size, align, BOOTMEM_LOW_LIMIT);
+}
+
static inline void * __init memblock_virt_alloc_nopanic(
phys_addr_t size, phys_addr_t align)
{
@@ -309,6 +328,14 @@ static inline void * __init memblock_virt_alloc_try_nid(phys_addr_t size,
min_addr);
}
+static inline void * __init memblock_virt_alloc_try_nid_raw(
+ phys_addr_t size, phys_addr_t align,
+ phys_addr_t min_addr, phys_addr_t max_addr, int nid)
+{
+ return ___alloc_bootmem_node_nopanic(NODE_DATA(nid), size, align,
+ min_addr, max_addr);
+}
+
static inline void * __init memblock_virt_alloc_try_nid_nopanic(
phys_addr_t size, phys_addr_t align,
phys_addr_t min_addr, phys_addr_t max_addr, int nid)
diff --git a/mm/memblock.c b/mm/memblock.c
index 91205780e6b1..1f299fb1eb08 100644
--- a/mm/memblock.c
+++ b/mm/memblock.c
@@ -1327,7 +1327,6 @@ static void * __init memblock_virt_alloc_internal(
return NULL;
done:
ptr = phys_to_virt(alloc);
- memset(ptr, 0, size);
/*
* The min_count is set to 0 so that bootmem allocated blocks
@@ -1340,6 +1339,45 @@ static void * __init memblock_virt_alloc_internal(
return ptr;
}
+/**
+ * memblock_virt_alloc_try_nid_raw - allocate boot memory block without zeroing
+ * memory and without panicking
+ * @size: size of memory block to be allocated in bytes
+ * @align: alignment of the region and block's size
+ * @min_addr: the lower bound of the memory region from where the allocation
+ * is preferred (phys address)
+ * @max_addr: the upper bound of the memory region from where the allocation
+ * is preferred (phys address), or %BOOTMEM_ALLOC_ACCESSIBLE to
+ * allocate only from memory limited by memblock.current_limit value
+ * @nid: nid of the free area to find, %NUMA_NO_NODE for any node
+ *
+ * Public function, provides additional debug information (including caller
+ * info), if enabled. Does not zero allocated memory, does not panic if request
+ * cannot be satisfied.
+ *
+ * RETURNS:
+ * Virtual address of allocated memory block on success, NULL on failure.
+ */
+void * __init memblock_virt_alloc_try_nid_raw(
+ phys_addr_t size, phys_addr_t align,
+ phys_addr_t min_addr, phys_addr_t max_addr,
+ int nid)
+{
+ void *ptr;
+
+ memblock_dbg("%s: %llu bytes align=0x%llx nid=%d from=0x%llx max_addr=0x%llx %pF\n",
+ __func__, (u64)size, (u64)align, nid, (u64)min_addr,
+ (u64)max_addr, (void *)_RET_IP_);
+
+ ptr = memblock_virt_alloc_internal(size, align,
+ min_addr, max_addr, nid);
+#ifdef CONFIG_DEBUG_VM
+ if (ptr && size > 0)
+ memset(ptr, 0xff, size);
+#endif
+ return ptr;
+}
+
/**
* memblock_virt_alloc_try_nid_nopanic - allocate boot memory block
* @size: size of memory block to be allocated in bytes
@@ -1351,8 +1389,8 @@ static void * __init memblock_virt_alloc_internal(
* allocate only from memory limited by memblock.current_limit value
* @nid: nid of the free area to find, %NUMA_NO_NODE for any node
*
- * Public version of _memblock_virt_alloc_try_nid_nopanic() which provides
- * additional debug information (including caller info), if enabled.
+ * Public function, provides additional debug information (including caller
+ * info), if enabled. This function zeroes the allocated memory.
*
* RETURNS:
* Virtual address of allocated memory block on success, NULL on failure.
@@ -1362,11 +1400,17 @@ void * __init memblock_virt_alloc_try_nid_nopanic(
phys_addr_t min_addr, phys_addr_t max_addr,
int nid)
{
+ void *ptr;
+
memblock_dbg("%s: %llu bytes align=0x%llx nid=%d from=0x%llx max_addr=0x%llx %pF\n",
__func__, (u64)size, (u64)align, nid, (u64)min_addr,
(u64)max_addr, (void *)_RET_IP_);
- return memblock_virt_alloc_internal(size, align, min_addr,
- max_addr, nid);
+
+ ptr = memblock_virt_alloc_internal(size, align,
+ min_addr, max_addr, nid);
+ if (ptr)
+ memset(ptr, 0, size);
+ return ptr;
}
/**
@@ -1380,7 +1424,7 @@ void * __init memblock_virt_alloc_try_nid_nopanic(
* allocate only from memory limited by memblock.current_limit value
* @nid: nid of the free area to find, %NUMA_NO_NODE for any node
*
- * Public panicking version of _memblock_virt_alloc_try_nid_nopanic()
+ * Public panicking version of memblock_virt_alloc_try_nid_nopanic()
* which provides debug information (including caller info), if enabled,
* and panics if the request can not be satisfied.
*
@@ -1399,8 +1443,10 @@ void * __init memblock_virt_alloc_try_nid(
(u64)max_addr, (void *)_RET_IP_);
ptr = memblock_virt_alloc_internal(size, align,
min_addr, max_addr, nid);
- if (ptr)
+ if (ptr) {
+ memset(ptr, 0, size);
return ptr;
+ }
panic("%s: Failed to allocate %llu bytes align=0x%llx nid=%d from=0x%llx max_addr=0x%llx\n",
__func__, (u64)size, (u64)align, nid, (u64)min_addr,
diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index c170ac569aec..8293815ca85d 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -7356,18 +7356,17 @@ void *__init alloc_large_system_hash(const char *tablename,
log2qty = ilog2(numentries);
- /*
- * memblock allocator returns zeroed memory already, so HASH_ZERO is
- * currently not used when HASH_EARLY is specified.
- */
gfp_flags = (flags & HASH_ZERO) ? GFP_ATOMIC | __GFP_ZERO : GFP_ATOMIC;
do {
size = bucketsize << log2qty;
- if (flags & HASH_EARLY)
- table = memblock_virt_alloc_nopanic(size, 0);
- else if (hashdist)
+ if (flags & HASH_EARLY) {
+ if (flags & HASH_ZERO)
+ table = memblock_virt_alloc_nopanic(size, 0);
+ else
+ table = memblock_virt_alloc_raw(size, 0);
+ } else if (hashdist) {
table = __vmalloc(size, gfp_flags, PAGE_KERNEL);
- else {
+ } else {
/*
* If bucketsize is not a power-of-two, we may free
* some pages@the end of hash table which
--
2.14.1
^ permalink raw reply related [flat|nested] 79+ messages in thread
* [PATCH v7 06/11] mm: zero struct pages during initialization
2017-08-29 2:02 ` Pavel Tatashin
(?)
(?)
@ 2017-08-29 2:02 ` Pavel Tatashin
-1 siblings, 0 replies; 79+ messages in thread
From: Pavel Tatashin @ 2017-08-29 2:02 UTC (permalink / raw)
To: linux-kernel, sparclinux, linux-mm, linuxppc-dev, linux-s390,
linux-arm-kernel, x86, kasan-dev, borntraeger, heiko.carstens,
davem, willy, mhocko, ard.biesheuvel, will.deacon,
catalin.marinas, sam, mgorman, Steven.Sistare, daniel.m.jordan,
bob.picco
Add struct page zeroing as a part of initialization of other fields in
__init_single_page().
This single thread performance collected on: Intel(R) Xeon(R) CPU E7-8895
v3 @ 2.60GHz with 1T of memory (268400646 pages in 8 nodes):
BASE FIX
sparse_init 11.244671836s 0.007199623s
zone_sizes_init 4.879775891s 8.355182299s
--------------------------
Total 16.124447727s 8.362381922s
sparse_init is where memory for struct pages is zeroed, and the zeroing
part is moved later in this patch into __init_single_page(), which is
called from zone_sizes_init().
Signed-off-by: Pavel Tatashin <pasha.tatashin@oracle.com>
Reviewed-by: Steven Sistare <steven.sistare@oracle.com>
Reviewed-by: Daniel Jordan <daniel.m.jordan@oracle.com>
Reviewed-by: Bob Picco <bob.picco@oracle.com>
Acked-by: Michal Hocko <mhocko@suse.com>
---
include/linux/mm.h | 9 +++++++++
mm/page_alloc.c | 1 +
2 files changed, 10 insertions(+)
diff --git a/include/linux/mm.h b/include/linux/mm.h
index 46b9ac5e8569..183ac5e733db 100644
--- a/include/linux/mm.h
+++ b/include/linux/mm.h
@@ -93,6 +93,15 @@ extern int mmap_rnd_compat_bits __read_mostly;
#define mm_forbids_zeropage(X) (0)
#endif
+/*
+ * On some architectures it is expensive to call memset() for small sizes.
+ * Those architectures should provide their own implementation of "struct page"
+ * zeroing by defining this macro in <asm/pgtable.h>.
+ */
+#ifndef mm_zero_struct_page
+#define mm_zero_struct_page(pp) ((void)memset((pp), 0, sizeof(struct page)))
+#endif
+
/*
* Default maximum number of active map areas, this limits the number of vmas
* per mm struct. Users can overwrite this number by sysctl but there is a
diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index 8293815ca85d..4d67fe3dd172 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -1169,6 +1169,7 @@ static void free_one_page(struct zone *zone,
static void __meminit __init_single_page(struct page *page, unsigned long pfn,
unsigned long zone, int nid)
{
+ mm_zero_struct_page(page);
set_page_links(page, zone, nid, pfn);
init_page_count(page);
page_mapcount_reset(page);
--
2.14.1
^ permalink raw reply related [flat|nested] 79+ messages in thread
* [PATCH v7 06/11] mm: zero struct pages during initialization
@ 2017-08-29 2:02 ` Pavel Tatashin
0 siblings, 0 replies; 79+ messages in thread
From: Pavel Tatashin @ 2017-08-29 2:02 UTC (permalink / raw)
To: linux-arm-kernel
Add struct page zeroing as a part of initialization of other fields in
__init_single_page().
This single thread performance collected on: Intel(R) Xeon(R) CPU E7-8895
v3 @ 2.60GHz with 1T of memory (268400646 pages in 8 nodes):
BASE FIX
sparse_init 11.244671836s 0.007199623s
zone_sizes_init 4.879775891s 8.355182299s
--------------------------
Total 16.124447727s 8.362381922s
sparse_init is where memory for struct pages is zeroed, and the zeroing
part is moved later in this patch into __init_single_page(), which is
called from zone_sizes_init().
Signed-off-by: Pavel Tatashin <pasha.tatashin@oracle.com>
Reviewed-by: Steven Sistare <steven.sistare@oracle.com>
Reviewed-by: Daniel Jordan <daniel.m.jordan@oracle.com>
Reviewed-by: Bob Picco <bob.picco@oracle.com>
Acked-by: Michal Hocko <mhocko@suse.com>
---
include/linux/mm.h | 9 +++++++++
mm/page_alloc.c | 1 +
2 files changed, 10 insertions(+)
diff --git a/include/linux/mm.h b/include/linux/mm.h
index 46b9ac5e8569..183ac5e733db 100644
--- a/include/linux/mm.h
+++ b/include/linux/mm.h
@@ -93,6 +93,15 @@ extern int mmap_rnd_compat_bits __read_mostly;
#define mm_forbids_zeropage(X) (0)
#endif
+/*
+ * On some architectures it is expensive to call memset() for small sizes.
+ * Those architectures should provide their own implementation of "struct page"
+ * zeroing by defining this macro in <asm/pgtable.h>.
+ */
+#ifndef mm_zero_struct_page
+#define mm_zero_struct_page(pp) ((void)memset((pp), 0, sizeof(struct page)))
+#endif
+
/*
* Default maximum number of active map areas, this limits the number of vmas
* per mm struct. Users can overwrite this number by sysctl but there is a
diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index 8293815ca85d..4d67fe3dd172 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -1169,6 +1169,7 @@ static void free_one_page(struct zone *zone,
static void __meminit __init_single_page(struct page *page, unsigned long pfn,
unsigned long zone, int nid)
{
+ mm_zero_struct_page(page);
set_page_links(page, zone, nid, pfn);
init_page_count(page);
page_mapcount_reset(page);
--
2.14.1
^ permalink raw reply related [flat|nested] 79+ messages in thread
* [PATCH v7 06/11] mm: zero struct pages during initialization
@ 2017-08-29 2:02 ` Pavel Tatashin
0 siblings, 0 replies; 79+ messages in thread
From: Pavel Tatashin @ 2017-08-29 2:02 UTC (permalink / raw)
To: linux-kernel, sparclinux, linux-mm, linuxppc-dev, linux-s390,
linux-arm-kernel, x86, kasan-dev, borntraeger, heiko.carstens,
davem, willy, mhocko, ard.biesheuvel, will.deacon,
catalin.marinas, sam, mgorman, Steven.Sistare, daniel.m.jordan,
bob.picco
Add struct page zeroing as a part of initialization of other fields in
__init_single_page().
This single thread performance collected on: Intel(R) Xeon(R) CPU E7-8895
v3 @ 2.60GHz with 1T of memory (268400646 pages in 8 nodes):
BASE FIX
sparse_init 11.244671836s 0.007199623s
zone_sizes_init 4.879775891s 8.355182299s
--------------------------
Total 16.124447727s 8.362381922s
sparse_init is where memory for struct pages is zeroed, and the zeroing
part is moved later in this patch into __init_single_page(), which is
called from zone_sizes_init().
Signed-off-by: Pavel Tatashin <pasha.tatashin@oracle.com>
Reviewed-by: Steven Sistare <steven.sistare@oracle.com>
Reviewed-by: Daniel Jordan <daniel.m.jordan@oracle.com>
Reviewed-by: Bob Picco <bob.picco@oracle.com>
Acked-by: Michal Hocko <mhocko@suse.com>
---
include/linux/mm.h | 9 +++++++++
mm/page_alloc.c | 1 +
2 files changed, 10 insertions(+)
diff --git a/include/linux/mm.h b/include/linux/mm.h
index 46b9ac5e8569..183ac5e733db 100644
--- a/include/linux/mm.h
+++ b/include/linux/mm.h
@@ -93,6 +93,15 @@ extern int mmap_rnd_compat_bits __read_mostly;
#define mm_forbids_zeropage(X) (0)
#endif
+/*
+ * On some architectures it is expensive to call memset() for small sizes.
+ * Those architectures should provide their own implementation of "struct page"
+ * zeroing by defining this macro in <asm/pgtable.h>.
+ */
+#ifndef mm_zero_struct_page
+#define mm_zero_struct_page(pp) ((void)memset((pp), 0, sizeof(struct page)))
+#endif
+
/*
* Default maximum number of active map areas, this limits the number of vmas
* per mm struct. Users can overwrite this number by sysctl but there is a
diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index 8293815ca85d..4d67fe3dd172 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -1169,6 +1169,7 @@ static void free_one_page(struct zone *zone,
static void __meminit __init_single_page(struct page *page, unsigned long pfn,
unsigned long zone, int nid)
{
+ mm_zero_struct_page(page);
set_page_links(page, zone, nid, pfn);
init_page_count(page);
page_mapcount_reset(page);
--
2.14.1
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply related [flat|nested] 79+ messages in thread
* [PATCH v7 06/11] mm: zero struct pages during initialization
@ 2017-08-29 2:02 ` Pavel Tatashin
0 siblings, 0 replies; 79+ messages in thread
From: Pavel Tatashin @ 2017-08-29 2:02 UTC (permalink / raw)
To: linux-arm-kernel
Add struct page zeroing as a part of initialization of other fields in
__init_single_page().
This single thread performance collected on: Intel(R) Xeon(R) CPU E7-8895
v3 @ 2.60GHz with 1T of memory (268400646 pages in 8 nodes):
BASE FIX
sparse_init 11.244671836s 0.007199623s
zone_sizes_init 4.879775891s 8.355182299s
--------------------------
Total 16.124447727s 8.362381922s
sparse_init is where memory for struct pages is zeroed, and the zeroing
part is moved later in this patch into __init_single_page(), which is
called from zone_sizes_init().
Signed-off-by: Pavel Tatashin <pasha.tatashin@oracle.com>
Reviewed-by: Steven Sistare <steven.sistare@oracle.com>
Reviewed-by: Daniel Jordan <daniel.m.jordan@oracle.com>
Reviewed-by: Bob Picco <bob.picco@oracle.com>
Acked-by: Michal Hocko <mhocko@suse.com>
---
include/linux/mm.h | 9 +++++++++
mm/page_alloc.c | 1 +
2 files changed, 10 insertions(+)
diff --git a/include/linux/mm.h b/include/linux/mm.h
index 46b9ac5e8569..183ac5e733db 100644
--- a/include/linux/mm.h
+++ b/include/linux/mm.h
@@ -93,6 +93,15 @@ extern int mmap_rnd_compat_bits __read_mostly;
#define mm_forbids_zeropage(X) (0)
#endif
+/*
+ * On some architectures it is expensive to call memset() for small sizes.
+ * Those architectures should provide their own implementation of "struct page"
+ * zeroing by defining this macro in <asm/pgtable.h>.
+ */
+#ifndef mm_zero_struct_page
+#define mm_zero_struct_page(pp) ((void)memset((pp), 0, sizeof(struct page)))
+#endif
+
/*
* Default maximum number of active map areas, this limits the number of vmas
* per mm struct. Users can overwrite this number by sysctl but there is a
diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index 8293815ca85d..4d67fe3dd172 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -1169,6 +1169,7 @@ static void free_one_page(struct zone *zone,
static void __meminit __init_single_page(struct page *page, unsigned long pfn,
unsigned long zone, int nid)
{
+ mm_zero_struct_page(page);
set_page_links(page, zone, nid, pfn);
init_page_count(page);
page_mapcount_reset(page);
--
2.14.1
^ permalink raw reply related [flat|nested] 79+ messages in thread
* [PATCH v7 07/11] sparc64: optimized struct page zeroing
2017-08-29 2:02 ` Pavel Tatashin
(?)
(?)
@ 2017-08-29 2:02 ` Pavel Tatashin
-1 siblings, 0 replies; 79+ messages in thread
From: Pavel Tatashin @ 2017-08-29 2:02 UTC (permalink / raw)
To: linux-kernel, sparclinux, linux-mm, linuxppc-dev, linux-s390,
linux-arm-kernel, x86, kasan-dev, borntraeger, heiko.carstens,
davem, willy, mhocko, ard.biesheuvel, will.deacon,
catalin.marinas, sam, mgorman, Steven.Sistare, daniel.m.jordan,
bob.picco
Add an optimized mm_zero_struct_page(), so struct page's are zeroed without
calling memset(). We do eight to ten regular stores based on the size of
struct page. Compiler optimizes out the conditions of switch() statement.
SPARC-M6 with 15T of memory, single thread performance:
BASE FIX OPTIMIZED_FIX
bootmem_init 28.440467985s 2.305674818s 2.305161615s
free_area_init_nodes 202.845901673s 225.343084508s 172.556506560s
--------------------------------------------
Total 231.286369658s 227.648759326s 174.861668175s
BASE: current linux
FIX: This patch series without "optimized struct page zeroing"
OPTIMIZED_FIX: This patch series including the current patch.
bootmem_init() is where memory for struct pages is zeroed during
allocation. Note, about two seconds in this function is a fixed time: it
does not increase as memory is increased.
Signed-off-by: Pavel Tatashin <pasha.tatashin@oracle.com>
Reviewed-by: Steven Sistare <steven.sistare@oracle.com>
Reviewed-by: Daniel Jordan <daniel.m.jordan@oracle.com>
Reviewed-by: Bob Picco <bob.picco@oracle.com>
---
arch/sparc/include/asm/pgtable_64.h | 30 ++++++++++++++++++++++++++++++
1 file changed, 30 insertions(+)
diff --git a/arch/sparc/include/asm/pgtable_64.h b/arch/sparc/include/asm/pgtable_64.h
index 6fbd931f0570..cee5cc7ccc51 100644
--- a/arch/sparc/include/asm/pgtable_64.h
+++ b/arch/sparc/include/asm/pgtable_64.h
@@ -230,6 +230,36 @@ extern unsigned long _PAGE_ALL_SZ_BITS;
extern struct page *mem_map_zero;
#define ZERO_PAGE(vaddr) (mem_map_zero)
+/* This macro must be updated when the size of struct page grows above 80
+ * or reduces below 64.
+ * The idea that compiler optimizes out switch() statement, and only
+ * leaves clrx instructions
+ */
+#define mm_zero_struct_page(pp) do { \
+ unsigned long *_pp = (void *)(pp); \
+ \
+ /* Check that struct page is either 64, 72, or 80 bytes */ \
+ BUILD_BUG_ON(sizeof(struct page) & 7); \
+ BUILD_BUG_ON(sizeof(struct page) < 64); \
+ BUILD_BUG_ON(sizeof(struct page) > 80); \
+ \
+ switch (sizeof(struct page)) { \
+ case 80: \
+ _pp[9] = 0; /* fallthrough */ \
+ case 72: \
+ _pp[8] = 0; /* fallthrough */ \
+ default: \
+ _pp[7] = 0; \
+ _pp[6] = 0; \
+ _pp[5] = 0; \
+ _pp[4] = 0; \
+ _pp[3] = 0; \
+ _pp[2] = 0; \
+ _pp[1] = 0; \
+ _pp[0] = 0; \
+ } \
+} while (0)
+
/* PFNs are real physical page numbers. However, mem_map only begins to record
* per-page information starting at pfn_base. This is to handle systems where
* the first physical page in the machine is at some huge physical address,
--
2.14.1
^ permalink raw reply related [flat|nested] 79+ messages in thread
* [PATCH v7 07/11] sparc64: optimized struct page zeroing
@ 2017-08-29 2:02 ` Pavel Tatashin
0 siblings, 0 replies; 79+ messages in thread
From: Pavel Tatashin @ 2017-08-29 2:02 UTC (permalink / raw)
To: linux-arm-kernel
Add an optimized mm_zero_struct_page(), so struct page's are zeroed without
calling memset(). We do eight to ten regular stores based on the size of
struct page. Compiler optimizes out the conditions of switch() statement.
SPARC-M6 with 15T of memory, single thread performance:
BASE FIX OPTIMIZED_FIX
bootmem_init 28.440467985s 2.305674818s 2.305161615s
free_area_init_nodes 202.845901673s 225.343084508s 172.556506560s
--------------------------------------------
Total 231.286369658s 227.648759326s 174.861668175s
BASE: current linux
FIX: This patch series without "optimized struct page zeroing"
OPTIMIZED_FIX: This patch series including the current patch.
bootmem_init() is where memory for struct pages is zeroed during
allocation. Note, about two seconds in this function is a fixed time: it
does not increase as memory is increased.
Signed-off-by: Pavel Tatashin <pasha.tatashin@oracle.com>
Reviewed-by: Steven Sistare <steven.sistare@oracle.com>
Reviewed-by: Daniel Jordan <daniel.m.jordan@oracle.com>
Reviewed-by: Bob Picco <bob.picco@oracle.com>
---
arch/sparc/include/asm/pgtable_64.h | 30 ++++++++++++++++++++++++++++++
1 file changed, 30 insertions(+)
diff --git a/arch/sparc/include/asm/pgtable_64.h b/arch/sparc/include/asm/pgtable_64.h
index 6fbd931f0570..cee5cc7ccc51 100644
--- a/arch/sparc/include/asm/pgtable_64.h
+++ b/arch/sparc/include/asm/pgtable_64.h
@@ -230,6 +230,36 @@ extern unsigned long _PAGE_ALL_SZ_BITS;
extern struct page *mem_map_zero;
#define ZERO_PAGE(vaddr) (mem_map_zero)
+/* This macro must be updated when the size of struct page grows above 80
+ * or reduces below 64.
+ * The idea that compiler optimizes out switch() statement, and only
+ * leaves clrx instructions
+ */
+#define mm_zero_struct_page(pp) do { \
+ unsigned long *_pp = (void *)(pp); \
+ \
+ /* Check that struct page is either 64, 72, or 80 bytes */ \
+ BUILD_BUG_ON(sizeof(struct page) & 7); \
+ BUILD_BUG_ON(sizeof(struct page) < 64); \
+ BUILD_BUG_ON(sizeof(struct page) > 80); \
+ \
+ switch (sizeof(struct page)) { \
+ case 80: \
+ _pp[9] = 0; /* fallthrough */ \
+ case 72: \
+ _pp[8] = 0; /* fallthrough */ \
+ default: \
+ _pp[7] = 0; \
+ _pp[6] = 0; \
+ _pp[5] = 0; \
+ _pp[4] = 0; \
+ _pp[3] = 0; \
+ _pp[2] = 0; \
+ _pp[1] = 0; \
+ _pp[0] = 0; \
+ } \
+} while (0)
+
/* PFNs are real physical page numbers. However, mem_map only begins to record
* per-page information starting at pfn_base. This is to handle systems where
* the first physical page in the machine is at some huge physical address,
--
2.14.1
^ permalink raw reply related [flat|nested] 79+ messages in thread
* [PATCH v7 07/11] sparc64: optimized struct page zeroing
@ 2017-08-29 2:02 ` Pavel Tatashin
0 siblings, 0 replies; 79+ messages in thread
From: Pavel Tatashin @ 2017-08-29 2:02 UTC (permalink / raw)
To: linux-kernel, sparclinux, linux-mm, linuxppc-dev, linux-s390,
linux-arm-kernel, x86, kasan-dev, borntraeger, heiko.carstens,
davem, willy, mhocko, ard.biesheuvel, will.deacon,
catalin.marinas, sam, mgorman, Steven.Sistare, daniel.m.jordan,
bob.picco
Add an optimized mm_zero_struct_page(), so struct page's are zeroed without
calling memset(). We do eight to ten regular stores based on the size of
struct page. Compiler optimizes out the conditions of switch() statement.
SPARC-M6 with 15T of memory, single thread performance:
BASE FIX OPTIMIZED_FIX
bootmem_init 28.440467985s 2.305674818s 2.305161615s
free_area_init_nodes 202.845901673s 225.343084508s 172.556506560s
--------------------------------------------
Total 231.286369658s 227.648759326s 174.861668175s
BASE: current linux
FIX: This patch series without "optimized struct page zeroing"
OPTIMIZED_FIX: This patch series including the current patch.
bootmem_init() is where memory for struct pages is zeroed during
allocation. Note, about two seconds in this function is a fixed time: it
does not increase as memory is increased.
Signed-off-by: Pavel Tatashin <pasha.tatashin@oracle.com>
Reviewed-by: Steven Sistare <steven.sistare@oracle.com>
Reviewed-by: Daniel Jordan <daniel.m.jordan@oracle.com>
Reviewed-by: Bob Picco <bob.picco@oracle.com>
---
arch/sparc/include/asm/pgtable_64.h | 30 ++++++++++++++++++++++++++++++
1 file changed, 30 insertions(+)
diff --git a/arch/sparc/include/asm/pgtable_64.h b/arch/sparc/include/asm/pgtable_64.h
index 6fbd931f0570..cee5cc7ccc51 100644
--- a/arch/sparc/include/asm/pgtable_64.h
+++ b/arch/sparc/include/asm/pgtable_64.h
@@ -230,6 +230,36 @@ extern unsigned long _PAGE_ALL_SZ_BITS;
extern struct page *mem_map_zero;
#define ZERO_PAGE(vaddr) (mem_map_zero)
+/* This macro must be updated when the size of struct page grows above 80
+ * or reduces below 64.
+ * The idea that compiler optimizes out switch() statement, and only
+ * leaves clrx instructions
+ */
+#define mm_zero_struct_page(pp) do { \
+ unsigned long *_pp = (void *)(pp); \
+ \
+ /* Check that struct page is either 64, 72, or 80 bytes */ \
+ BUILD_BUG_ON(sizeof(struct page) & 7); \
+ BUILD_BUG_ON(sizeof(struct page) < 64); \
+ BUILD_BUG_ON(sizeof(struct page) > 80); \
+ \
+ switch (sizeof(struct page)) { \
+ case 80: \
+ _pp[9] = 0; /* fallthrough */ \
+ case 72: \
+ _pp[8] = 0; /* fallthrough */ \
+ default: \
+ _pp[7] = 0; \
+ _pp[6] = 0; \
+ _pp[5] = 0; \
+ _pp[4] = 0; \
+ _pp[3] = 0; \
+ _pp[2] = 0; \
+ _pp[1] = 0; \
+ _pp[0] = 0; \
+ } \
+} while (0)
+
/* PFNs are real physical page numbers. However, mem_map only begins to record
* per-page information starting at pfn_base. This is to handle systems where
* the first physical page in the machine is at some huge physical address,
--
2.14.1
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply related [flat|nested] 79+ messages in thread
* [PATCH v7 07/11] sparc64: optimized struct page zeroing
@ 2017-08-29 2:02 ` Pavel Tatashin
0 siblings, 0 replies; 79+ messages in thread
From: Pavel Tatashin @ 2017-08-29 2:02 UTC (permalink / raw)
To: linux-arm-kernel
Add an optimized mm_zero_struct_page(), so struct page's are zeroed without
calling memset(). We do eight to ten regular stores based on the size of
struct page. Compiler optimizes out the conditions of switch() statement.
SPARC-M6 with 15T of memory, single thread performance:
BASE FIX OPTIMIZED_FIX
bootmem_init 28.440467985s 2.305674818s 2.305161615s
free_area_init_nodes 202.845901673s 225.343084508s 172.556506560s
--------------------------------------------
Total 231.286369658s 227.648759326s 174.861668175s
BASE: current linux
FIX: This patch series without "optimized struct page zeroing"
OPTIMIZED_FIX: This patch series including the current patch.
bootmem_init() is where memory for struct pages is zeroed during
allocation. Note, about two seconds in this function is a fixed time: it
does not increase as memory is increased.
Signed-off-by: Pavel Tatashin <pasha.tatashin@oracle.com>
Reviewed-by: Steven Sistare <steven.sistare@oracle.com>
Reviewed-by: Daniel Jordan <daniel.m.jordan@oracle.com>
Reviewed-by: Bob Picco <bob.picco@oracle.com>
---
arch/sparc/include/asm/pgtable_64.h | 30 ++++++++++++++++++++++++++++++
1 file changed, 30 insertions(+)
diff --git a/arch/sparc/include/asm/pgtable_64.h b/arch/sparc/include/asm/pgtable_64.h
index 6fbd931f0570..cee5cc7ccc51 100644
--- a/arch/sparc/include/asm/pgtable_64.h
+++ b/arch/sparc/include/asm/pgtable_64.h
@@ -230,6 +230,36 @@ extern unsigned long _PAGE_ALL_SZ_BITS;
extern struct page *mem_map_zero;
#define ZERO_PAGE(vaddr) (mem_map_zero)
+/* This macro must be updated when the size of struct page grows above 80
+ * or reduces below 64.
+ * The idea that compiler optimizes out switch() statement, and only
+ * leaves clrx instructions
+ */
+#define mm_zero_struct_page(pp) do { \
+ unsigned long *_pp = (void *)(pp); \
+ \
+ /* Check that struct page is either 64, 72, or 80 bytes */ \
+ BUILD_BUG_ON(sizeof(struct page) & 7); \
+ BUILD_BUG_ON(sizeof(struct page) < 64); \
+ BUILD_BUG_ON(sizeof(struct page) > 80); \
+ \
+ switch (sizeof(struct page)) { \
+ case 80: \
+ _pp[9] = 0; /* fallthrough */ \
+ case 72: \
+ _pp[8] = 0; /* fallthrough */ \
+ default: \
+ _pp[7] = 0; \
+ _pp[6] = 0; \
+ _pp[5] = 0; \
+ _pp[4] = 0; \
+ _pp[3] = 0; \
+ _pp[2] = 0; \
+ _pp[1] = 0; \
+ _pp[0] = 0; \
+ } \
+} while (0)
+
/* PFNs are real physical page numbers. However, mem_map only begins to record
* per-page information starting at pfn_base. This is to handle systems where
* the first physical page in the machine is at some huge physical address,
--
2.14.1
^ permalink raw reply related [flat|nested] 79+ messages in thread
* [PATCH v7 08/11] mm: zero reserved and unavailable struct pages
2017-08-29 2:02 ` Pavel Tatashin
(?)
(?)
@ 2017-08-29 2:02 ` Pavel Tatashin
-1 siblings, 0 replies; 79+ messages in thread
From: Pavel Tatashin @ 2017-08-29 2:02 UTC (permalink / raw)
To: linux-kernel, sparclinux, linux-mm, linuxppc-dev, linux-s390,
linux-arm-kernel, x86, kasan-dev, borntraeger, heiko.carstens,
davem, willy, mhocko, ard.biesheuvel, will.deacon,
catalin.marinas, sam, mgorman, Steven.Sistare, daniel.m.jordan,
bob.picco
Some memory is reserved but unavailable: not present in memblock.memory
(because not backed by physical pages), but present in memblock.reserved.
Such memory has backing struct pages, but they are not initialized by going
through __init_single_page().
In some cases these struct pages are accessed even if they do not contain
any data. One example is page_to_pfn() might access page->flags if this is
where section information is stored (CONFIG_SPARSEMEM,
SECTION_IN_PAGE_FLAGS).
Since, struct pages are zeroed in __init_single_page(), and not during
allocation time, we must zero such struct pages explicitly.
The patch involves adding a new memblock iterator:
for_each_resv_unavail_range(i, p_start, p_end)
Which iterates through reserved && !memory lists, and we zero struct pages
explicitly by calling mm_zero_struct_page().
Signed-off-by: Pavel Tatashin <pasha.tatashin@oracle.com>
Reviewed-by: Steven Sistare <steven.sistare@oracle.com>
Reviewed-by: Daniel Jordan <daniel.m.jordan@oracle.com>
Reviewed-by: Bob Picco <bob.picco@oracle.com>
---
include/linux/memblock.h | 16 ++++++++++++++++
include/linux/mm.h | 6 ++++++
mm/page_alloc.c | 30 ++++++++++++++++++++++++++++++
3 files changed, 52 insertions(+)
diff --git a/include/linux/memblock.h b/include/linux/memblock.h
index bae11c7e7bf3..bdd4268f9323 100644
--- a/include/linux/memblock.h
+++ b/include/linux/memblock.h
@@ -237,6 +237,22 @@ unsigned long memblock_next_valid_pfn(unsigned long pfn, unsigned long max_pfn);
for_each_mem_range_rev(i, &memblock.memory, &memblock.reserved, \
nid, flags, p_start, p_end, p_nid)
+/**
+ * for_each_resv_unavail_range - iterate through reserved and unavailable memory
+ * @i: u64 used as loop variable
+ * @flags: pick from blocks based on memory attributes
+ * @p_start: ptr to phys_addr_t for start address of the range, can be %NULL
+ * @p_end: ptr to phys_addr_t for end address of the range, can be %NULL
+ *
+ * Walks over unavailabled but reserved (reserved && !memory) areas of memblock.
+ * Available as soon as memblock is initialized.
+ * Note: because this memory does not belong to any physical node, flags and
+ * nid arguments do not make sense and thus not exported as arguments.
+ */
+#define for_each_resv_unavail_range(i, p_start, p_end) \
+ for_each_mem_range(i, &memblock.reserved, &memblock.memory, \
+ NUMA_NO_NODE, MEMBLOCK_NONE, p_start, p_end, NULL)
+
static inline void memblock_set_region_flags(struct memblock_region *r,
unsigned long flags)
{
diff --git a/include/linux/mm.h b/include/linux/mm.h
index 183ac5e733db..0a440ff8f226 100644
--- a/include/linux/mm.h
+++ b/include/linux/mm.h
@@ -1968,6 +1968,12 @@ extern int __meminit __early_pfn_to_nid(unsigned long pfn,
struct mminit_pfnnid_cache *state);
#endif
+#ifdef CONFIG_HAVE_MEMBLOCK
+void zero_resv_unavail(void);
+#else
+static inline void __paginginit zero_resv_unavail(void) {}
+#endif
+
extern void set_dma_reserve(unsigned long new_dma_reserve);
extern void memmap_init_zone(unsigned long, int, unsigned long,
unsigned long, enum memmap_context);
diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index 4d67fe3dd172..484c16fb5f0d 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -6261,6 +6261,34 @@ void __paginginit free_area_init_node(int nid, unsigned long *zones_size,
free_area_init_core(pgdat);
}
+#ifdef CONFIG_HAVE_MEMBLOCK
+/*
+ * Only struct pages that are backed by physical memory are zeroed and
+ * initialized by going through __init_single_page(). But, there are some
+ * struct pages which are reserved in memblock allocator and their fields
+ * may be accessed (for example page_to_pfn() on some configuration accesses
+ * flags). We must explicitly zero those struct pages.
+ */
+void __paginginit zero_resv_unavail(void)
+{
+ phys_addr_t start, end;
+ unsigned long pfn;
+ u64 i, pgcnt;
+
+ /* Loop through ranges that are reserved, but do not have reported
+ * physical memory backing.
+ */
+ pgcnt = 0;
+ for_each_resv_unavail_range(i, &start, &end) {
+ for (pfn = PFN_DOWN(start); pfn < PFN_UP(end); pfn++) {
+ mm_zero_struct_page(pfn_to_page(pfn));
+ pgcnt++;
+ }
+ }
+ pr_info("Reserved but unavailable: %lld pages", pgcnt);
+}
+#endif /* CONFIG_HAVE_MEMBLOCK */
+
#ifdef CONFIG_HAVE_MEMBLOCK_NODE_MAP
#if MAX_NUMNODES > 1
@@ -6684,6 +6712,7 @@ void __init free_area_init_nodes(unsigned long *max_zone_pfn)
node_set_state(nid, N_MEMORY);
check_for_memory(pgdat, nid);
}
+ zero_resv_unavail();
}
static int __init cmdline_parse_core(char *p, unsigned long *core)
@@ -6847,6 +6876,7 @@ void __init free_area_init(unsigned long *zones_size)
{
free_area_init_node(0, zones_size,
__pa(PAGE_OFFSET) >> PAGE_SHIFT, NULL);
+ zero_resv_unavail();
}
static int page_alloc_cpu_dead(unsigned int cpu)
--
2.14.1
^ permalink raw reply related [flat|nested] 79+ messages in thread
* [PATCH v7 08/11] mm: zero reserved and unavailable struct pages
@ 2017-08-29 2:02 ` Pavel Tatashin
0 siblings, 0 replies; 79+ messages in thread
From: Pavel Tatashin @ 2017-08-29 2:02 UTC (permalink / raw)
To: linux-arm-kernel
Some memory is reserved but unavailable: not present in memblock.memory
(because not backed by physical pages), but present in memblock.reserved.
Such memory has backing struct pages, but they are not initialized by going
through __init_single_page().
In some cases these struct pages are accessed even if they do not contain
any data. One example is page_to_pfn() might access page->flags if this is
where section information is stored (CONFIG_SPARSEMEM,
SECTION_IN_PAGE_FLAGS).
Since, struct pages are zeroed in __init_single_page(), and not during
allocation time, we must zero such struct pages explicitly.
The patch involves adding a new memblock iterator:
for_each_resv_unavail_range(i, p_start, p_end)
Which iterates through reserved && !memory lists, and we zero struct pages
explicitly by calling mm_zero_struct_page().
Signed-off-by: Pavel Tatashin <pasha.tatashin@oracle.com>
Reviewed-by: Steven Sistare <steven.sistare@oracle.com>
Reviewed-by: Daniel Jordan <daniel.m.jordan@oracle.com>
Reviewed-by: Bob Picco <bob.picco@oracle.com>
---
include/linux/memblock.h | 16 ++++++++++++++++
include/linux/mm.h | 6 ++++++
mm/page_alloc.c | 30 ++++++++++++++++++++++++++++++
3 files changed, 52 insertions(+)
diff --git a/include/linux/memblock.h b/include/linux/memblock.h
index bae11c7e7bf3..bdd4268f9323 100644
--- a/include/linux/memblock.h
+++ b/include/linux/memblock.h
@@ -237,6 +237,22 @@ unsigned long memblock_next_valid_pfn(unsigned long pfn, unsigned long max_pfn);
for_each_mem_range_rev(i, &memblock.memory, &memblock.reserved, \
nid, flags, p_start, p_end, p_nid)
+/**
+ * for_each_resv_unavail_range - iterate through reserved and unavailable memory
+ * @i: u64 used as loop variable
+ * @flags: pick from blocks based on memory attributes
+ * @p_start: ptr to phys_addr_t for start address of the range, can be %NULL
+ * @p_end: ptr to phys_addr_t for end address of the range, can be %NULL
+ *
+ * Walks over unavailabled but reserved (reserved && !memory) areas of memblock.
+ * Available as soon as memblock is initialized.
+ * Note: because this memory does not belong to any physical node, flags and
+ * nid arguments do not make sense and thus not exported as arguments.
+ */
+#define for_each_resv_unavail_range(i, p_start, p_end) \
+ for_each_mem_range(i, &memblock.reserved, &memblock.memory, \
+ NUMA_NO_NODE, MEMBLOCK_NONE, p_start, p_end, NULL)
+
static inline void memblock_set_region_flags(struct memblock_region *r,
unsigned long flags)
{
diff --git a/include/linux/mm.h b/include/linux/mm.h
index 183ac5e733db..0a440ff8f226 100644
--- a/include/linux/mm.h
+++ b/include/linux/mm.h
@@ -1968,6 +1968,12 @@ extern int __meminit __early_pfn_to_nid(unsigned long pfn,
struct mminit_pfnnid_cache *state);
#endif
+#ifdef CONFIG_HAVE_MEMBLOCK
+void zero_resv_unavail(void);
+#else
+static inline void __paginginit zero_resv_unavail(void) {}
+#endif
+
extern void set_dma_reserve(unsigned long new_dma_reserve);
extern void memmap_init_zone(unsigned long, int, unsigned long,
unsigned long, enum memmap_context);
diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index 4d67fe3dd172..484c16fb5f0d 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -6261,6 +6261,34 @@ void __paginginit free_area_init_node(int nid, unsigned long *zones_size,
free_area_init_core(pgdat);
}
+#ifdef CONFIG_HAVE_MEMBLOCK
+/*
+ * Only struct pages that are backed by physical memory are zeroed and
+ * initialized by going through __init_single_page(). But, there are some
+ * struct pages which are reserved in memblock allocator and their fields
+ * may be accessed (for example page_to_pfn() on some configuration accesses
+ * flags). We must explicitly zero those struct pages.
+ */
+void __paginginit zero_resv_unavail(void)
+{
+ phys_addr_t start, end;
+ unsigned long pfn;
+ u64 i, pgcnt;
+
+ /* Loop through ranges that are reserved, but do not have reported
+ * physical memory backing.
+ */
+ pgcnt = 0;
+ for_each_resv_unavail_range(i, &start, &end) {
+ for (pfn = PFN_DOWN(start); pfn < PFN_UP(end); pfn++) {
+ mm_zero_struct_page(pfn_to_page(pfn));
+ pgcnt++;
+ }
+ }
+ pr_info("Reserved but unavailable: %lld pages", pgcnt);
+}
+#endif /* CONFIG_HAVE_MEMBLOCK */
+
#ifdef CONFIG_HAVE_MEMBLOCK_NODE_MAP
#if MAX_NUMNODES > 1
@@ -6684,6 +6712,7 @@ void __init free_area_init_nodes(unsigned long *max_zone_pfn)
node_set_state(nid, N_MEMORY);
check_for_memory(pgdat, nid);
}
+ zero_resv_unavail();
}
static int __init cmdline_parse_core(char *p, unsigned long *core)
@@ -6847,6 +6876,7 @@ void __init free_area_init(unsigned long *zones_size)
{
free_area_init_node(0, zones_size,
__pa(PAGE_OFFSET) >> PAGE_SHIFT, NULL);
+ zero_resv_unavail();
}
static int page_alloc_cpu_dead(unsigned int cpu)
--
2.14.1
^ permalink raw reply related [flat|nested] 79+ messages in thread
* [PATCH v7 08/11] mm: zero reserved and unavailable struct pages
@ 2017-08-29 2:02 ` Pavel Tatashin
0 siblings, 0 replies; 79+ messages in thread
From: Pavel Tatashin @ 2017-08-29 2:02 UTC (permalink / raw)
To: linux-kernel, sparclinux, linux-mm, linuxppc-dev, linux-s390,
linux-arm-kernel, x86, kasan-dev, borntraeger, heiko.carstens,
davem, willy, mhocko, ard.biesheuvel, will.deacon,
catalin.marinas, sam, mgorman, Steven.Sistare, daniel.m.jordan,
bob.picco
Some memory is reserved but unavailable: not present in memblock.memory
(because not backed by physical pages), but present in memblock.reserved.
Such memory has backing struct pages, but they are not initialized by going
through __init_single_page().
In some cases these struct pages are accessed even if they do not contain
any data. One example is page_to_pfn() might access page->flags if this is
where section information is stored (CONFIG_SPARSEMEM,
SECTION_IN_PAGE_FLAGS).
Since, struct pages are zeroed in __init_single_page(), and not during
allocation time, we must zero such struct pages explicitly.
The patch involves adding a new memblock iterator:
for_each_resv_unavail_range(i, p_start, p_end)
Which iterates through reserved && !memory lists, and we zero struct pages
explicitly by calling mm_zero_struct_page().
Signed-off-by: Pavel Tatashin <pasha.tatashin@oracle.com>
Reviewed-by: Steven Sistare <steven.sistare@oracle.com>
Reviewed-by: Daniel Jordan <daniel.m.jordan@oracle.com>
Reviewed-by: Bob Picco <bob.picco@oracle.com>
---
include/linux/memblock.h | 16 ++++++++++++++++
include/linux/mm.h | 6 ++++++
mm/page_alloc.c | 30 ++++++++++++++++++++++++++++++
3 files changed, 52 insertions(+)
diff --git a/include/linux/memblock.h b/include/linux/memblock.h
index bae11c7e7bf3..bdd4268f9323 100644
--- a/include/linux/memblock.h
+++ b/include/linux/memblock.h
@@ -237,6 +237,22 @@ unsigned long memblock_next_valid_pfn(unsigned long pfn, unsigned long max_pfn);
for_each_mem_range_rev(i, &memblock.memory, &memblock.reserved, \
nid, flags, p_start, p_end, p_nid)
+/**
+ * for_each_resv_unavail_range - iterate through reserved and unavailable memory
+ * @i: u64 used as loop variable
+ * @flags: pick from blocks based on memory attributes
+ * @p_start: ptr to phys_addr_t for start address of the range, can be %NULL
+ * @p_end: ptr to phys_addr_t for end address of the range, can be %NULL
+ *
+ * Walks over unavailabled but reserved (reserved && !memory) areas of memblock.
+ * Available as soon as memblock is initialized.
+ * Note: because this memory does not belong to any physical node, flags and
+ * nid arguments do not make sense and thus not exported as arguments.
+ */
+#define for_each_resv_unavail_range(i, p_start, p_end) \
+ for_each_mem_range(i, &memblock.reserved, &memblock.memory, \
+ NUMA_NO_NODE, MEMBLOCK_NONE, p_start, p_end, NULL)
+
static inline void memblock_set_region_flags(struct memblock_region *r,
unsigned long flags)
{
diff --git a/include/linux/mm.h b/include/linux/mm.h
index 183ac5e733db..0a440ff8f226 100644
--- a/include/linux/mm.h
+++ b/include/linux/mm.h
@@ -1968,6 +1968,12 @@ extern int __meminit __early_pfn_to_nid(unsigned long pfn,
struct mminit_pfnnid_cache *state);
#endif
+#ifdef CONFIG_HAVE_MEMBLOCK
+void zero_resv_unavail(void);
+#else
+static inline void __paginginit zero_resv_unavail(void) {}
+#endif
+
extern void set_dma_reserve(unsigned long new_dma_reserve);
extern void memmap_init_zone(unsigned long, int, unsigned long,
unsigned long, enum memmap_context);
diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index 4d67fe3dd172..484c16fb5f0d 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -6261,6 +6261,34 @@ void __paginginit free_area_init_node(int nid, unsigned long *zones_size,
free_area_init_core(pgdat);
}
+#ifdef CONFIG_HAVE_MEMBLOCK
+/*
+ * Only struct pages that are backed by physical memory are zeroed and
+ * initialized by going through __init_single_page(). But, there are some
+ * struct pages which are reserved in memblock allocator and their fields
+ * may be accessed (for example page_to_pfn() on some configuration accesses
+ * flags). We must explicitly zero those struct pages.
+ */
+void __paginginit zero_resv_unavail(void)
+{
+ phys_addr_t start, end;
+ unsigned long pfn;
+ u64 i, pgcnt;
+
+ /* Loop through ranges that are reserved, but do not have reported
+ * physical memory backing.
+ */
+ pgcnt = 0;
+ for_each_resv_unavail_range(i, &start, &end) {
+ for (pfn = PFN_DOWN(start); pfn < PFN_UP(end); pfn++) {
+ mm_zero_struct_page(pfn_to_page(pfn));
+ pgcnt++;
+ }
+ }
+ pr_info("Reserved but unavailable: %lld pages", pgcnt);
+}
+#endif /* CONFIG_HAVE_MEMBLOCK */
+
#ifdef CONFIG_HAVE_MEMBLOCK_NODE_MAP
#if MAX_NUMNODES > 1
@@ -6684,6 +6712,7 @@ void __init free_area_init_nodes(unsigned long *max_zone_pfn)
node_set_state(nid, N_MEMORY);
check_for_memory(pgdat, nid);
}
+ zero_resv_unavail();
}
static int __init cmdline_parse_core(char *p, unsigned long *core)
@@ -6847,6 +6876,7 @@ void __init free_area_init(unsigned long *zones_size)
{
free_area_init_node(0, zones_size,
__pa(PAGE_OFFSET) >> PAGE_SHIFT, NULL);
+ zero_resv_unavail();
}
static int page_alloc_cpu_dead(unsigned int cpu)
--
2.14.1
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply related [flat|nested] 79+ messages in thread
* [PATCH v7 08/11] mm: zero reserved and unavailable struct pages
@ 2017-08-29 2:02 ` Pavel Tatashin
0 siblings, 0 replies; 79+ messages in thread
From: Pavel Tatashin @ 2017-08-29 2:02 UTC (permalink / raw)
To: linux-arm-kernel
Some memory is reserved but unavailable: not present in memblock.memory
(because not backed by physical pages), but present in memblock.reserved.
Such memory has backing struct pages, but they are not initialized by going
through __init_single_page().
In some cases these struct pages are accessed even if they do not contain
any data. One example is page_to_pfn() might access page->flags if this is
where section information is stored (CONFIG_SPARSEMEM,
SECTION_IN_PAGE_FLAGS).
Since, struct pages are zeroed in __init_single_page(), and not during
allocation time, we must zero such struct pages explicitly.
The patch involves adding a new memblock iterator:
for_each_resv_unavail_range(i, p_start, p_end)
Which iterates through reserved && !memory lists, and we zero struct pages
explicitly by calling mm_zero_struct_page().
Signed-off-by: Pavel Tatashin <pasha.tatashin@oracle.com>
Reviewed-by: Steven Sistare <steven.sistare@oracle.com>
Reviewed-by: Daniel Jordan <daniel.m.jordan@oracle.com>
Reviewed-by: Bob Picco <bob.picco@oracle.com>
---
include/linux/memblock.h | 16 ++++++++++++++++
include/linux/mm.h | 6 ++++++
mm/page_alloc.c | 30 ++++++++++++++++++++++++++++++
3 files changed, 52 insertions(+)
diff --git a/include/linux/memblock.h b/include/linux/memblock.h
index bae11c7e7bf3..bdd4268f9323 100644
--- a/include/linux/memblock.h
+++ b/include/linux/memblock.h
@@ -237,6 +237,22 @@ unsigned long memblock_next_valid_pfn(unsigned long pfn, unsigned long max_pfn);
for_each_mem_range_rev(i, &memblock.memory, &memblock.reserved, \
nid, flags, p_start, p_end, p_nid)
+/**
+ * for_each_resv_unavail_range - iterate through reserved and unavailable memory
+ * @i: u64 used as loop variable
+ * @flags: pick from blocks based on memory attributes
+ * @p_start: ptr to phys_addr_t for start address of the range, can be %NULL
+ * @p_end: ptr to phys_addr_t for end address of the range, can be %NULL
+ *
+ * Walks over unavailabled but reserved (reserved && !memory) areas of memblock.
+ * Available as soon as memblock is initialized.
+ * Note: because this memory does not belong to any physical node, flags and
+ * nid arguments do not make sense and thus not exported as arguments.
+ */
+#define for_each_resv_unavail_range(i, p_start, p_end) \
+ for_each_mem_range(i, &memblock.reserved, &memblock.memory, \
+ NUMA_NO_NODE, MEMBLOCK_NONE, p_start, p_end, NULL)
+
static inline void memblock_set_region_flags(struct memblock_region *r,
unsigned long flags)
{
diff --git a/include/linux/mm.h b/include/linux/mm.h
index 183ac5e733db..0a440ff8f226 100644
--- a/include/linux/mm.h
+++ b/include/linux/mm.h
@@ -1968,6 +1968,12 @@ extern int __meminit __early_pfn_to_nid(unsigned long pfn,
struct mminit_pfnnid_cache *state);
#endif
+#ifdef CONFIG_HAVE_MEMBLOCK
+void zero_resv_unavail(void);
+#else
+static inline void __paginginit zero_resv_unavail(void) {}
+#endif
+
extern void set_dma_reserve(unsigned long new_dma_reserve);
extern void memmap_init_zone(unsigned long, int, unsigned long,
unsigned long, enum memmap_context);
diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index 4d67fe3dd172..484c16fb5f0d 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -6261,6 +6261,34 @@ void __paginginit free_area_init_node(int nid, unsigned long *zones_size,
free_area_init_core(pgdat);
}
+#ifdef CONFIG_HAVE_MEMBLOCK
+/*
+ * Only struct pages that are backed by physical memory are zeroed and
+ * initialized by going through __init_single_page(). But, there are some
+ * struct pages which are reserved in memblock allocator and their fields
+ * may be accessed (for example page_to_pfn() on some configuration accesses
+ * flags). We must explicitly zero those struct pages.
+ */
+void __paginginit zero_resv_unavail(void)
+{
+ phys_addr_t start, end;
+ unsigned long pfn;
+ u64 i, pgcnt;
+
+ /* Loop through ranges that are reserved, but do not have reported
+ * physical memory backing.
+ */
+ pgcnt = 0;
+ for_each_resv_unavail_range(i, &start, &end) {
+ for (pfn = PFN_DOWN(start); pfn < PFN_UP(end); pfn++) {
+ mm_zero_struct_page(pfn_to_page(pfn));
+ pgcnt++;
+ }
+ }
+ pr_info("Reserved but unavailable: %lld pages", pgcnt);
+}
+#endif /* CONFIG_HAVE_MEMBLOCK */
+
#ifdef CONFIG_HAVE_MEMBLOCK_NODE_MAP
#if MAX_NUMNODES > 1
@@ -6684,6 +6712,7 @@ void __init free_area_init_nodes(unsigned long *max_zone_pfn)
node_set_state(nid, N_MEMORY);
check_for_memory(pgdat, nid);
}
+ zero_resv_unavail();
}
static int __init cmdline_parse_core(char *p, unsigned long *core)
@@ -6847,6 +6876,7 @@ void __init free_area_init(unsigned long *zones_size)
{
free_area_init_node(0, zones_size,
__pa(PAGE_OFFSET) >> PAGE_SHIFT, NULL);
+ zero_resv_unavail();
}
static int page_alloc_cpu_dead(unsigned int cpu)
--
2.14.1
^ permalink raw reply related [flat|nested] 79+ messages in thread
* [PATCH v7 09/11] x86/kasan: explicitly zero kasan shadow memory
2017-08-29 2:02 ` Pavel Tatashin
(?)
(?)
@ 2017-08-29 2:02 ` Pavel Tatashin
-1 siblings, 0 replies; 79+ messages in thread
From: Pavel Tatashin @ 2017-08-29 2:02 UTC (permalink / raw)
To: linux-kernel, sparclinux, linux-mm, linuxppc-dev, linux-s390,
linux-arm-kernel, x86, kasan-dev, borntraeger, heiko.carstens,
davem, willy, mhocko, ard.biesheuvel, will.deacon,
catalin.marinas, sam, mgorman, Steven.Sistare, daniel.m.jordan,
bob.picco
To optimize the performance of struct page initialization,
vmemmap_populate() will no longer zero memory.
We must explicitly zero the memory that is allocated by vmemmap_populate()
for kasan, as this memory does not go through struct page initialization
path.
Signed-off-by: Pavel Tatashin <pasha.tatashin@oracle.com>
Reviewed-by: Steven Sistare <steven.sistare@oracle.com>
Reviewed-by: Daniel Jordan <daniel.m.jordan@oracle.com>
Reviewed-by: Bob Picco <bob.picco@oracle.com>
---
arch/x86/mm/kasan_init_64.c | 66 +++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 66 insertions(+)
diff --git a/arch/x86/mm/kasan_init_64.c b/arch/x86/mm/kasan_init_64.c
index 02c9d7553409..96fde5bf9597 100644
--- a/arch/x86/mm/kasan_init_64.c
+++ b/arch/x86/mm/kasan_init_64.c
@@ -84,6 +84,66 @@ static struct notifier_block kasan_die_notifier = {
};
#endif
+/*
+ * x86 variant of vmemmap_populate() uses either PMD_SIZE pages or base pages
+ * to map allocated memory. This routine determines the page size for the given
+ * address from vmemmap.
+ */
+static u64 get_vmemmap_pgsz(u64 addr)
+{
+ pgd_t *pgd;
+ p4d_t *p4d;
+ pud_t *pud;
+ pmd_t *pmd;
+
+ pgd = pgd_offset_k(addr);
+ BUG_ON(pgd_none(*pgd) || pgd_large(*pgd));
+
+ p4d = p4d_offset(pgd, addr);
+ BUG_ON(p4d_none(*p4d) || p4d_large(*p4d));
+
+ pud = pud_offset(p4d, addr);
+ BUG_ON(pud_none(*pud) || pud_large(*pud));
+
+ pmd = pmd_offset(pud, addr);
+ BUG_ON(pmd_none(*pmd));
+
+ if (pmd_large(*pmd))
+ return PMD_SIZE;
+ return PAGE_SIZE;
+}
+
+/*
+ * Memory that was allocated by vmemmap_populate is not zeroed, so we must
+ * zero it here explicitly.
+ */
+static void
+zero_vmemmap_populated_memory(void)
+{
+ u64 i, start, end;
+
+ for (i = 0; i < E820_MAX_ENTRIES && pfn_mapped[i].end; i++) {
+ void *kaddr_start = pfn_to_kaddr(pfn_mapped[i].start);
+ void *kaddr_end = pfn_to_kaddr(pfn_mapped[i].end);
+
+ start = (u64)kasan_mem_to_shadow(kaddr_start);
+ end = (u64)kasan_mem_to_shadow(kaddr_end);
+
+ /* Round to the start end of the mapped pages */
+ start = rounddown(start, get_vmemmap_pgsz(start));
+ end = roundup(end, get_vmemmap_pgsz(start));
+ memset((void *)start, 0, end - start);
+ }
+
+ start = (u64)kasan_mem_to_shadow(_stext);
+ end = (u64)kasan_mem_to_shadow(_end);
+
+ /* Round to the start end of the mapped pages */
+ start = rounddown(start, get_vmemmap_pgsz(start));
+ end = roundup(end, get_vmemmap_pgsz(start));
+ memset((void *)start, 0, end - start);
+}
+
void __init kasan_early_init(void)
{
int i;
@@ -146,6 +206,12 @@ void __init kasan_init(void)
load_cr3(init_top_pgt);
__flush_tlb_all();
+ /*
+ * vmemmap_populate does not zero the memory, so we need to zero it
+ * explicitly
+ */
+ zero_vmemmap_populated_memory();
+
/*
* kasan_zero_page has been used as early shadow memory, thus it may
* contain some garbage. Now we can clear and write protect it, since
--
2.14.1
^ permalink raw reply related [flat|nested] 79+ messages in thread
* [PATCH v7 09/11] x86/kasan: explicitly zero kasan shadow memory
@ 2017-08-29 2:02 ` Pavel Tatashin
0 siblings, 0 replies; 79+ messages in thread
From: Pavel Tatashin @ 2017-08-29 2:02 UTC (permalink / raw)
To: linux-arm-kernel
To optimize the performance of struct page initialization,
vmemmap_populate() will no longer zero memory.
We must explicitly zero the memory that is allocated by vmemmap_populate()
for kasan, as this memory does not go through struct page initialization
path.
Signed-off-by: Pavel Tatashin <pasha.tatashin@oracle.com>
Reviewed-by: Steven Sistare <steven.sistare@oracle.com>
Reviewed-by: Daniel Jordan <daniel.m.jordan@oracle.com>
Reviewed-by: Bob Picco <bob.picco@oracle.com>
---
arch/x86/mm/kasan_init_64.c | 66 +++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 66 insertions(+)
diff --git a/arch/x86/mm/kasan_init_64.c b/arch/x86/mm/kasan_init_64.c
index 02c9d7553409..96fde5bf9597 100644
--- a/arch/x86/mm/kasan_init_64.c
+++ b/arch/x86/mm/kasan_init_64.c
@@ -84,6 +84,66 @@ static struct notifier_block kasan_die_notifier = {
};
#endif
+/*
+ * x86 variant of vmemmap_populate() uses either PMD_SIZE pages or base pages
+ * to map allocated memory. This routine determines the page size for the given
+ * address from vmemmap.
+ */
+static u64 get_vmemmap_pgsz(u64 addr)
+{
+ pgd_t *pgd;
+ p4d_t *p4d;
+ pud_t *pud;
+ pmd_t *pmd;
+
+ pgd = pgd_offset_k(addr);
+ BUG_ON(pgd_none(*pgd) || pgd_large(*pgd));
+
+ p4d = p4d_offset(pgd, addr);
+ BUG_ON(p4d_none(*p4d) || p4d_large(*p4d));
+
+ pud = pud_offset(p4d, addr);
+ BUG_ON(pud_none(*pud) || pud_large(*pud));
+
+ pmd = pmd_offset(pud, addr);
+ BUG_ON(pmd_none(*pmd));
+
+ if (pmd_large(*pmd))
+ return PMD_SIZE;
+ return PAGE_SIZE;
+}
+
+/*
+ * Memory that was allocated by vmemmap_populate is not zeroed, so we must
+ * zero it here explicitly.
+ */
+static void
+zero_vmemmap_populated_memory(void)
+{
+ u64 i, start, end;
+
+ for (i = 0; i < E820_MAX_ENTRIES && pfn_mapped[i].end; i++) {
+ void *kaddr_start = pfn_to_kaddr(pfn_mapped[i].start);
+ void *kaddr_end = pfn_to_kaddr(pfn_mapped[i].end);
+
+ start = (u64)kasan_mem_to_shadow(kaddr_start);
+ end = (u64)kasan_mem_to_shadow(kaddr_end);
+
+ /* Round to the start end of the mapped pages */
+ start = rounddown(start, get_vmemmap_pgsz(start));
+ end = roundup(end, get_vmemmap_pgsz(start));
+ memset((void *)start, 0, end - start);
+ }
+
+ start = (u64)kasan_mem_to_shadow(_stext);
+ end = (u64)kasan_mem_to_shadow(_end);
+
+ /* Round to the start end of the mapped pages */
+ start = rounddown(start, get_vmemmap_pgsz(start));
+ end = roundup(end, get_vmemmap_pgsz(start));
+ memset((void *)start, 0, end - start);
+}
+
void __init kasan_early_init(void)
{
int i;
@@ -146,6 +206,12 @@ void __init kasan_init(void)
load_cr3(init_top_pgt);
__flush_tlb_all();
+ /*
+ * vmemmap_populate does not zero the memory, so we need to zero it
+ * explicitly
+ */
+ zero_vmemmap_populated_memory();
+
/*
* kasan_zero_page has been used as early shadow memory, thus it may
* contain some garbage. Now we can clear and write protect it, since
--
2.14.1
^ permalink raw reply related [flat|nested] 79+ messages in thread
* [PATCH v7 09/11] x86/kasan: explicitly zero kasan shadow memory
@ 2017-08-29 2:02 ` Pavel Tatashin
0 siblings, 0 replies; 79+ messages in thread
From: Pavel Tatashin @ 2017-08-29 2:02 UTC (permalink / raw)
To: linux-kernel, sparclinux, linux-mm, linuxppc-dev, linux-s390,
linux-arm-kernel, x86, kasan-dev, borntraeger, heiko.carstens,
davem, willy, mhocko, ard.biesheuvel, will.deacon,
catalin.marinas, sam, mgorman, Steven.Sistare, daniel.m.jordan,
bob.picco
To optimize the performance of struct page initialization,
vmemmap_populate() will no longer zero memory.
We must explicitly zero the memory that is allocated by vmemmap_populate()
for kasan, as this memory does not go through struct page initialization
path.
Signed-off-by: Pavel Tatashin <pasha.tatashin@oracle.com>
Reviewed-by: Steven Sistare <steven.sistare@oracle.com>
Reviewed-by: Daniel Jordan <daniel.m.jordan@oracle.com>
Reviewed-by: Bob Picco <bob.picco@oracle.com>
---
arch/x86/mm/kasan_init_64.c | 66 +++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 66 insertions(+)
diff --git a/arch/x86/mm/kasan_init_64.c b/arch/x86/mm/kasan_init_64.c
index 02c9d7553409..96fde5bf9597 100644
--- a/arch/x86/mm/kasan_init_64.c
+++ b/arch/x86/mm/kasan_init_64.c
@@ -84,6 +84,66 @@ static struct notifier_block kasan_die_notifier = {
};
#endif
+/*
+ * x86 variant of vmemmap_populate() uses either PMD_SIZE pages or base pages
+ * to map allocated memory. This routine determines the page size for the given
+ * address from vmemmap.
+ */
+static u64 get_vmemmap_pgsz(u64 addr)
+{
+ pgd_t *pgd;
+ p4d_t *p4d;
+ pud_t *pud;
+ pmd_t *pmd;
+
+ pgd = pgd_offset_k(addr);
+ BUG_ON(pgd_none(*pgd) || pgd_large(*pgd));
+
+ p4d = p4d_offset(pgd, addr);
+ BUG_ON(p4d_none(*p4d) || p4d_large(*p4d));
+
+ pud = pud_offset(p4d, addr);
+ BUG_ON(pud_none(*pud) || pud_large(*pud));
+
+ pmd = pmd_offset(pud, addr);
+ BUG_ON(pmd_none(*pmd));
+
+ if (pmd_large(*pmd))
+ return PMD_SIZE;
+ return PAGE_SIZE;
+}
+
+/*
+ * Memory that was allocated by vmemmap_populate is not zeroed, so we must
+ * zero it here explicitly.
+ */
+static void
+zero_vmemmap_populated_memory(void)
+{
+ u64 i, start, end;
+
+ for (i = 0; i < E820_MAX_ENTRIES && pfn_mapped[i].end; i++) {
+ void *kaddr_start = pfn_to_kaddr(pfn_mapped[i].start);
+ void *kaddr_end = pfn_to_kaddr(pfn_mapped[i].end);
+
+ start = (u64)kasan_mem_to_shadow(kaddr_start);
+ end = (u64)kasan_mem_to_shadow(kaddr_end);
+
+ /* Round to the start end of the mapped pages */
+ start = rounddown(start, get_vmemmap_pgsz(start));
+ end = roundup(end, get_vmemmap_pgsz(start));
+ memset((void *)start, 0, end - start);
+ }
+
+ start = (u64)kasan_mem_to_shadow(_stext);
+ end = (u64)kasan_mem_to_shadow(_end);
+
+ /* Round to the start end of the mapped pages */
+ start = rounddown(start, get_vmemmap_pgsz(start));
+ end = roundup(end, get_vmemmap_pgsz(start));
+ memset((void *)start, 0, end - start);
+}
+
void __init kasan_early_init(void)
{
int i;
@@ -146,6 +206,12 @@ void __init kasan_init(void)
load_cr3(init_top_pgt);
__flush_tlb_all();
+ /*
+ * vmemmap_populate does not zero the memory, so we need to zero it
+ * explicitly
+ */
+ zero_vmemmap_populated_memory();
+
/*
* kasan_zero_page has been used as early shadow memory, thus it may
* contain some garbage. Now we can clear and write protect it, since
--
2.14.1
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply related [flat|nested] 79+ messages in thread
* [PATCH v7 09/11] x86/kasan: explicitly zero kasan shadow memory
@ 2017-08-29 2:02 ` Pavel Tatashin
0 siblings, 0 replies; 79+ messages in thread
From: Pavel Tatashin @ 2017-08-29 2:02 UTC (permalink / raw)
To: linux-arm-kernel
To optimize the performance of struct page initialization,
vmemmap_populate() will no longer zero memory.
We must explicitly zero the memory that is allocated by vmemmap_populate()
for kasan, as this memory does not go through struct page initialization
path.
Signed-off-by: Pavel Tatashin <pasha.tatashin@oracle.com>
Reviewed-by: Steven Sistare <steven.sistare@oracle.com>
Reviewed-by: Daniel Jordan <daniel.m.jordan@oracle.com>
Reviewed-by: Bob Picco <bob.picco@oracle.com>
---
arch/x86/mm/kasan_init_64.c | 66 +++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 66 insertions(+)
diff --git a/arch/x86/mm/kasan_init_64.c b/arch/x86/mm/kasan_init_64.c
index 02c9d7553409..96fde5bf9597 100644
--- a/arch/x86/mm/kasan_init_64.c
+++ b/arch/x86/mm/kasan_init_64.c
@@ -84,6 +84,66 @@ static struct notifier_block kasan_die_notifier = {
};
#endif
+/*
+ * x86 variant of vmemmap_populate() uses either PMD_SIZE pages or base pages
+ * to map allocated memory. This routine determines the page size for the given
+ * address from vmemmap.
+ */
+static u64 get_vmemmap_pgsz(u64 addr)
+{
+ pgd_t *pgd;
+ p4d_t *p4d;
+ pud_t *pud;
+ pmd_t *pmd;
+
+ pgd = pgd_offset_k(addr);
+ BUG_ON(pgd_none(*pgd) || pgd_large(*pgd));
+
+ p4d = p4d_offset(pgd, addr);
+ BUG_ON(p4d_none(*p4d) || p4d_large(*p4d));
+
+ pud = pud_offset(p4d, addr);
+ BUG_ON(pud_none(*pud) || pud_large(*pud));
+
+ pmd = pmd_offset(pud, addr);
+ BUG_ON(pmd_none(*pmd));
+
+ if (pmd_large(*pmd))
+ return PMD_SIZE;
+ return PAGE_SIZE;
+}
+
+/*
+ * Memory that was allocated by vmemmap_populate is not zeroed, so we must
+ * zero it here explicitly.
+ */
+static void
+zero_vmemmap_populated_memory(void)
+{
+ u64 i, start, end;
+
+ for (i = 0; i < E820_MAX_ENTRIES && pfn_mapped[i].end; i++) {
+ void *kaddr_start = pfn_to_kaddr(pfn_mapped[i].start);
+ void *kaddr_end = pfn_to_kaddr(pfn_mapped[i].end);
+
+ start = (u64)kasan_mem_to_shadow(kaddr_start);
+ end = (u64)kasan_mem_to_shadow(kaddr_end);
+
+ /* Round to the start end of the mapped pages */
+ start = rounddown(start, get_vmemmap_pgsz(start));
+ end = roundup(end, get_vmemmap_pgsz(start));
+ memset((void *)start, 0, end - start);
+ }
+
+ start = (u64)kasan_mem_to_shadow(_stext);
+ end = (u64)kasan_mem_to_shadow(_end);
+
+ /* Round to the start end of the mapped pages */
+ start = rounddown(start, get_vmemmap_pgsz(start));
+ end = roundup(end, get_vmemmap_pgsz(start));
+ memset((void *)start, 0, end - start);
+}
+
void __init kasan_early_init(void)
{
int i;
@@ -146,6 +206,12 @@ void __init kasan_init(void)
load_cr3(init_top_pgt);
__flush_tlb_all();
+ /*
+ * vmemmap_populate does not zero the memory, so we need to zero it
+ * explicitly
+ */
+ zero_vmemmap_populated_memory();
+
/*
* kasan_zero_page has been used as early shadow memory, thus it may
* contain some garbage. Now we can clear and write protect it, since
--
2.14.1
^ permalink raw reply related [flat|nested] 79+ messages in thread
* [PATCH v7 10/11] arm64/kasan: explicitly zero kasan shadow memory
2017-08-29 2:02 ` Pavel Tatashin
(?)
(?)
@ 2017-08-29 2:02 ` Pavel Tatashin
-1 siblings, 0 replies; 79+ messages in thread
From: Pavel Tatashin @ 2017-08-29 2:02 UTC (permalink / raw)
To: linux-kernel, sparclinux, linux-mm, linuxppc-dev, linux-s390,
linux-arm-kernel, x86, kasan-dev, borntraeger, heiko.carstens,
davem, willy, mhocko, ard.biesheuvel, will.deacon,
catalin.marinas, sam, mgorman, Steven.Sistare, daniel.m.jordan,
bob.picco
To optimize the performance of struct page initialization,
vmemmap_populate() will no longer zero memory.
We must explicitly zero the memory that is allocated by vmemmap_populate()
for kasan, as this memory does not go through struct page initialization
path.
Signed-off-by: Pavel Tatashin <pasha.tatashin@oracle.com>
Reviewed-by: Steven Sistare <steven.sistare@oracle.com>
Reviewed-by: Daniel Jordan <daniel.m.jordan@oracle.com>
Reviewed-by: Bob Picco <bob.picco@oracle.com>
---
arch/arm64/mm/kasan_init.c | 42 ++++++++++++++++++++++++++++++++++++++++++
1 file changed, 42 insertions(+)
diff --git a/arch/arm64/mm/kasan_init.c b/arch/arm64/mm/kasan_init.c
index 81f03959a4ab..e78a9ecbb687 100644
--- a/arch/arm64/mm/kasan_init.c
+++ b/arch/arm64/mm/kasan_init.c
@@ -135,6 +135,41 @@ static void __init clear_pgds(unsigned long start,
set_pgd(pgd_offset_k(start), __pgd(0));
}
+/*
+ * Memory that was allocated by vmemmap_populate is not zeroed, so we must
+ * zero it here explicitly.
+ */
+static void
+zero_vmemmap_populated_memory(void)
+{
+ struct memblock_region *reg;
+ u64 start, end;
+
+ for_each_memblock(memory, reg) {
+ start = __phys_to_virt(reg->base);
+ end = __phys_to_virt(reg->base + reg->size);
+
+ if (start >= end)
+ break;
+
+ start = (u64)kasan_mem_to_shadow((void *)start);
+ end = (u64)kasan_mem_to_shadow((void *)end);
+
+ /* Round to the start end of the mapped pages */
+ start = round_down(start, SWAPPER_BLOCK_SIZE);
+ end = round_up(end, SWAPPER_BLOCK_SIZE);
+ memset((void *)start, 0, end - start);
+ }
+
+ start = (u64)kasan_mem_to_shadow(_text);
+ end = (u64)kasan_mem_to_shadow(_end);
+
+ /* Round to the start end of the mapped pages */
+ start = round_down(start, SWAPPER_BLOCK_SIZE);
+ end = round_up(end, SWAPPER_BLOCK_SIZE);
+ memset((void *)start, 0, end - start);
+}
+
void __init kasan_init(void)
{
u64 kimg_shadow_start, kimg_shadow_end;
@@ -205,8 +240,15 @@ void __init kasan_init(void)
pfn_pte(sym_to_pfn(kasan_zero_page), PAGE_KERNEL_RO));
memset(kasan_zero_page, 0, PAGE_SIZE);
+
cpu_replace_ttbr1(lm_alias(swapper_pg_dir));
+ /*
+ * vmemmap_populate does not zero the memory, so we need to zero it
+ * explicitly
+ */
+ zero_vmemmap_populated_memory();
+
/* At this point kasan is fully initialized. Enable error messages */
init_task.kasan_depth = 0;
pr_info("KernelAddressSanitizer initialized\n");
--
2.14.1
^ permalink raw reply related [flat|nested] 79+ messages in thread
* [PATCH v7 10/11] arm64/kasan: explicitly zero kasan shadow memory
@ 2017-08-29 2:02 ` Pavel Tatashin
0 siblings, 0 replies; 79+ messages in thread
From: Pavel Tatashin @ 2017-08-29 2:02 UTC (permalink / raw)
To: linux-arm-kernel
To optimize the performance of struct page initialization,
vmemmap_populate() will no longer zero memory.
We must explicitly zero the memory that is allocated by vmemmap_populate()
for kasan, as this memory does not go through struct page initialization
path.
Signed-off-by: Pavel Tatashin <pasha.tatashin@oracle.com>
Reviewed-by: Steven Sistare <steven.sistare@oracle.com>
Reviewed-by: Daniel Jordan <daniel.m.jordan@oracle.com>
Reviewed-by: Bob Picco <bob.picco@oracle.com>
---
arch/arm64/mm/kasan_init.c | 42 ++++++++++++++++++++++++++++++++++++++++++
1 file changed, 42 insertions(+)
diff --git a/arch/arm64/mm/kasan_init.c b/arch/arm64/mm/kasan_init.c
index 81f03959a4ab..e78a9ecbb687 100644
--- a/arch/arm64/mm/kasan_init.c
+++ b/arch/arm64/mm/kasan_init.c
@@ -135,6 +135,41 @@ static void __init clear_pgds(unsigned long start,
set_pgd(pgd_offset_k(start), __pgd(0));
}
+/*
+ * Memory that was allocated by vmemmap_populate is not zeroed, so we must
+ * zero it here explicitly.
+ */
+static void
+zero_vmemmap_populated_memory(void)
+{
+ struct memblock_region *reg;
+ u64 start, end;
+
+ for_each_memblock(memory, reg) {
+ start = __phys_to_virt(reg->base);
+ end = __phys_to_virt(reg->base + reg->size);
+
+ if (start >= end)
+ break;
+
+ start = (u64)kasan_mem_to_shadow((void *)start);
+ end = (u64)kasan_mem_to_shadow((void *)end);
+
+ /* Round to the start end of the mapped pages */
+ start = round_down(start, SWAPPER_BLOCK_SIZE);
+ end = round_up(end, SWAPPER_BLOCK_SIZE);
+ memset((void *)start, 0, end - start);
+ }
+
+ start = (u64)kasan_mem_to_shadow(_text);
+ end = (u64)kasan_mem_to_shadow(_end);
+
+ /* Round to the start end of the mapped pages */
+ start = round_down(start, SWAPPER_BLOCK_SIZE);
+ end = round_up(end, SWAPPER_BLOCK_SIZE);
+ memset((void *)start, 0, end - start);
+}
+
void __init kasan_init(void)
{
u64 kimg_shadow_start, kimg_shadow_end;
@@ -205,8 +240,15 @@ void __init kasan_init(void)
pfn_pte(sym_to_pfn(kasan_zero_page), PAGE_KERNEL_RO));
memset(kasan_zero_page, 0, PAGE_SIZE);
+
cpu_replace_ttbr1(lm_alias(swapper_pg_dir));
+ /*
+ * vmemmap_populate does not zero the memory, so we need to zero it
+ * explicitly
+ */
+ zero_vmemmap_populated_memory();
+
/* At this point kasan is fully initialized. Enable error messages */
init_task.kasan_depth = 0;
pr_info("KernelAddressSanitizer initialized\n");
--
2.14.1
^ permalink raw reply related [flat|nested] 79+ messages in thread
* [PATCH v7 10/11] arm64/kasan: explicitly zero kasan shadow memory
@ 2017-08-29 2:02 ` Pavel Tatashin
0 siblings, 0 replies; 79+ messages in thread
From: Pavel Tatashin @ 2017-08-29 2:02 UTC (permalink / raw)
To: linux-kernel, sparclinux, linux-mm, linuxppc-dev, linux-s390,
linux-arm-kernel, x86, kasan-dev, borntraeger, heiko.carstens,
davem, willy, mhocko, ard.biesheuvel, will.deacon,
catalin.marinas, sam, mgorman, Steven.Sistare, daniel.m.jordan,
bob.picco
To optimize the performance of struct page initialization,
vmemmap_populate() will no longer zero memory.
We must explicitly zero the memory that is allocated by vmemmap_populate()
for kasan, as this memory does not go through struct page initialization
path.
Signed-off-by: Pavel Tatashin <pasha.tatashin@oracle.com>
Reviewed-by: Steven Sistare <steven.sistare@oracle.com>
Reviewed-by: Daniel Jordan <daniel.m.jordan@oracle.com>
Reviewed-by: Bob Picco <bob.picco@oracle.com>
---
arch/arm64/mm/kasan_init.c | 42 ++++++++++++++++++++++++++++++++++++++++++
1 file changed, 42 insertions(+)
diff --git a/arch/arm64/mm/kasan_init.c b/arch/arm64/mm/kasan_init.c
index 81f03959a4ab..e78a9ecbb687 100644
--- a/arch/arm64/mm/kasan_init.c
+++ b/arch/arm64/mm/kasan_init.c
@@ -135,6 +135,41 @@ static void __init clear_pgds(unsigned long start,
set_pgd(pgd_offset_k(start), __pgd(0));
}
+/*
+ * Memory that was allocated by vmemmap_populate is not zeroed, so we must
+ * zero it here explicitly.
+ */
+static void
+zero_vmemmap_populated_memory(void)
+{
+ struct memblock_region *reg;
+ u64 start, end;
+
+ for_each_memblock(memory, reg) {
+ start = __phys_to_virt(reg->base);
+ end = __phys_to_virt(reg->base + reg->size);
+
+ if (start >= end)
+ break;
+
+ start = (u64)kasan_mem_to_shadow((void *)start);
+ end = (u64)kasan_mem_to_shadow((void *)end);
+
+ /* Round to the start end of the mapped pages */
+ start = round_down(start, SWAPPER_BLOCK_SIZE);
+ end = round_up(end, SWAPPER_BLOCK_SIZE);
+ memset((void *)start, 0, end - start);
+ }
+
+ start = (u64)kasan_mem_to_shadow(_text);
+ end = (u64)kasan_mem_to_shadow(_end);
+
+ /* Round to the start end of the mapped pages */
+ start = round_down(start, SWAPPER_BLOCK_SIZE);
+ end = round_up(end, SWAPPER_BLOCK_SIZE);
+ memset((void *)start, 0, end - start);
+}
+
void __init kasan_init(void)
{
u64 kimg_shadow_start, kimg_shadow_end;
@@ -205,8 +240,15 @@ void __init kasan_init(void)
pfn_pte(sym_to_pfn(kasan_zero_page), PAGE_KERNEL_RO));
memset(kasan_zero_page, 0, PAGE_SIZE);
+
cpu_replace_ttbr1(lm_alias(swapper_pg_dir));
+ /*
+ * vmemmap_populate does not zero the memory, so we need to zero it
+ * explicitly
+ */
+ zero_vmemmap_populated_memory();
+
/* At this point kasan is fully initialized. Enable error messages */
init_task.kasan_depth = 0;
pr_info("KernelAddressSanitizer initialized\n");
--
2.14.1
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply related [flat|nested] 79+ messages in thread
* [PATCH v7 10/11] arm64/kasan: explicitly zero kasan shadow memory
@ 2017-08-29 2:02 ` Pavel Tatashin
0 siblings, 0 replies; 79+ messages in thread
From: Pavel Tatashin @ 2017-08-29 2:02 UTC (permalink / raw)
To: linux-arm-kernel
To optimize the performance of struct page initialization,
vmemmap_populate() will no longer zero memory.
We must explicitly zero the memory that is allocated by vmemmap_populate()
for kasan, as this memory does not go through struct page initialization
path.
Signed-off-by: Pavel Tatashin <pasha.tatashin@oracle.com>
Reviewed-by: Steven Sistare <steven.sistare@oracle.com>
Reviewed-by: Daniel Jordan <daniel.m.jordan@oracle.com>
Reviewed-by: Bob Picco <bob.picco@oracle.com>
---
arch/arm64/mm/kasan_init.c | 42 ++++++++++++++++++++++++++++++++++++++++++
1 file changed, 42 insertions(+)
diff --git a/arch/arm64/mm/kasan_init.c b/arch/arm64/mm/kasan_init.c
index 81f03959a4ab..e78a9ecbb687 100644
--- a/arch/arm64/mm/kasan_init.c
+++ b/arch/arm64/mm/kasan_init.c
@@ -135,6 +135,41 @@ static void __init clear_pgds(unsigned long start,
set_pgd(pgd_offset_k(start), __pgd(0));
}
+/*
+ * Memory that was allocated by vmemmap_populate is not zeroed, so we must
+ * zero it here explicitly.
+ */
+static void
+zero_vmemmap_populated_memory(void)
+{
+ struct memblock_region *reg;
+ u64 start, end;
+
+ for_each_memblock(memory, reg) {
+ start = __phys_to_virt(reg->base);
+ end = __phys_to_virt(reg->base + reg->size);
+
+ if (start >= end)
+ break;
+
+ start = (u64)kasan_mem_to_shadow((void *)start);
+ end = (u64)kasan_mem_to_shadow((void *)end);
+
+ /* Round to the start end of the mapped pages */
+ start = round_down(start, SWAPPER_BLOCK_SIZE);
+ end = round_up(end, SWAPPER_BLOCK_SIZE);
+ memset((void *)start, 0, end - start);
+ }
+
+ start = (u64)kasan_mem_to_shadow(_text);
+ end = (u64)kasan_mem_to_shadow(_end);
+
+ /* Round to the start end of the mapped pages */
+ start = round_down(start, SWAPPER_BLOCK_SIZE);
+ end = round_up(end, SWAPPER_BLOCK_SIZE);
+ memset((void *)start, 0, end - start);
+}
+
void __init kasan_init(void)
{
u64 kimg_shadow_start, kimg_shadow_end;
@@ -205,8 +240,15 @@ void __init kasan_init(void)
pfn_pte(sym_to_pfn(kasan_zero_page), PAGE_KERNEL_RO));
memset(kasan_zero_page, 0, PAGE_SIZE);
+
cpu_replace_ttbr1(lm_alias(swapper_pg_dir));
+ /*
+ * vmemmap_populate does not zero the memory, so we need to zero it
+ * explicitly
+ */
+ zero_vmemmap_populated_memory();
+
/* At this point kasan is fully initialized. Enable error messages */
init_task.kasan_depth = 0;
pr_info("KernelAddressSanitizer initialized\n");
--
2.14.1
^ permalink raw reply related [flat|nested] 79+ messages in thread
* [PATCH v7 11/11] mm: stop zeroing memory during allocation in vmemmap
2017-08-29 2:02 ` Pavel Tatashin
(?)
(?)
@ 2017-08-29 2:02 ` Pavel Tatashin
-1 siblings, 0 replies; 79+ messages in thread
From: Pavel Tatashin @ 2017-08-29 2:02 UTC (permalink / raw)
To: linux-kernel, sparclinux, linux-mm, linuxppc-dev, linux-s390,
linux-arm-kernel, x86, kasan-dev, borntraeger, heiko.carstens,
davem, willy, mhocko, ard.biesheuvel, will.deacon,
catalin.marinas, sam, mgorman, Steven.Sistare, daniel.m.jordan,
bob.picco
vmemmap_alloc_block() will no longer zero the block, so zero memory
at its call sites for everything except struct pages. Struct page memory
is zero'd by struct page initialization.
Replace allocators in sprase-vmemmap to use the non-zeroing version. So,
we will get the performance improvement by zeroing the memory in parallel
when struct pages are zeroed.
Signed-off-by: Pavel Tatashin <pasha.tatashin@oracle.com>
Reviewed-by: Steven Sistare <steven.sistare@oracle.com>
Reviewed-by: Daniel Jordan <daniel.m.jordan@oracle.com>
Reviewed-by: Bob Picco <bob.picco@oracle.com>
---
include/linux/mm.h | 11 +++++++++++
mm/sparse-vmemmap.c | 14 +++++++-------
mm/sparse.c | 6 +++---
3 files changed, 21 insertions(+), 10 deletions(-)
diff --git a/include/linux/mm.h b/include/linux/mm.h
index 0a440ff8f226..fba540aef1da 100644
--- a/include/linux/mm.h
+++ b/include/linux/mm.h
@@ -2456,6 +2456,17 @@ static inline void *vmemmap_alloc_block_buf(unsigned long size, int node)
return __vmemmap_alloc_block_buf(size, node, NULL);
}
+static inline void *vmemmap_alloc_block_zero(unsigned long size, int node)
+{
+ void *p = vmemmap_alloc_block(size, node);
+
+ if (!p)
+ return NULL;
+ memset(p, 0, size);
+
+ return p;
+}
+
void vmemmap_verify(pte_t *, int, unsigned long, unsigned long);
int vmemmap_populate_basepages(unsigned long start, unsigned long end,
int node);
diff --git a/mm/sparse-vmemmap.c b/mm/sparse-vmemmap.c
index c50b1a14d55e..423d4da85a91 100644
--- a/mm/sparse-vmemmap.c
+++ b/mm/sparse-vmemmap.c
@@ -41,7 +41,7 @@ static void * __ref __earlyonly_bootmem_alloc(int node,
unsigned long align,
unsigned long goal)
{
- return memblock_virt_alloc_try_nid(size, align, goal,
+ return memblock_virt_alloc_try_nid_raw(size, align, goal,
BOOTMEM_ALLOC_ACCESSIBLE, node);
}
@@ -56,11 +56,11 @@ void * __meminit vmemmap_alloc_block(unsigned long size, int node)
if (node_state(node, N_HIGH_MEMORY))
page = alloc_pages_node(
- node, GFP_KERNEL | __GFP_ZERO | __GFP_RETRY_MAYFAIL,
+ node, GFP_KERNEL | __GFP_RETRY_MAYFAIL,
get_order(size));
else
page = alloc_pages(
- GFP_KERNEL | __GFP_ZERO | __GFP_RETRY_MAYFAIL,
+ GFP_KERNEL | __GFP_RETRY_MAYFAIL,
get_order(size));
if (page)
return page_address(page);
@@ -188,7 +188,7 @@ pmd_t * __meminit vmemmap_pmd_populate(pud_t *pud, unsigned long addr, int node)
{
pmd_t *pmd = pmd_offset(pud, addr);
if (pmd_none(*pmd)) {
- void *p = vmemmap_alloc_block(PAGE_SIZE, node);
+ void *p = vmemmap_alloc_block_zero(PAGE_SIZE, node);
if (!p)
return NULL;
pmd_populate_kernel(&init_mm, pmd, p);
@@ -200,7 +200,7 @@ pud_t * __meminit vmemmap_pud_populate(p4d_t *p4d, unsigned long addr, int node)
{
pud_t *pud = pud_offset(p4d, addr);
if (pud_none(*pud)) {
- void *p = vmemmap_alloc_block(PAGE_SIZE, node);
+ void *p = vmemmap_alloc_block_zero(PAGE_SIZE, node);
if (!p)
return NULL;
pud_populate(&init_mm, pud, p);
@@ -212,7 +212,7 @@ p4d_t * __meminit vmemmap_p4d_populate(pgd_t *pgd, unsigned long addr, int node)
{
p4d_t *p4d = p4d_offset(pgd, addr);
if (p4d_none(*p4d)) {
- void *p = vmemmap_alloc_block(PAGE_SIZE, node);
+ void *p = vmemmap_alloc_block_zero(PAGE_SIZE, node);
if (!p)
return NULL;
p4d_populate(&init_mm, p4d, p);
@@ -224,7 +224,7 @@ pgd_t * __meminit vmemmap_pgd_populate(unsigned long addr, int node)
{
pgd_t *pgd = pgd_offset_k(addr);
if (pgd_none(*pgd)) {
- void *p = vmemmap_alloc_block(PAGE_SIZE, node);
+ void *p = vmemmap_alloc_block_zero(PAGE_SIZE, node);
if (!p)
return NULL;
pgd_populate(&init_mm, pgd, p);
diff --git a/mm/sparse.c b/mm/sparse.c
index 7b4be3fd5cac..0e315766ad11 100644
--- a/mm/sparse.c
+++ b/mm/sparse.c
@@ -441,9 +441,9 @@ void __init sparse_mem_maps_populate_node(struct page **map_map,
}
size = PAGE_ALIGN(size);
- map = memblock_virt_alloc_try_nid(size * map_count,
- PAGE_SIZE, __pa(MAX_DMA_ADDRESS),
- BOOTMEM_ALLOC_ACCESSIBLE, nodeid);
+ map = memblock_virt_alloc_try_nid_raw(size * map_count,
+ PAGE_SIZE, __pa(MAX_DMA_ADDRESS),
+ BOOTMEM_ALLOC_ACCESSIBLE, nodeid);
if (map) {
for (pnum = pnum_begin; pnum < pnum_end; pnum++) {
if (!present_section_nr(pnum))
--
2.14.1
^ permalink raw reply related [flat|nested] 79+ messages in thread
* [PATCH v7 11/11] mm: stop zeroing memory during allocation in vmemmap
@ 2017-08-29 2:02 ` Pavel Tatashin
0 siblings, 0 replies; 79+ messages in thread
From: Pavel Tatashin @ 2017-08-29 2:02 UTC (permalink / raw)
To: linux-arm-kernel
vmemmap_alloc_block() will no longer zero the block, so zero memory
at its call sites for everything except struct pages. Struct page memory
is zero'd by struct page initialization.
Replace allocators in sprase-vmemmap to use the non-zeroing version. So,
we will get the performance improvement by zeroing the memory in parallel
when struct pages are zeroed.
Signed-off-by: Pavel Tatashin <pasha.tatashin@oracle.com>
Reviewed-by: Steven Sistare <steven.sistare@oracle.com>
Reviewed-by: Daniel Jordan <daniel.m.jordan@oracle.com>
Reviewed-by: Bob Picco <bob.picco@oracle.com>
---
include/linux/mm.h | 11 +++++++++++
mm/sparse-vmemmap.c | 14 +++++++-------
mm/sparse.c | 6 +++---
3 files changed, 21 insertions(+), 10 deletions(-)
diff --git a/include/linux/mm.h b/include/linux/mm.h
index 0a440ff8f226..fba540aef1da 100644
--- a/include/linux/mm.h
+++ b/include/linux/mm.h
@@ -2456,6 +2456,17 @@ static inline void *vmemmap_alloc_block_buf(unsigned long size, int node)
return __vmemmap_alloc_block_buf(size, node, NULL);
}
+static inline void *vmemmap_alloc_block_zero(unsigned long size, int node)
+{
+ void *p = vmemmap_alloc_block(size, node);
+
+ if (!p)
+ return NULL;
+ memset(p, 0, size);
+
+ return p;
+}
+
void vmemmap_verify(pte_t *, int, unsigned long, unsigned long);
int vmemmap_populate_basepages(unsigned long start, unsigned long end,
int node);
diff --git a/mm/sparse-vmemmap.c b/mm/sparse-vmemmap.c
index c50b1a14d55e..423d4da85a91 100644
--- a/mm/sparse-vmemmap.c
+++ b/mm/sparse-vmemmap.c
@@ -41,7 +41,7 @@ static void * __ref __earlyonly_bootmem_alloc(int node,
unsigned long align,
unsigned long goal)
{
- return memblock_virt_alloc_try_nid(size, align, goal,
+ return memblock_virt_alloc_try_nid_raw(size, align, goal,
BOOTMEM_ALLOC_ACCESSIBLE, node);
}
@@ -56,11 +56,11 @@ void * __meminit vmemmap_alloc_block(unsigned long size, int node)
if (node_state(node, N_HIGH_MEMORY))
page = alloc_pages_node(
- node, GFP_KERNEL | __GFP_ZERO | __GFP_RETRY_MAYFAIL,
+ node, GFP_KERNEL | __GFP_RETRY_MAYFAIL,
get_order(size));
else
page = alloc_pages(
- GFP_KERNEL | __GFP_ZERO | __GFP_RETRY_MAYFAIL,
+ GFP_KERNEL | __GFP_RETRY_MAYFAIL,
get_order(size));
if (page)
return page_address(page);
@@ -188,7 +188,7 @@ pmd_t * __meminit vmemmap_pmd_populate(pud_t *pud, unsigned long addr, int node)
{
pmd_t *pmd = pmd_offset(pud, addr);
if (pmd_none(*pmd)) {
- void *p = vmemmap_alloc_block(PAGE_SIZE, node);
+ void *p = vmemmap_alloc_block_zero(PAGE_SIZE, node);
if (!p)
return NULL;
pmd_populate_kernel(&init_mm, pmd, p);
@@ -200,7 +200,7 @@ pud_t * __meminit vmemmap_pud_populate(p4d_t *p4d, unsigned long addr, int node)
{
pud_t *pud = pud_offset(p4d, addr);
if (pud_none(*pud)) {
- void *p = vmemmap_alloc_block(PAGE_SIZE, node);
+ void *p = vmemmap_alloc_block_zero(PAGE_SIZE, node);
if (!p)
return NULL;
pud_populate(&init_mm, pud, p);
@@ -212,7 +212,7 @@ p4d_t * __meminit vmemmap_p4d_populate(pgd_t *pgd, unsigned long addr, int node)
{
p4d_t *p4d = p4d_offset(pgd, addr);
if (p4d_none(*p4d)) {
- void *p = vmemmap_alloc_block(PAGE_SIZE, node);
+ void *p = vmemmap_alloc_block_zero(PAGE_SIZE, node);
if (!p)
return NULL;
p4d_populate(&init_mm, p4d, p);
@@ -224,7 +224,7 @@ pgd_t * __meminit vmemmap_pgd_populate(unsigned long addr, int node)
{
pgd_t *pgd = pgd_offset_k(addr);
if (pgd_none(*pgd)) {
- void *p = vmemmap_alloc_block(PAGE_SIZE, node);
+ void *p = vmemmap_alloc_block_zero(PAGE_SIZE, node);
if (!p)
return NULL;
pgd_populate(&init_mm, pgd, p);
diff --git a/mm/sparse.c b/mm/sparse.c
index 7b4be3fd5cac..0e315766ad11 100644
--- a/mm/sparse.c
+++ b/mm/sparse.c
@@ -441,9 +441,9 @@ void __init sparse_mem_maps_populate_node(struct page **map_map,
}
size = PAGE_ALIGN(size);
- map = memblock_virt_alloc_try_nid(size * map_count,
- PAGE_SIZE, __pa(MAX_DMA_ADDRESS),
- BOOTMEM_ALLOC_ACCESSIBLE, nodeid);
+ map = memblock_virt_alloc_try_nid_raw(size * map_count,
+ PAGE_SIZE, __pa(MAX_DMA_ADDRESS),
+ BOOTMEM_ALLOC_ACCESSIBLE, nodeid);
if (map) {
for (pnum = pnum_begin; pnum < pnum_end; pnum++) {
if (!present_section_nr(pnum))
--
2.14.1
^ permalink raw reply related [flat|nested] 79+ messages in thread
* [PATCH v7 11/11] mm: stop zeroing memory during allocation in vmemmap
@ 2017-08-29 2:02 ` Pavel Tatashin
0 siblings, 0 replies; 79+ messages in thread
From: Pavel Tatashin @ 2017-08-29 2:02 UTC (permalink / raw)
To: linux-kernel, sparclinux, linux-mm, linuxppc-dev, linux-s390,
linux-arm-kernel, x86, kasan-dev, borntraeger, heiko.carstens,
davem, willy, mhocko, ard.biesheuvel, will.deacon,
catalin.marinas, sam, mgorman, Steven.Sistare, daniel.m.jordan,
bob.picco
vmemmap_alloc_block() will no longer zero the block, so zero memory
at its call sites for everything except struct pages. Struct page memory
is zero'd by struct page initialization.
Replace allocators in sprase-vmemmap to use the non-zeroing version. So,
we will get the performance improvement by zeroing the memory in parallel
when struct pages are zeroed.
Signed-off-by: Pavel Tatashin <pasha.tatashin@oracle.com>
Reviewed-by: Steven Sistare <steven.sistare@oracle.com>
Reviewed-by: Daniel Jordan <daniel.m.jordan@oracle.com>
Reviewed-by: Bob Picco <bob.picco@oracle.com>
---
include/linux/mm.h | 11 +++++++++++
mm/sparse-vmemmap.c | 14 +++++++-------
mm/sparse.c | 6 +++---
3 files changed, 21 insertions(+), 10 deletions(-)
diff --git a/include/linux/mm.h b/include/linux/mm.h
index 0a440ff8f226..fba540aef1da 100644
--- a/include/linux/mm.h
+++ b/include/linux/mm.h
@@ -2456,6 +2456,17 @@ static inline void *vmemmap_alloc_block_buf(unsigned long size, int node)
return __vmemmap_alloc_block_buf(size, node, NULL);
}
+static inline void *vmemmap_alloc_block_zero(unsigned long size, int node)
+{
+ void *p = vmemmap_alloc_block(size, node);
+
+ if (!p)
+ return NULL;
+ memset(p, 0, size);
+
+ return p;
+}
+
void vmemmap_verify(pte_t *, int, unsigned long, unsigned long);
int vmemmap_populate_basepages(unsigned long start, unsigned long end,
int node);
diff --git a/mm/sparse-vmemmap.c b/mm/sparse-vmemmap.c
index c50b1a14d55e..423d4da85a91 100644
--- a/mm/sparse-vmemmap.c
+++ b/mm/sparse-vmemmap.c
@@ -41,7 +41,7 @@ static void * __ref __earlyonly_bootmem_alloc(int node,
unsigned long align,
unsigned long goal)
{
- return memblock_virt_alloc_try_nid(size, align, goal,
+ return memblock_virt_alloc_try_nid_raw(size, align, goal,
BOOTMEM_ALLOC_ACCESSIBLE, node);
}
@@ -56,11 +56,11 @@ void * __meminit vmemmap_alloc_block(unsigned long size, int node)
if (node_state(node, N_HIGH_MEMORY))
page = alloc_pages_node(
- node, GFP_KERNEL | __GFP_ZERO | __GFP_RETRY_MAYFAIL,
+ node, GFP_KERNEL | __GFP_RETRY_MAYFAIL,
get_order(size));
else
page = alloc_pages(
- GFP_KERNEL | __GFP_ZERO | __GFP_RETRY_MAYFAIL,
+ GFP_KERNEL | __GFP_RETRY_MAYFAIL,
get_order(size));
if (page)
return page_address(page);
@@ -188,7 +188,7 @@ pmd_t * __meminit vmemmap_pmd_populate(pud_t *pud, unsigned long addr, int node)
{
pmd_t *pmd = pmd_offset(pud, addr);
if (pmd_none(*pmd)) {
- void *p = vmemmap_alloc_block(PAGE_SIZE, node);
+ void *p = vmemmap_alloc_block_zero(PAGE_SIZE, node);
if (!p)
return NULL;
pmd_populate_kernel(&init_mm, pmd, p);
@@ -200,7 +200,7 @@ pud_t * __meminit vmemmap_pud_populate(p4d_t *p4d, unsigned long addr, int node)
{
pud_t *pud = pud_offset(p4d, addr);
if (pud_none(*pud)) {
- void *p = vmemmap_alloc_block(PAGE_SIZE, node);
+ void *p = vmemmap_alloc_block_zero(PAGE_SIZE, node);
if (!p)
return NULL;
pud_populate(&init_mm, pud, p);
@@ -212,7 +212,7 @@ p4d_t * __meminit vmemmap_p4d_populate(pgd_t *pgd, unsigned long addr, int node)
{
p4d_t *p4d = p4d_offset(pgd, addr);
if (p4d_none(*p4d)) {
- void *p = vmemmap_alloc_block(PAGE_SIZE, node);
+ void *p = vmemmap_alloc_block_zero(PAGE_SIZE, node);
if (!p)
return NULL;
p4d_populate(&init_mm, p4d, p);
@@ -224,7 +224,7 @@ pgd_t * __meminit vmemmap_pgd_populate(unsigned long addr, int node)
{
pgd_t *pgd = pgd_offset_k(addr);
if (pgd_none(*pgd)) {
- void *p = vmemmap_alloc_block(PAGE_SIZE, node);
+ void *p = vmemmap_alloc_block_zero(PAGE_SIZE, node);
if (!p)
return NULL;
pgd_populate(&init_mm, pgd, p);
diff --git a/mm/sparse.c b/mm/sparse.c
index 7b4be3fd5cac..0e315766ad11 100644
--- a/mm/sparse.c
+++ b/mm/sparse.c
@@ -441,9 +441,9 @@ void __init sparse_mem_maps_populate_node(struct page **map_map,
}
size = PAGE_ALIGN(size);
- map = memblock_virt_alloc_try_nid(size * map_count,
- PAGE_SIZE, __pa(MAX_DMA_ADDRESS),
- BOOTMEM_ALLOC_ACCESSIBLE, nodeid);
+ map = memblock_virt_alloc_try_nid_raw(size * map_count,
+ PAGE_SIZE, __pa(MAX_DMA_ADDRESS),
+ BOOTMEM_ALLOC_ACCESSIBLE, nodeid);
if (map) {
for (pnum = pnum_begin; pnum < pnum_end; pnum++) {
if (!present_section_nr(pnum))
--
2.14.1
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply related [flat|nested] 79+ messages in thread
* [PATCH v7 11/11] mm: stop zeroing memory during allocation in vmemmap
@ 2017-08-29 2:02 ` Pavel Tatashin
0 siblings, 0 replies; 79+ messages in thread
From: Pavel Tatashin @ 2017-08-29 2:02 UTC (permalink / raw)
To: linux-arm-kernel
vmemmap_alloc_block() will no longer zero the block, so zero memory
at its call sites for everything except struct pages. Struct page memory
is zero'd by struct page initialization.
Replace allocators in sprase-vmemmap to use the non-zeroing version. So,
we will get the performance improvement by zeroing the memory in parallel
when struct pages are zeroed.
Signed-off-by: Pavel Tatashin <pasha.tatashin@oracle.com>
Reviewed-by: Steven Sistare <steven.sistare@oracle.com>
Reviewed-by: Daniel Jordan <daniel.m.jordan@oracle.com>
Reviewed-by: Bob Picco <bob.picco@oracle.com>
---
include/linux/mm.h | 11 +++++++++++
mm/sparse-vmemmap.c | 14 +++++++-------
mm/sparse.c | 6 +++---
3 files changed, 21 insertions(+), 10 deletions(-)
diff --git a/include/linux/mm.h b/include/linux/mm.h
index 0a440ff8f226..fba540aef1da 100644
--- a/include/linux/mm.h
+++ b/include/linux/mm.h
@@ -2456,6 +2456,17 @@ static inline void *vmemmap_alloc_block_buf(unsigned long size, int node)
return __vmemmap_alloc_block_buf(size, node, NULL);
}
+static inline void *vmemmap_alloc_block_zero(unsigned long size, int node)
+{
+ void *p = vmemmap_alloc_block(size, node);
+
+ if (!p)
+ return NULL;
+ memset(p, 0, size);
+
+ return p;
+}
+
void vmemmap_verify(pte_t *, int, unsigned long, unsigned long);
int vmemmap_populate_basepages(unsigned long start, unsigned long end,
int node);
diff --git a/mm/sparse-vmemmap.c b/mm/sparse-vmemmap.c
index c50b1a14d55e..423d4da85a91 100644
--- a/mm/sparse-vmemmap.c
+++ b/mm/sparse-vmemmap.c
@@ -41,7 +41,7 @@ static void * __ref __earlyonly_bootmem_alloc(int node,
unsigned long align,
unsigned long goal)
{
- return memblock_virt_alloc_try_nid(size, align, goal,
+ return memblock_virt_alloc_try_nid_raw(size, align, goal,
BOOTMEM_ALLOC_ACCESSIBLE, node);
}
@@ -56,11 +56,11 @@ void * __meminit vmemmap_alloc_block(unsigned long size, int node)
if (node_state(node, N_HIGH_MEMORY))
page = alloc_pages_node(
- node, GFP_KERNEL | __GFP_ZERO | __GFP_RETRY_MAYFAIL,
+ node, GFP_KERNEL | __GFP_RETRY_MAYFAIL,
get_order(size));
else
page = alloc_pages(
- GFP_KERNEL | __GFP_ZERO | __GFP_RETRY_MAYFAIL,
+ GFP_KERNEL | __GFP_RETRY_MAYFAIL,
get_order(size));
if (page)
return page_address(page);
@@ -188,7 +188,7 @@ pmd_t * __meminit vmemmap_pmd_populate(pud_t *pud, unsigned long addr, int node)
{
pmd_t *pmd = pmd_offset(pud, addr);
if (pmd_none(*pmd)) {
- void *p = vmemmap_alloc_block(PAGE_SIZE, node);
+ void *p = vmemmap_alloc_block_zero(PAGE_SIZE, node);
if (!p)
return NULL;
pmd_populate_kernel(&init_mm, pmd, p);
@@ -200,7 +200,7 @@ pud_t * __meminit vmemmap_pud_populate(p4d_t *p4d, unsigned long addr, int node)
{
pud_t *pud = pud_offset(p4d, addr);
if (pud_none(*pud)) {
- void *p = vmemmap_alloc_block(PAGE_SIZE, node);
+ void *p = vmemmap_alloc_block_zero(PAGE_SIZE, node);
if (!p)
return NULL;
pud_populate(&init_mm, pud, p);
@@ -212,7 +212,7 @@ p4d_t * __meminit vmemmap_p4d_populate(pgd_t *pgd, unsigned long addr, int node)
{
p4d_t *p4d = p4d_offset(pgd, addr);
if (p4d_none(*p4d)) {
- void *p = vmemmap_alloc_block(PAGE_SIZE, node);
+ void *p = vmemmap_alloc_block_zero(PAGE_SIZE, node);
if (!p)
return NULL;
p4d_populate(&init_mm, p4d, p);
@@ -224,7 +224,7 @@ pgd_t * __meminit vmemmap_pgd_populate(unsigned long addr, int node)
{
pgd_t *pgd = pgd_offset_k(addr);
if (pgd_none(*pgd)) {
- void *p = vmemmap_alloc_block(PAGE_SIZE, node);
+ void *p = vmemmap_alloc_block_zero(PAGE_SIZE, node);
if (!p)
return NULL;
pgd_populate(&init_mm, pgd, p);
diff --git a/mm/sparse.c b/mm/sparse.c
index 7b4be3fd5cac..0e315766ad11 100644
--- a/mm/sparse.c
+++ b/mm/sparse.c
@@ -441,9 +441,9 @@ void __init sparse_mem_maps_populate_node(struct page **map_map,
}
size = PAGE_ALIGN(size);
- map = memblock_virt_alloc_try_nid(size * map_count,
- PAGE_SIZE, __pa(MAX_DMA_ADDRESS),
- BOOTMEM_ALLOC_ACCESSIBLE, nodeid);
+ map = memblock_virt_alloc_try_nid_raw(size * map_count,
+ PAGE_SIZE, __pa(MAX_DMA_ADDRESS),
+ BOOTMEM_ALLOC_ACCESSIBLE, nodeid);
if (map) {
for (pnum = pnum_begin; pnum < pnum_end; pnum++) {
if (!present_section_nr(pnum))
--
2.14.1
^ permalink raw reply related [flat|nested] 79+ messages in thread
* Re: [PATCH v7 04/11] sparc64: simplify vmemmap_populate
2017-08-29 2:02 ` Pavel Tatashin
(?)
(?)
@ 2017-08-30 1:08 ` David Miller
-1 siblings, 0 replies; 79+ messages in thread
From: David Miller @ 2017-08-30 1:08 UTC (permalink / raw)
To: pasha.tatashin
Cc: linux-kernel, sparclinux, linux-mm, linuxppc-dev, linux-s390,
linux-arm-kernel, x86, kasan-dev, borntraeger, heiko.carstens,
willy, mhocko, ard.biesheuvel, will.deacon, catalin.marinas, sam,
mgorman, Steven.Sistare, daniel.m.jordan, bob.picco
From: Pavel Tatashin <pasha.tatashin@oracle.com>
Date: Mon, 28 Aug 2017 22:02:15 -0400
> Remove duplicating code by using common functions
> vmemmap_pud_populate and vmemmap_pgd_populate.
>
> Signed-off-by: Pavel Tatashin <pasha.tatashin@oracle.com>
> Reviewed-by: Steven Sistare <steven.sistare@oracle.com>
> Reviewed-by: Daniel Jordan <daniel.m.jordan@oracle.com>
> Reviewed-by: Bob Picco <bob.picco@oracle.com>
Acked-by: David S. Miller <davem@davemloft.net>
^ permalink raw reply [flat|nested] 79+ messages in thread
* Re: [PATCH v7 04/11] sparc64: simplify vmemmap_populate
@ 2017-08-30 1:08 ` David Miller
0 siblings, 0 replies; 79+ messages in thread
From: David Miller @ 2017-08-30 1:08 UTC (permalink / raw)
To: linux-arm-kernel
From: Pavel Tatashin <pasha.tatashin@oracle.com>
Date: Mon, 28 Aug 2017 22:02:15 -0400
> Remove duplicating code by using common functions
> vmemmap_pud_populate and vmemmap_pgd_populate.
>
> Signed-off-by: Pavel Tatashin <pasha.tatashin@oracle.com>
> Reviewed-by: Steven Sistare <steven.sistare@oracle.com>
> Reviewed-by: Daniel Jordan <daniel.m.jordan@oracle.com>
> Reviewed-by: Bob Picco <bob.picco@oracle.com>
Acked-by: David S. Miller <davem@davemloft.net>
^ permalink raw reply [flat|nested] 79+ messages in thread
* Re: [PATCH v7 04/11] sparc64: simplify vmemmap_populate
@ 2017-08-30 1:08 ` David Miller
0 siblings, 0 replies; 79+ messages in thread
From: David Miller @ 2017-08-30 1:08 UTC (permalink / raw)
To: pasha.tatashin
Cc: linux-kernel, sparclinux, linux-mm, linuxppc-dev, linux-s390,
linux-arm-kernel, x86, kasan-dev, borntraeger, heiko.carstens,
willy, mhocko, ard.biesheuvel, will.deacon, catalin.marinas, sam,
mgorman, Steven.Sistare, daniel.m.jordan, bob.picco
From: Pavel Tatashin <pasha.tatashin@oracle.com>
Date: Mon, 28 Aug 2017 22:02:15 -0400
> Remove duplicating code by using common functions
> vmemmap_pud_populate and vmemmap_pgd_populate.
>
> Signed-off-by: Pavel Tatashin <pasha.tatashin@oracle.com>
> Reviewed-by: Steven Sistare <steven.sistare@oracle.com>
> Reviewed-by: Daniel Jordan <daniel.m.jordan@oracle.com>
> Reviewed-by: Bob Picco <bob.picco@oracle.com>
Acked-by: David S. Miller <davem@davemloft.net>
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 79+ messages in thread
* [PATCH v7 04/11] sparc64: simplify vmemmap_populate
@ 2017-08-30 1:08 ` David Miller
0 siblings, 0 replies; 79+ messages in thread
From: David Miller @ 2017-08-30 1:08 UTC (permalink / raw)
To: linux-arm-kernel
From: Pavel Tatashin <pasha.tatashin@oracle.com>
Date: Mon, 28 Aug 2017 22:02:15 -0400
> Remove duplicating code by using common functions
> vmemmap_pud_populate and vmemmap_pgd_populate.
>
> Signed-off-by: Pavel Tatashin <pasha.tatashin@oracle.com>
> Reviewed-by: Steven Sistare <steven.sistare@oracle.com>
> Reviewed-by: Daniel Jordan <daniel.m.jordan@oracle.com>
> Reviewed-by: Bob Picco <bob.picco@oracle.com>
Acked-by: David S. Miller <davem@davemloft.net>
^ permalink raw reply [flat|nested] 79+ messages in thread
* Re: [PATCH v7 02/11] sparc64/mm: setting fields in deferred pages
2017-08-29 2:02 ` Pavel Tatashin
(?)
(?)
@ 2017-08-30 1:09 ` David Miller
-1 siblings, 0 replies; 79+ messages in thread
From: David Miller @ 2017-08-30 1:09 UTC (permalink / raw)
To: pasha.tatashin
Cc: linux-kernel, sparclinux, linux-mm, linuxppc-dev, linux-s390,
linux-arm-kernel, x86, kasan-dev, borntraeger, heiko.carstens,
willy, mhocko, ard.biesheuvel, will.deacon, catalin.marinas, sam,
mgorman, Steven.Sistare, daniel.m.jordan, bob.picco
From: Pavel Tatashin <pasha.tatashin@oracle.com>
Date: Mon, 28 Aug 2017 22:02:13 -0400
> Without deferred struct page feature (CONFIG_DEFERRED_STRUCT_PAGE_INIT),
> flags and other fields in "struct page"es are never changed prior to first
> initializing struct pages by going through __init_single_page().
>
> With deferred struct page feature enabled there is a case where we set some
> fields prior to initializing:
>
> mem_init() {
> register_page_bootmem_info();
> free_all_bootmem();
> ...
> }
>
> When register_page_bootmem_info() is called only non-deferred struct pages
> are initialized. But, this function goes through some reserved pages which
> might be part of the deferred, and thus are not yet initialized.
>
> mem_init
> register_page_bootmem_info
> register_page_bootmem_info_node
> get_page_bootmem
> .. setting fields here ..
> such as: page->freelist = (void *)type;
>
> free_all_bootmem()
> free_low_memory_core_early()
> for_each_reserved_mem_region()
> reserve_bootmem_region()
> init_reserved_page() <- Only if this is deferred reserved page
> __init_single_pfn()
> __init_single_page()
> memset(0) <-- Loose the set fields here
>
> We end-up with similar issue as in the previous patch, where currently we
> do not observe problem as memory is zeroed. But, if flag asserts are
> changed we can start hitting issues.
>
> Also, because in this patch series we will stop zeroing struct page memory
> during allocation, we must make sure that struct pages are properly
> initialized prior to using them.
>
> The deferred-reserved pages are initialized in free_all_bootmem().
> Therefore, the fix is to switch the above calls.
>
> Signed-off-by: Pavel Tatashin <pasha.tatashin@oracle.com>
> Reviewed-by: Steven Sistare <steven.sistare@oracle.com>
> Reviewed-by: Daniel Jordan <daniel.m.jordan@oracle.com>
> Reviewed-by: Bob Picco <bob.picco@oracle.com>
Acked-by: David S. Miller <davem@davemloft.net>
^ permalink raw reply [flat|nested] 79+ messages in thread
* Re: [PATCH v7 02/11] sparc64/mm: setting fields in deferred pages
@ 2017-08-30 1:09 ` David Miller
0 siblings, 0 replies; 79+ messages in thread
From: David Miller @ 2017-08-30 1:09 UTC (permalink / raw)
To: linux-arm-kernel
From: Pavel Tatashin <pasha.tatashin@oracle.com>
Date: Mon, 28 Aug 2017 22:02:13 -0400
> Without deferred struct page feature (CONFIG_DEFERRED_STRUCT_PAGE_INIT),
> flags and other fields in "struct page"es are never changed prior to first
> initializing struct pages by going through __init_single_page().
>
> With deferred struct page feature enabled there is a case where we set some
> fields prior to initializing:
>
> mem_init() {
> register_page_bootmem_info();
> free_all_bootmem();
> ...
> }
>
> When register_page_bootmem_info() is called only non-deferred struct pages
> are initialized. But, this function goes through some reserved pages which
> might be part of the deferred, and thus are not yet initialized.
>
> mem_init
> register_page_bootmem_info
> register_page_bootmem_info_node
> get_page_bootmem
> .. setting fields here ..
> such as: page->freelist = (void *)type;
>
> free_all_bootmem()
> free_low_memory_core_early()
> for_each_reserved_mem_region()
> reserve_bootmem_region()
> init_reserved_page() <- Only if this is deferred reserved page
> __init_single_pfn()
> __init_single_page()
> memset(0) <-- Loose the set fields here
>
> We end-up with similar issue as in the previous patch, where currently we
> do not observe problem as memory is zeroed. But, if flag asserts are
> changed we can start hitting issues.
>
> Also, because in this patch series we will stop zeroing struct page memory
> during allocation, we must make sure that struct pages are properly
> initialized prior to using them.
>
> The deferred-reserved pages are initialized in free_all_bootmem().
> Therefore, the fix is to switch the above calls.
>
> Signed-off-by: Pavel Tatashin <pasha.tatashin@oracle.com>
> Reviewed-by: Steven Sistare <steven.sistare@oracle.com>
> Reviewed-by: Daniel Jordan <daniel.m.jordan@oracle.com>
> Reviewed-by: Bob Picco <bob.picco@oracle.com>
Acked-by: David S. Miller <davem@davemloft.net>
^ permalink raw reply [flat|nested] 79+ messages in thread
* Re: [PATCH v7 02/11] sparc64/mm: setting fields in deferred pages
@ 2017-08-30 1:09 ` David Miller
0 siblings, 0 replies; 79+ messages in thread
From: David Miller @ 2017-08-30 1:09 UTC (permalink / raw)
To: pasha.tatashin
Cc: linux-kernel, sparclinux, linux-mm, linuxppc-dev, linux-s390,
linux-arm-kernel, x86, kasan-dev, borntraeger, heiko.carstens,
willy, mhocko, ard.biesheuvel, will.deacon, catalin.marinas, sam,
mgorman, Steven.Sistare, daniel.m.jordan, bob.picco
From: Pavel Tatashin <pasha.tatashin@oracle.com>
Date: Mon, 28 Aug 2017 22:02:13 -0400
> Without deferred struct page feature (CONFIG_DEFERRED_STRUCT_PAGE_INIT),
> flags and other fields in "struct page"es are never changed prior to first
> initializing struct pages by going through __init_single_page().
>
> With deferred struct page feature enabled there is a case where we set some
> fields prior to initializing:
>
> mem_init() {
> register_page_bootmem_info();
> free_all_bootmem();
> ...
> }
>
> When register_page_bootmem_info() is called only non-deferred struct pages
> are initialized. But, this function goes through some reserved pages which
> might be part of the deferred, and thus are not yet initialized.
>
> mem_init
> register_page_bootmem_info
> register_page_bootmem_info_node
> get_page_bootmem
> .. setting fields here ..
> such as: page->freelist = (void *)type;
>
> free_all_bootmem()
> free_low_memory_core_early()
> for_each_reserved_mem_region()
> reserve_bootmem_region()
> init_reserved_page() <- Only if this is deferred reserved page
> __init_single_pfn()
> __init_single_page()
> memset(0) <-- Loose the set fields here
>
> We end-up with similar issue as in the previous patch, where currently we
> do not observe problem as memory is zeroed. But, if flag asserts are
> changed we can start hitting issues.
>
> Also, because in this patch series we will stop zeroing struct page memory
> during allocation, we must make sure that struct pages are properly
> initialized prior to using them.
>
> The deferred-reserved pages are initialized in free_all_bootmem().
> Therefore, the fix is to switch the above calls.
>
> Signed-off-by: Pavel Tatashin <pasha.tatashin@oracle.com>
> Reviewed-by: Steven Sistare <steven.sistare@oracle.com>
> Reviewed-by: Daniel Jordan <daniel.m.jordan@oracle.com>
> Reviewed-by: Bob Picco <bob.picco@oracle.com>
Acked-by: David S. Miller <davem@davemloft.net>
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 79+ messages in thread
* [PATCH v7 02/11] sparc64/mm: setting fields in deferred pages
@ 2017-08-30 1:09 ` David Miller
0 siblings, 0 replies; 79+ messages in thread
From: David Miller @ 2017-08-30 1:09 UTC (permalink / raw)
To: linux-arm-kernel
From: Pavel Tatashin <pasha.tatashin@oracle.com>
Date: Mon, 28 Aug 2017 22:02:13 -0400
> Without deferred struct page feature (CONFIG_DEFERRED_STRUCT_PAGE_INIT),
> flags and other fields in "struct page"es are never changed prior to first
> initializing struct pages by going through __init_single_page().
>
> With deferred struct page feature enabled there is a case where we set some
> fields prior to initializing:
>
> mem_init() {
> register_page_bootmem_info();
> free_all_bootmem();
> ...
> }
>
> When register_page_bootmem_info() is called only non-deferred struct pages
> are initialized. But, this function goes through some reserved pages which
> might be part of the deferred, and thus are not yet initialized.
>
> mem_init
> register_page_bootmem_info
> register_page_bootmem_info_node
> get_page_bootmem
> .. setting fields here ..
> such as: page->freelist = (void *)type;
>
> free_all_bootmem()
> free_low_memory_core_early()
> for_each_reserved_mem_region()
> reserve_bootmem_region()
> init_reserved_page() <- Only if this is deferred reserved page
> __init_single_pfn()
> __init_single_page()
> memset(0) <-- Loose the set fields here
>
> We end-up with similar issue as in the previous patch, where currently we
> do not observe problem as memory is zeroed. But, if flag asserts are
> changed we can start hitting issues.
>
> Also, because in this patch series we will stop zeroing struct page memory
> during allocation, we must make sure that struct pages are properly
> initialized prior to using them.
>
> The deferred-reserved pages are initialized in free_all_bootmem().
> Therefore, the fix is to switch the above calls.
>
> Signed-off-by: Pavel Tatashin <pasha.tatashin@oracle.com>
> Reviewed-by: Steven Sistare <steven.sistare@oracle.com>
> Reviewed-by: Daniel Jordan <daniel.m.jordan@oracle.com>
> Reviewed-by: Bob Picco <bob.picco@oracle.com>
Acked-by: David S. Miller <davem@davemloft.net>
^ permalink raw reply [flat|nested] 79+ messages in thread
* Re: [PATCH v7 07/11] sparc64: optimized struct page zeroing
2017-08-29 2:02 ` Pavel Tatashin
(?)
(?)
@ 2017-08-30 1:12 ` David Miller
-1 siblings, 0 replies; 79+ messages in thread
From: David Miller @ 2017-08-30 1:12 UTC (permalink / raw)
To: pasha.tatashin
Cc: linux-kernel, sparclinux, linux-mm, linuxppc-dev, linux-s390,
linux-arm-kernel, x86, kasan-dev, borntraeger, heiko.carstens,
willy, mhocko, ard.biesheuvel, will.deacon, catalin.marinas, sam,
mgorman, Steven.Sistare, daniel.m.jordan, bob.picco
From: Pavel Tatashin <pasha.tatashin@oracle.com>
Date: Mon, 28 Aug 2017 22:02:18 -0400
> Add an optimized mm_zero_struct_page(), so struct page's are zeroed without
> calling memset(). We do eight to ten regular stores based on the size of
> struct page. Compiler optimizes out the conditions of switch() statement.
>
> SPARC-M6 with 15T of memory, single thread performance:
>
> BASE FIX OPTIMIZED_FIX
> bootmem_init 28.440467985s 2.305674818s 2.305161615s
> free_area_init_nodes 202.845901673s 225.343084508s 172.556506560s
> --------------------------------------------
> Total 231.286369658s 227.648759326s 174.861668175s
>
> BASE: current linux
> FIX: This patch series without "optimized struct page zeroing"
> OPTIMIZED_FIX: This patch series including the current patch.
>
> bootmem_init() is where memory for struct pages is zeroed during
> allocation. Note, about two seconds in this function is a fixed time: it
> does not increase as memory is increased.
>
> Signed-off-by: Pavel Tatashin <pasha.tatashin@oracle.com>
> Reviewed-by: Steven Sistare <steven.sistare@oracle.com>
> Reviewed-by: Daniel Jordan <daniel.m.jordan@oracle.com>
> Reviewed-by: Bob Picco <bob.picco@oracle.com>
You should probably use initializing stores when you are doing 8
stores and we thus know the page struct is cache line aligned.
But other than that:
Acked-by: David S. Miller <davem@davemloft.net>
^ permalink raw reply [flat|nested] 79+ messages in thread
* Re: [PATCH v7 07/11] sparc64: optimized struct page zeroing
@ 2017-08-30 1:12 ` David Miller
0 siblings, 0 replies; 79+ messages in thread
From: David Miller @ 2017-08-30 1:12 UTC (permalink / raw)
To: linux-arm-kernel
From: Pavel Tatashin <pasha.tatashin@oracle.com>
Date: Mon, 28 Aug 2017 22:02:18 -0400
> Add an optimized mm_zero_struct_page(), so struct page's are zeroed without
> calling memset(). We do eight to ten regular stores based on the size of
> struct page. Compiler optimizes out the conditions of switch() statement.
>
> SPARC-M6 with 15T of memory, single thread performance:
>
> BASE FIX OPTIMIZED_FIX
> bootmem_init 28.440467985s 2.305674818s 2.305161615s
> free_area_init_nodes 202.845901673s 225.343084508s 172.556506560s
> --------------------------------------------
> Total 231.286369658s 227.648759326s 174.861668175s
>
> BASE: current linux
> FIX: This patch series without "optimized struct page zeroing"
> OPTIMIZED_FIX: This patch series including the current patch.
>
> bootmem_init() is where memory for struct pages is zeroed during
> allocation. Note, about two seconds in this function is a fixed time: it
> does not increase as memory is increased.
>
> Signed-off-by: Pavel Tatashin <pasha.tatashin@oracle.com>
> Reviewed-by: Steven Sistare <steven.sistare@oracle.com>
> Reviewed-by: Daniel Jordan <daniel.m.jordan@oracle.com>
> Reviewed-by: Bob Picco <bob.picco@oracle.com>
You should probably use initializing stores when you are doing 8
stores and we thus know the page struct is cache line aligned.
But other than that:
Acked-by: David S. Miller <davem@davemloft.net>
^ permalink raw reply [flat|nested] 79+ messages in thread
* Re: [PATCH v7 07/11] sparc64: optimized struct page zeroing
@ 2017-08-30 1:12 ` David Miller
0 siblings, 0 replies; 79+ messages in thread
From: David Miller @ 2017-08-30 1:12 UTC (permalink / raw)
To: pasha.tatashin
Cc: linux-kernel, sparclinux, linux-mm, linuxppc-dev, linux-s390,
linux-arm-kernel, x86, kasan-dev, borntraeger, heiko.carstens,
willy, mhocko, ard.biesheuvel, will.deacon, catalin.marinas, sam,
mgorman, Steven.Sistare, daniel.m.jordan, bob.picco
From: Pavel Tatashin <pasha.tatashin@oracle.com>
Date: Mon, 28 Aug 2017 22:02:18 -0400
> Add an optimized mm_zero_struct_page(), so struct page's are zeroed without
> calling memset(). We do eight to ten regular stores based on the size of
> struct page. Compiler optimizes out the conditions of switch() statement.
>
> SPARC-M6 with 15T of memory, single thread performance:
>
> BASE FIX OPTIMIZED_FIX
> bootmem_init 28.440467985s 2.305674818s 2.305161615s
> free_area_init_nodes 202.845901673s 225.343084508s 172.556506560s
> --------------------------------------------
> Total 231.286369658s 227.648759326s 174.861668175s
>
> BASE: current linux
> FIX: This patch series without "optimized struct page zeroing"
> OPTIMIZED_FIX: This patch series including the current patch.
>
> bootmem_init() is where memory for struct pages is zeroed during
> allocation. Note, about two seconds in this function is a fixed time: it
> does not increase as memory is increased.
>
> Signed-off-by: Pavel Tatashin <pasha.tatashin@oracle.com>
> Reviewed-by: Steven Sistare <steven.sistare@oracle.com>
> Reviewed-by: Daniel Jordan <daniel.m.jordan@oracle.com>
> Reviewed-by: Bob Picco <bob.picco@oracle.com>
You should probably use initializing stores when you are doing 8
stores and we thus know the page struct is cache line aligned.
But other than that:
Acked-by: David S. Miller <davem@davemloft.net>
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 79+ messages in thread
* [PATCH v7 07/11] sparc64: optimized struct page zeroing
@ 2017-08-30 1:12 ` David Miller
0 siblings, 0 replies; 79+ messages in thread
From: David Miller @ 2017-08-30 1:12 UTC (permalink / raw)
To: linux-arm-kernel
From: Pavel Tatashin <pasha.tatashin@oracle.com>
Date: Mon, 28 Aug 2017 22:02:18 -0400
> Add an optimized mm_zero_struct_page(), so struct page's are zeroed without
> calling memset(). We do eight to ten regular stores based on the size of
> struct page. Compiler optimizes out the conditions of switch() statement.
>
> SPARC-M6 with 15T of memory, single thread performance:
>
> BASE FIX OPTIMIZED_FIX
> bootmem_init 28.440467985s 2.305674818s 2.305161615s
> free_area_init_nodes 202.845901673s 225.343084508s 172.556506560s
> --------------------------------------------
> Total 231.286369658s 227.648759326s 174.861668175s
>
> BASE: current linux
> FIX: This patch series without "optimized struct page zeroing"
> OPTIMIZED_FIX: This patch series including the current patch.
>
> bootmem_init() is where memory for struct pages is zeroed during
> allocation. Note, about two seconds in this function is a fixed time: it
> does not increase as memory is increased.
>
> Signed-off-by: Pavel Tatashin <pasha.tatashin@oracle.com>
> Reviewed-by: Steven Sistare <steven.sistare@oracle.com>
> Reviewed-by: Daniel Jordan <daniel.m.jordan@oracle.com>
> Reviewed-by: Bob Picco <bob.picco@oracle.com>
You should probably use initializing stores when you are doing 8
stores and we thus know the page struct is cache line aligned.
But other than that:
Acked-by: David S. Miller <davem@davemloft.net>
^ permalink raw reply [flat|nested] 79+ messages in thread
* Re: [PATCH v7 07/11] sparc64: optimized struct page zeroing
2017-08-30 1:12 ` David Miller
(?)
(?)
@ 2017-08-30 13:19 ` Pasha Tatashin
-1 siblings, 0 replies; 79+ messages in thread
From: Pasha Tatashin @ 2017-08-30 13:19 UTC (permalink / raw)
To: David Miller
Cc: linux-kernel, sparclinux, linux-mm, linuxppc-dev, linux-s390,
linux-arm-kernel, x86, kasan-dev, borntraeger, heiko.carstens,
willy, mhocko, ard.biesheuvel, will.deacon, catalin.marinas, sam,
mgorman, Steven.Sistare, daniel.m.jordan, bob.picco
Hi Dave,
Thank you for acking.
The reason I am not doing initializing stores is because they require a
membar, even if only regular stores are following (I hoped to do a
membar before first load). This is something I was thinking was not
true, but after consulting with colleagues and checking processor
manual, I verified that it is the case.
Pasha
>
> You should probably use initializing stores when you are doing 8
> stores and we thus know the page struct is cache line aligned.
>
> But other than that:
>
> Acked-by: David S. Miller <davem@davemloft.net>
^ permalink raw reply [flat|nested] 79+ messages in thread
* Re: [PATCH v7 07/11] sparc64: optimized struct page zeroing
@ 2017-08-30 13:19 ` Pasha Tatashin
0 siblings, 0 replies; 79+ messages in thread
From: Pasha Tatashin @ 2017-08-30 13:19 UTC (permalink / raw)
To: linux-arm-kernel
Hi Dave,
Thank you for acking.
The reason I am not doing initializing stores is because they require a
membar, even if only regular stores are following (I hoped to do a
membar before first load). This is something I was thinking was not
true, but after consulting with colleagues and checking processor
manual, I verified that it is the case.
Pasha
>
> You should probably use initializing stores when you are doing 8
> stores and we thus know the page struct is cache line aligned.
>
> But other than that:
>
> Acked-by: David S. Miller <davem@davemloft.net>
^ permalink raw reply [flat|nested] 79+ messages in thread
* Re: [PATCH v7 07/11] sparc64: optimized struct page zeroing
@ 2017-08-30 13:19 ` Pasha Tatashin
0 siblings, 0 replies; 79+ messages in thread
From: Pasha Tatashin @ 2017-08-30 13:19 UTC (permalink / raw)
To: David Miller
Cc: linux-kernel, sparclinux, linux-mm, linuxppc-dev, linux-s390,
linux-arm-kernel, x86, kasan-dev, borntraeger, heiko.carstens,
willy, mhocko, ard.biesheuvel, will.deacon, catalin.marinas, sam,
mgorman, Steven.Sistare, daniel.m.jordan, bob.picco
Hi Dave,
Thank you for acking.
The reason I am not doing initializing stores is because they require a
membar, even if only regular stores are following (I hoped to do a
membar before first load). This is something I was thinking was not
true, but after consulting with colleagues and checking processor
manual, I verified that it is the case.
Pasha
>
> You should probably use initializing stores when you are doing 8
> stores and we thus know the page struct is cache line aligned.
>
> But other than that:
>
> Acked-by: David S. Miller <davem@davemloft.net>
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 79+ messages in thread
* [PATCH v7 07/11] sparc64: optimized struct page zeroing
@ 2017-08-30 13:19 ` Pasha Tatashin
0 siblings, 0 replies; 79+ messages in thread
From: Pasha Tatashin @ 2017-08-30 13:19 UTC (permalink / raw)
To: linux-arm-kernel
Hi Dave,
Thank you for acking.
The reason I am not doing initializing stores is because they require a
membar, even if only regular stores are following (I hoped to do a
membar before first load). This is something I was thinking was not
true, but after consulting with colleagues and checking processor
manual, I verified that it is the case.
Pasha
>
> You should probably use initializing stores when you are doing 8
> stores and we thus know the page struct is cache line aligned.
>
> But other than that:
>
> Acked-by: David S. Miller <davem@davemloft.net>
^ permalink raw reply [flat|nested] 79+ messages in thread
* Re: [PATCH v7 07/11] sparc64: optimized struct page zeroing
2017-08-30 13:19 ` Pasha Tatashin
(?)
(?)
@ 2017-08-30 17:46 ` David Miller
-1 siblings, 0 replies; 79+ messages in thread
From: David Miller @ 2017-08-30 17:46 UTC (permalink / raw)
To: pasha.tatashin
Cc: linux-kernel, sparclinux, linux-mm, linuxppc-dev, linux-s390,
linux-arm-kernel, x86, kasan-dev, borntraeger, heiko.carstens,
willy, mhocko, ard.biesheuvel, will.deacon, catalin.marinas, sam,
mgorman, Steven.Sistare, daniel.m.jordan, bob.picco
From: Pasha Tatashin <pasha.tatashin@oracle.com>
Date: Wed, 30 Aug 2017 09:19:58 -0400
> The reason I am not doing initializing stores is because they require
> a membar, even if only regular stores are following (I hoped to do a
> membar before first load). This is something I was thinking was not
> true, but after consulting with colleagues and checking processor
> manual, I verified that it is the case.
Oh yes, that's right, now I remember.
^ permalink raw reply [flat|nested] 79+ messages in thread
* Re: [PATCH v7 07/11] sparc64: optimized struct page zeroing
@ 2017-08-30 17:46 ` David Miller
0 siblings, 0 replies; 79+ messages in thread
From: David Miller @ 2017-08-30 17:46 UTC (permalink / raw)
To: linux-arm-kernel
From: Pasha Tatashin <pasha.tatashin@oracle.com>
Date: Wed, 30 Aug 2017 09:19:58 -0400
> The reason I am not doing initializing stores is because they require
> a membar, even if only regular stores are following (I hoped to do a
> membar before first load). This is something I was thinking was not
> true, but after consulting with colleagues and checking processor
> manual, I verified that it is the case.
Oh yes, that's right, now I remember.
^ permalink raw reply [flat|nested] 79+ messages in thread
* Re: [PATCH v7 07/11] sparc64: optimized struct page zeroing
@ 2017-08-30 17:46 ` David Miller
0 siblings, 0 replies; 79+ messages in thread
From: David Miller @ 2017-08-30 17:46 UTC (permalink / raw)
To: pasha.tatashin
Cc: linux-kernel, sparclinux, linux-mm, linuxppc-dev, linux-s390,
linux-arm-kernel, x86, kasan-dev, borntraeger, heiko.carstens,
willy, mhocko, ard.biesheuvel, will.deacon, catalin.marinas, sam,
mgorman, Steven.Sistare, daniel.m.jordan, bob.picco
From: Pasha Tatashin <pasha.tatashin@oracle.com>
Date: Wed, 30 Aug 2017 09:19:58 -0400
> The reason I am not doing initializing stores is because they require
> a membar, even if only regular stores are following (I hoped to do a
> membar before first load). This is something I was thinking was not
> true, but after consulting with colleagues and checking processor
> manual, I verified that it is the case.
Oh yes, that's right, now I remember.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 79+ messages in thread
* [PATCH v7 07/11] sparc64: optimized struct page zeroing
@ 2017-08-30 17:46 ` David Miller
0 siblings, 0 replies; 79+ messages in thread
From: David Miller @ 2017-08-30 17:46 UTC (permalink / raw)
To: linux-arm-kernel
From: Pasha Tatashin <pasha.tatashin@oracle.com>
Date: Wed, 30 Aug 2017 09:19:58 -0400
> The reason I am not doing initializing stores is because they require
> a membar, even if only regular stores are following (I hoped to do a
> membar before first load). This is something I was thinking was not
> true, but after consulting with colleagues and checking processor
> manual, I verified that it is the case.
Oh yes, that's right, now I remember.
^ permalink raw reply [flat|nested] 79+ messages in thread
* Re: [PATCH v7 08/11] mm: zero reserved and unavailable struct pages
2017-08-29 2:02 ` Pavel Tatashin
(?)
@ 2017-08-30 21:22 ` kbuild test robot
-1 siblings, 0 replies; 79+ messages in thread
From: kbuild test robot @ 2017-08-30 21:22 UTC (permalink / raw)
To: Pavel Tatashin
Cc: kbuild-all, linux-kernel, sparclinux, linux-mm, linuxppc-dev,
linux-s390, linux-arm-kernel, x86, kasan-dev, borntraeger,
heiko.carstens, davem, willy, mhocko, ard.biesheuvel,
will.deacon, catalin.marinas, sam, mgorman, Steven.Sistare,
daniel.m.jordan, bob.picco
[-- Attachment #1: Type: text/plain, Size: 1694 bytes --]
Hi Pavel,
[auto build test ERROR on sparc/master]
[also build test ERROR on v4.13-rc7 next-20170829]
[cannot apply to mmotm/master]
[if your patch is applied to the wrong git tree, please drop us a note to help improve the system]
url: https://github.com/0day-ci/linux/commits/Pavel-Tatashin/complete-deferred-page-initialization/20170831-041021
base: https://git.kernel.org/pub/scm/linux/kernel/git/davem/sparc.git master
config: tile-allmodconfig (attached as .config)
compiler: tilegx-linux-gcc (GCC) 4.6.2
reproduce:
wget https://raw.githubusercontent.com/01org/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross
chmod +x ~/bin/make.cross
# save the attached .config to linux build tree
make.cross ARCH=tile
All errors (new ones prefixed by >>):
In file included from include/linux/pid_namespace.h:6:0,
from include/linux/ptrace.h:9,
from arch/tile/kernel/asm-offsets.c:35:
>> include/linux/mm.h:1974:33: error: expected '=', ',', ';', 'asm' or '__attribute__' before 'zero_resv_unavail'
make[2]: *** [arch/tile/kernel/asm-offsets.s] Error 1
make[2]: Target '__build' not remade because of errors.
make[1]: *** [prepare0] Error 2
make[1]: Target 'prepare' not remade because of errors.
make: *** [sub-make] Error 2
vim +1974 include/linux/mm.h
1970
1971 #ifdef CONFIG_HAVE_MEMBLOCK
1972 void zero_resv_unavail(void);
1973 #else
> 1974 static inline void __paginginit zero_resv_unavail(void) {}
1975 #endif
1976
---
0-DAY kernel test infrastructure Open Source Technology Center
https://lists.01.org/pipermail/kbuild-all Intel Corporation
[-- Attachment #2: .config.gz --]
[-- Type: application/gzip, Size: 49789 bytes --]
^ permalink raw reply [flat|nested] 79+ messages in thread
* Re: [PATCH v7 08/11] mm: zero reserved and unavailable struct pages
@ 2017-08-30 21:22 ` kbuild test robot
0 siblings, 0 replies; 79+ messages in thread
From: kbuild test robot @ 2017-08-30 21:22 UTC (permalink / raw)
To: sparclinux
[-- Attachment #1: Type: text/plain, Size: 1694 bytes --]
Hi Pavel,
[auto build test ERROR on sparc/master]
[also build test ERROR on v4.13-rc7 next-20170829]
[cannot apply to mmotm/master]
[if your patch is applied to the wrong git tree, please drop us a note to help improve the system]
url: https://github.com/0day-ci/linux/commits/Pavel-Tatashin/complete-deferred-page-initialization/20170831-041021
base: https://git.kernel.org/pub/scm/linux/kernel/git/davem/sparc.git master
config: tile-allmodconfig (attached as .config)
compiler: tilegx-linux-gcc (GCC) 4.6.2
reproduce:
wget https://raw.githubusercontent.com/01org/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross
chmod +x ~/bin/make.cross
# save the attached .config to linux build tree
make.cross ARCH=tile
All errors (new ones prefixed by >>):
In file included from include/linux/pid_namespace.h:6:0,
from include/linux/ptrace.h:9,
from arch/tile/kernel/asm-offsets.c:35:
>> include/linux/mm.h:1974:33: error: expected '=', ',', ';', 'asm' or '__attribute__' before 'zero_resv_unavail'
make[2]: *** [arch/tile/kernel/asm-offsets.s] Error 1
make[2]: Target '__build' not remade because of errors.
make[1]: *** [prepare0] Error 2
make[1]: Target 'prepare' not remade because of errors.
make: *** [sub-make] Error 2
vim +1974 include/linux/mm.h
1970
1971 #ifdef CONFIG_HAVE_MEMBLOCK
1972 void zero_resv_unavail(void);
1973 #else
> 1974 static inline void __paginginit zero_resv_unavail(void) {}
1975 #endif
1976
---
0-DAY kernel test infrastructure Open Source Technology Center
https://lists.01.org/pipermail/kbuild-all Intel Corporation
[-- Attachment #2: .config.gz --]
[-- Type: application/gzip, Size: 49789 bytes --]
^ permalink raw reply [flat|nested] 79+ messages in thread
* [PATCH v7 08/11] mm: zero reserved and unavailable struct pages
@ 2017-08-30 21:22 ` kbuild test robot
0 siblings, 0 replies; 79+ messages in thread
From: kbuild test robot @ 2017-08-30 21:22 UTC (permalink / raw)
To: linux-arm-kernel
Hi Pavel,
[auto build test ERROR on sparc/master]
[also build test ERROR on v4.13-rc7 next-20170829]
[cannot apply to mmotm/master]
[if your patch is applied to the wrong git tree, please drop us a note to help improve the system]
url: https://github.com/0day-ci/linux/commits/Pavel-Tatashin/complete-deferred-page-initialization/20170831-041021
base: https://git.kernel.org/pub/scm/linux/kernel/git/davem/sparc.git master
config: tile-allmodconfig (attached as .config)
compiler: tilegx-linux-gcc (GCC) 4.6.2
reproduce:
wget https://raw.githubusercontent.com/01org/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross
chmod +x ~/bin/make.cross
# save the attached .config to linux build tree
make.cross ARCH=tile
All errors (new ones prefixed by >>):
In file included from include/linux/pid_namespace.h:6:0,
from include/linux/ptrace.h:9,
from arch/tile/kernel/asm-offsets.c:35:
>> include/linux/mm.h:1974:33: error: expected '=', ',', ';', 'asm' or '__attribute__' before 'zero_resv_unavail'
make[2]: *** [arch/tile/kernel/asm-offsets.s] Error 1
make[2]: Target '__build' not remade because of errors.
make[1]: *** [prepare0] Error 2
make[1]: Target 'prepare' not remade because of errors.
make: *** [sub-make] Error 2
vim +1974 include/linux/mm.h
1970
1971 #ifdef CONFIG_HAVE_MEMBLOCK
1972 void zero_resv_unavail(void);
1973 #else
> 1974 static inline void __paginginit zero_resv_unavail(void) {}
1975 #endif
1976
---
0-DAY kernel test infrastructure Open Source Technology Center
https://lists.01.org/pipermail/kbuild-all Intel Corporation
-------------- next part --------------
A non-text attachment was scrubbed...
Name: .config.gz
Type: application/gzip
Size: 49789 bytes
Desc: not available
URL: <http://lists.infradead.org/pipermail/linux-arm-kernel/attachments/20170831/61f40b2f/attachment-0001.gz>
^ permalink raw reply [flat|nested] 79+ messages in thread
* Re: [PATCH v7 08/11] mm: zero reserved and unavailable struct pages
2017-08-30 21:22 ` kbuild test robot
(?)
(?)
@ 2017-08-30 23:12 ` kbuild test robot
-1 siblings, 0 replies; 79+ messages in thread
From: kbuild test robot @ 2017-08-30 23:12 UTC (permalink / raw)
To: Pavel Tatashin
Cc: kbuild-all, linux-kernel, sparclinux, linux-mm, linuxppc-dev,
linux-s390, linux-arm-kernel, x86, kasan-dev, borntraeger,
heiko.carstens, davem, willy, mhocko, ard.biesheuvel,
will.deacon, catalin.marinas, sam, mgorman, Steven.Sistare,
daniel.m.jordan, bob.picco
[-- Attachment #1: Type: text/plain, Size: 1768 bytes --]
Hi Pavel,
[auto build test ERROR on sparc/master]
[also build test ERROR on v4.13-rc7 next-20170829]
[cannot apply to mmotm/master]
[if your patch is applied to the wrong git tree, please drop us a note to help improve the system]
url: https://github.com/0day-ci/linux/commits/Pavel-Tatashin/complete-deferred-page-initialization/20170831-041021
base: https://git.kernel.org/pub/scm/linux/kernel/git/davem/sparc.git master
config: cris-allmodconfig (attached as .config)
compiler: cris-linux-gcc (GCC) 6.2.0
reproduce:
wget https://raw.githubusercontent.com/01org/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross
chmod +x ~/bin/make.cross
# save the attached .config to linux build tree
make.cross ARCH=cris
All errors (new ones prefixed by >>):
In file included from mm/page_alloc.c:18:0:
include/linux/mm.h:1974:33: error: expected '=', ',', ';', 'asm' or '__attribute__' before 'zero_resv_unavail'
static inline void __paginginit zero_resv_unavail(void) {}
^~~~~~~~~~~~~~~~~
mm/page_alloc.c: In function 'free_area_init':
>> mm/page_alloc.c:6863:2: error: implicit declaration of function 'zero_resv_unavail' [-Werror=implicit-function-declaration]
zero_resv_unavail();
^~~~~~~~~~~~~~~~~
cc1: some warnings being treated as errors
vim +/zero_resv_unavail +6863 mm/page_alloc.c
6858
6859 void __init free_area_init(unsigned long *zones_size)
6860 {
6861 free_area_init_node(0, zones_size,
6862 __pa(PAGE_OFFSET) >> PAGE_SHIFT, NULL);
> 6863 zero_resv_unavail();
6864 }
6865
---
0-DAY kernel test infrastructure Open Source Technology Center
https://lists.01.org/pipermail/kbuild-all Intel Corporation
[-- Attachment #2: .config.gz --]
[-- Type: application/gzip, Size: 42983 bytes --]
^ permalink raw reply [flat|nested] 79+ messages in thread
* Re: [PATCH v7 08/11] mm: zero reserved and unavailable struct pages
@ 2017-08-30 23:12 ` kbuild test robot
0 siblings, 0 replies; 79+ messages in thread
From: kbuild test robot @ 2017-08-30 23:12 UTC (permalink / raw)
To: linux-s390, linux-mm, linux-sparc, linux-kernel,
linuxppc-embedded, linux-arm-kernel
[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: multipart/mixed; boundary="--n8g4imXOkfNTN/H1", Size: 60131 bytes --]
--n8g4imXOkfNTN/H1
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
Hi Pavel,
[auto build test ERROR on sparc/master]
[also build test ERROR on v4.13-rc7 next-20170829]
[cannot apply to mmotm/master]
[if your patch is applied to the wrong git tree, please drop us a note to help improve the system]
url: https://github.com/0day-ci/linux/commits/Pavel-Tatashin/complete-deferred-page-initialization/20170831-041021
base: https://git.kernel.org/pub/scm/linux/kernel/git/davem/sparc.git master
config: cris-allmodconfig (attached as .config)
compiler: cris-linux-gcc (GCC) 6.2.0
reproduce:
wget https://raw.githubusercontent.com/01org/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross
chmod +x ~/bin/make.cross
# save the attached .config to linux build tree
make.cross ARCH=cris
All errors (new ones prefixed by >>):
In file included from mm/page_alloc.c:18:0:
include/linux/mm.h:1974:33: error: expected '=', ',', ';', 'asm' or '__attribute__' before 'zero_resv_unavail'
static inline void __paginginit zero_resv_unavail(void) {}
^~~~~~~~~~~~~~~~~
mm/page_alloc.c: In function 'free_area_init':
>> mm/page_alloc.c:6863:2: error: implicit declaration of function 'zero_resv_unavail' [-Werror=implicit-function-declaration]
zero_resv_unavail();
^~~~~~~~~~~~~~~~~
cc1: some warnings being treated as errors
vim +/zero_resv_unavail +6863 mm/page_alloc.c
6858
6859 void __init free_area_init(unsigned long *zones_size)
6860 {
6861 free_area_init_node(0, zones_size,
6862 __pa(PAGE_OFFSET) >> PAGE_SHIFT, NULL);
> 6863 zero_resv_unavail();
6864 }
6865
---
0-DAY kernel test infrastructure Open Source Technology Center
https://lists.01.org/pipermail/kbuild-all Intel Corporation
--n8g4imXOkfNTN/H1
Content-Type: application/gzip
Content-Disposition: attachment; filename=".config.gz"
Content-Transfer-Encoding: base64
H4sICCkwp1kAAy5jb25maWcAlFxLc+M4kr73r1C497B7mC6/Su3eDR1AEiQxIgmaACXbF4bK
pep2tG05bFXP9L/fTPCFF6maupT5fQkQj0QiMwHq559+XpDvx8PL7vj0uHt+/nvx+/51/747
7r8uvj097/9vEfFFweWCRkz+AsLZ0+v3f396fH/6WFz/cnH1y/k/3h+Xi/X+/XX/vAgPr9+e
fv8OxZ8Orz/9/FPIi5glTZ7Xq7/7hwde0CbKyYhUW0HzJqEFrVjYiJIVGQ/XwP+86CRIFaZN
SkTDMp5cNvXV5QLe/3o4Lj72x2mx5bUu1gn170m3lCWpHJvREyHJWFARCa2kGbkfBQreMF7y
SjY5KUc45lVIAbpTXeNVRKvVsieFJOFaVgQkRF1i4bFgWDExPqUPq4vz8+FdVROWtVhd9EBQ
s0yyoolksDo769GIxt1fGRNydfbp+enLp5fD1+/P+49P/1UXJKdNRTNKBP30y6OanaEs/Cdk
VYeSV1ozWHXbbHmFww8T+PMiUerwjCP4/W2c0qDia1o0vGhEro0FK5hsaLGBqcAm5Uyuri7H
/nIh4LV5yTKqdaJDGkmFNjqgAyTb0EowXpg9JnUmm5QLid1bnf336+F1/z+DgNjqkyPuxYaV
oQPg/6HMRrzkgt01+W1Na+pHnSJtf3Ka8+q+IRJmOtW0IiVFlGlV1YKCXo3PpIYV1Y8yjPri
4/uXj78/jvuXcZR7lcRJESnfusqKTJiy0pzAiOeEFT4Mxi+oE389ioo1XVDrCdotQExKllMe
x4IOjQYF/SR3H38ujk8v+8Xu9evi47g7fix2j4+H76/Hp9ffx55IFq5RoxsShrwuQJO1RgQi
asqKhxSGE3g5zTSbq5GURKxhfUlhQu2ytSpSxJ0HY9xskupZFdYL4U4HiNw3wGmmK6wbeldS
fVkLQ0KVsSBsd1fPYLuwJuhMluFyyHnhNXAoVFAaNYImYYBL3mPg0FJETcCKS03t2br9Y/Vi
I2p89WWHNcSgbiyWq4tfh5VQsUKuG0FiastcDQsiqXhdatNRkoQ2anDBJg4orJgwsR6tZTti
YE1IkNFI04ls3b1pxJTqepn2udlWTNKAqG3FZESY6rXHhFWNlwlj0QSwqLcsktpCh+3AL96i
JYuEA1bG/teBcUXpgz5OHR7RDQuprisdAcsJFdqjAp1AUMae2oz1L3i4Higi9ValNFyXHGYd
dhABm4Rmy9DwihI2Na1rtRRNoW8kYGT1Z7AjlQHAyBjPBZXGsxpPMJOSW1MKdhimIqJlRUPY
pqNpptlcahNl7uaoLDC0asuqtDrUM8mhHsFr2Nm1vaeKmuRBt7UABABcGkj2YDg3UXP3YPHc
er7WRj1seAmmlj1QdCwaMC3wX04KSwMsMQF/ePTA3rFIARs2K3ikT1xKNrSpWXSx1AZH1xzb
wFmyOWzHDGdXmwfYL3K0rtgAsGn2DPlgaKiLt/svbj26j7IGGXGfe5CmLT2M04gHgmc1+HTQ
FVg3nrEaRANwlpSySLbRXQFlAO3npsiZNmz60qJZDLZPXzaq5rjWOxhDm+60MiU3hoUlBcli
TTvVUOgA3dBC6gBMnWd8U7Comh4wTQVJtGGC9mWsFaucLb16qCcgVcX06QaIRpG+EMvw4vy6
30274KDcv387vL/sXh/3C/rX/hU8BQI+Q4i+wv79Y9xmN3nbq37n0G1CVgeOCUOs2zCUdnHN
80HfkkhwV414QmQk8C0XqMkU434xgi+sEtp7o3pjgENLjptzU8F2wXNDzSWEOmhoG3BtWczA
TDG9vbAZxyxrXZGhHWqFLq8D8K0hNkkKNIsh+kSexilZQymVF7clMJ5okktSwWT3frVpUlSw
Ai2QFIMCT+UyBQ8S64OVoc1KzqM6Ay8RFEZpPS4erUuJxB28yWBSQb0urX71QVvq9XiYILCq
YP2WzLfPZRhM4ra+JVUkNDvMwQWC1SNqUdIicnASSrv74JtCgEdjmBOGygeurrdFY6M3EAa2
YzYdjaKV5rCGmzWtCppBrHv3Hwn3GjYf70Icx8A5/pF3aOLttNniw24Zq+XX26s2GAz55h9f
dh/7r4s/20X99n749vRsuPoo1DXFG8MrvtVzivbaF6OjiNr3pHIAIooqqdemS1w1197+6jLX
za/Ts9kHQeCVgcFIaQXzP7HuWRHruzcMIhpwfTEoIy9yNL/n1gKxV0yXOsg4iRyqLrxwW2Ig
h34A3a1pv9Z2xSEQ6cQmRr6XY4nzasG6XIeXMbYbDRcpubAaqlGXl/6ps6Q+L39A6urmR+r6
fHE5221lilZnH3/sLs4sFvcScIXdaeyJ3ju0Xz3wdw++eK3LdnWPWRARzflCN1WEgsGCvK2N
BEnvwAYi8YJGtmH0diVNIBbyOMKYvYpcWKawIcjMDNgdDpR4a/JhHgFB2w2nMrltIB2gEbcu
lt/aL8U4T89RqPGBDZGXZDBT5e79+IQpyIX8+22veRbQFsmk0v5ogz611l8CvloxSkwSTViD
O06meUoFv5umWSimSRLFM2zJt+CI03BaomIiZPrLwXv2dImL2NvTHHZGLyFJxXxETkIvLCIu
fATmcyIm1mBJqW4vIOi6a0QdeIqA3w4vh7Vzs/TVWENJ2Pypr9osyn1FELZ9yMTbPdh4K/8I
itqrK2sCu42PoLH3BZiNXN74GG35OIMIKp/fNhsGDO91nvGFePxjj5lf3ZdmvA2jC871HGGH
RpSot2hZoY4J49sRhIcux9HRY019Otasv0d78bPXw+FtNKe3Mw3QyPV9AIbDaVqgNy2YbhoR
xYWhJ4UaUDxfUBuobnTH3Mts0ht2/1yii2tEeWaQh09NVOflMAroEqfQUSOS6eoSYcVKCf2x
/Etee1N7baEcVrk2O/BCfJ/WJlmRu16DVmfvh8Nx9enr/q9PL8evX54Pj39enZmiWyLDNOKJ
XUWPNwUDrwP+SO5tkZhAnKOiUptZow/Ewca67g/shjQvUWsKI63R4xuI1QuwOffeDb2T8oxP
X155rtrsQixvxL8INJgGwbixO80xJyDgHIs06O8pSZ8/WGYQkZUy421uW6yutSgZDwUCHAFj
6bZAG5aG1or3YGCQK2KHKmV6DwFRFFWNbKNCn6bACOieGhqLRnKMpTRrJXJ3zeYY1eQY58Eb
Vtfnvy0tHx5DSQFbf6nys553qwx1SdUJFmiB1sOMwqZLYJnqK49DdUby98F6LDnXjPpDUEej
8j9cxRDSac+i1boR6cNP6FZpODG9KIZwmrKwKKNt4hqP7dZGkbjCA7WNCo81bVYppaY/ilAW
JN29f/3X7n1vmxC1NC7Oz7M7fUZHtNn4jzaVxEREiueIuBOnMD+ypOGVZ07aJdmGrRfnzrs7
5srnG7cCEXRdpRlXN5YByMBhRr1q8+KXHvaiLXhu1ah8OTwS9bRnJC+mx0OTmhk1Terqh6Sa
ovbGRp1ZS6JgqtUDN9foQWiuzYOQ0eT22A2me/HXxfni8IY+rrbZK0UAR9w8n9bmD61ATTIV
o66W5+0/U64kTUbVkYXVojJQxEybQ1Gen5IpuCNh8EBeJLYSIVitrhzwMhkt7ghWq88OeOWr
86qyQQjSQjzopZVnckdybnZHqbnpHaWmFysY5KYCa8skWhbYPn/7vFv6RDA/1sKri3NrQATO
u6MLqpii+oJfL7/tfr359WZaEE8bwBbegMZc35xf+ATV4gEFili1uniclcCjJlv1dJGgb9i8
EL7phAS+6ds3a1R4DK5XWsuIbwt7dHocvC25urA0RIVgQS0lb/nLQdcKqm0ieJYFo2VmCxCk
PaaWcrE//uvw/ufT6+/uYgZfYE2lto+pZ+gx0Q5xMfoxnywBmYnx4S6uNF3Ap4bHsZlnUijJ
Em4WU4cBFgTxGoxHxsJ7q3jrtVALRS1mQhrxryJYia7PWDmO05reO4BbLzMGnZXtWVRIhIn2
wX4DgYJxEg1czALY6BltrJsEfWUlXkxBB8LkVE2dBNGPgwduA3rCBfUwIeyJgkUGUxal/dxE
aeiC6Ji6aEWq0tK+kllDysoEIxOa13c20ci6wHStK++rIqhAY5xBzlXnPNDsOJYsF7AvXfhA
LZ0v7tFn5mtGhd3NjWRmI+vI35+Y1w4w9l2YWtWQVIs81coVpYsM68dkbI1WoNJ1u2GK8YLt
SsIYBQxPIdTVsUmJ+QoCSu2yppVoWxGWPhiH0wNXZOuDEQIdE7LimlXAquHPxJN2G6iAaU77
gIa1H9/CK7acRx4qhb98sJjA74OMePANTYjw4MXGA+LRKSq3h8p8L93Qgnvge6qr3QCzDAJ5
znytiUJ/r8Io8aBBoNnwPuCrsC1OGNiXWZ29718PZ3pVefTZOB+ANbjU1ACeOkOL8XhsynUm
0DxGUUR7nwH3B9izI3M1Lp3luHTX43J6QS7dFYmvzFlpN5zputAWnVy3ywn05Mpdnli6y9m1
q7NqNLubIG2ewOyOYRwVIph0kWZp3IBBtIiYCFXmQ96X1CKdRiNo7BYKMSxuj/gLz+wR2MQ6
kBV1YHfLGcATFbo7TPsemiybbNu10MO1OUQfk0IAbmxNVi4aELxWC8JhTqq1uYuVsuy8gvje
LVKm9yrDBh5KbqYwQCJmmeHSDJCd4h4J1wgHFYsSqlX30oWah/c9Oqnfnp6P+/ep691jzT6X
t6NwRFixNnZgk2pvbM7w7d3cGYGMa0avwNs3RaGSOAaKtxfbO5YODBVFdOOvo7GmTafcSdVZ
PFgTExxevoynSHULZopU93FrOcMqfZnglXZaVUtsjeSwp4SlnzEdQo0QoZwoAu5DxvRFajSD
5KSIyMSAx7KcYNKry6sJilXhBDO6rX4eJj9gXF1i9AuIIp9qUFlOtlWQgk5RbKqQdPouPStI
hwd9mKBTmpV6BOeuniSrITYxFaogZoUF5s0pNe56dfCE7oyUTxNG1tEgpDzqgbA9OIjZ846Y
Pb6IOSOLYEUjVlG/9YHQA1p4d28U6jYVF2pDUg/umhYIIO5kGlUmllNJTKSS5nNR5wktTCy0
ZAR66GrPdHF1wcFBAyYxa2jW2t3cNkDLyMruMw+zE0TcWp3AEbb6QaxSPPgn+osGZtt8BXFn
iOg/qT0ELebMh+yuBJqYOyYxCxzAndyoLr0zO4XH28jFB1W7G9RK7b53x92X5/3H4vHw8uXp
df910X3Y49t572S7P3lrVYZlhhaqV8Y7j7v33/fHqVdJUiUYI6tPWPx1diLqFrmo8xNSve8z
LzXfC02q34/nBU80PRJhOS+RZif4043A06b29HRWDD+AmBcwVqVHYKYp5kL0lC2oZRt8MvHJ
JhTxpAenCXHbY/MIYRaQihOtnjPqo5SkJxokbevvk8EP/OZFfkglIbrOhTgpAwEfXuEs7UX7
sjs+/jFjHyQeEUZRpSI6/0taIbz+P8d3H9nMimS1kJNq3cmAFw4e7gmZogjuJZ0alVGqDbhO
Slm7lV9qZqpGoTlF7aTKepZX3tKsAN2cHuoZQ9UK0LCY58V8edwdT4/btIc5iszPj+cgwBWp
SJHMay8E5fPakl3K+bdktEhkOi9ycjwwITDPn9CxNoVhZI88UkU8FTcPIlzML2e+LU5MXHfM
MyuS3otJv6aXWcuTtsd271yJeevfyVCSTTkdvUR4yvaomGRWgJtndD4RPLs9KaHyniekKkz9
zInM7h6dCLgaswL11eXIs7JzDY1n/NR9dfl5aaFtANGw0pEfGGNFmKSVJC2HSMVXYYebC8jk
5upDbrpWZAtPr4eXun1Q1CQBlc3WOUfMcdNdBJLFhkfSseqjJHtKdWOpHtuE/t8mZmUTWxDi
FZxAgcfk7S1UML2L4/vu9ePt8H7Er0OOh8fD8+L5sPu6+LJ73r0+4mH3x/c35LVrqqq6NhMg
rVPPgaijCYK0W5iXmyRI6se7RMTYnY/+Wq3d3KqyB27rQlnoCLlQzG2Eb2KnpsAtiJjzyii1
EeEiekDRQsVt70+qbot0uuegY8PU32hldm9vz0+PKj28+GP//OaWNLIv3XvjUDpTQbvkTVf3
//5AFjrGs6uKqKT8tRGlh2N20KZaC+7ifTbHwjGgxV9R6E6xHLZPOjgEJgRcVOUUJl6NJ/p2
qsGRxaS1LYiYIzjRsDZ1NtFJH6dATO/UtCKRbwiQ9I4MRGP+6jCvilfSmJvB86edFWNnXBE0
88KgSoCz0k7WtXgXDqV+3HCZdaIqhyMSDytlZhN+8SFGNRNXBulmHlvaiNeNEuPETAjYkbzV
GDtg7rtWJNlUjV2cx6Yq9QxkH8i6Y1WRrQ1B3Fyr75UsHLTeP69kaoaAGLvS2ZW/lv+pZVka
SmdYFpMaLYuJj5ZlufIsusGyLO310y9gi+jsgoV2lsV8tU90quLejJhgZxK8LfdxHnNhle3N
hdPdzlwYB/TLqQW9nFrRGkFrtrye4HB2JyhMtkxQaTZBYLvbLz4mBPKpRvqUV6elQ3hykR0z
UdOk6dFZn+1Z+o3B0rNyl1NLd+kxYPp7/RZMlyjKIVkd0fB1f/yBFQyChUpAwlZCgjojeH/f
syjbc3BTE7uzcfdcpiPcs4f293Gsqvoj9rihga2/HQcEHlLW0i2GlHQm1CCNQdWYm/PL5srL
kJzrEaXO6C6FhrMpeOnFrRyJxpihm0Y4GQKNE9L/+k1GiqluVLTM7r1kNDVg2LbGT7k7pN68
qQqNxLiGWylz2KXMfGB7oS4cr+W1Sg/AIgxZ9DGl7V1FDQpdegK3gbyagKfKyLgKG+OzYoPp
S43N7H75I909/mn8REBfzH2PmXLBpyYKEjwaDPUvulqiu6rWXgxVN3Dwbpp+c39SDj9L997P
nyyB36r5PgJDebcFU2z3Obw+w+0bjauUVSSMh8a45IeANXISf+PuRX8CgwV1mjEzkVpKDB7A
edNXdI+oHzoMc7Ngkxn3GBDJS05MJKgulzfXPgzm1r6pZGZh8Wn4BToT1X/zTQHMLkf1ZK1h
JhLDlOWuXXNWJksgGhH4Pav5yXvLoq3p7LBBtz+7ok4NtU/De+DFAmC/wRrD3BFVjK8ORdBJ
Zi0e/AS097er8ys/mcu1nwCflmXWtbKBvA21RqgBgT3mQjvwH7Em2egX0zUiN4h2gx5r6DZs
+75+pmdA4MHIVd4ZD+qXDirzG/Zsrb9h05CyzKgJszKKSuuxoUWof1B1d/lZawUptYsCZcqN
fiwzvi313akD3N9a7IkiDV1pANWlaj+Dzqt5jqazKS/9hOlc60zOA5YZjpvO4qQYqWidrCPP
2xIg6B34qFHlb04yVxJNka+leq3+wdElTA/fJ2F5XoxSiqr6+dqHNUXW/aF+HY3h+BP9yugo
aR8SaJSjHrBF2O9st4j2y3a1s95+33/fw3b6qfve39hZO+kmDG6dKppUBh4wFqGLGltFD5YV
4y6qjqk8b6usOwsKFLGnCSL2FJf0NvOgQeyCifdVkXBO2BQO/1NP56Kq8vTt1t/nMOVr6sK3
vo6EPLI/RUE4vp1mPLOUevpdMk8b+ju4rnRWJ55uD79gNrhAvfcT33o9pNE5gtbPSvRdnBUS
5mssFryGmKsfGXA/Uei6sDp7+/b07dB8230cz7p7y8//T9m1NbeNI+u/otqHUzNVm41EWY78
sA8gSIpY8WaCujgvLI2jTFzjS8pWZpJ/f9AASXUDkJQ8+MKvG00ADeLSALp3b28PnztTNP06
eGZdIVKAY33s4IaLIoq3LkH3FVcunmxcjGypdYD263jMRo+6J8P1y+S68mRBodeeHIAjHgf1
HNgw5bYOegwirP1gjWsTBPh5IpRYw9Ytx2Fnky+RA2hE4vaFvw7XZz28FFKNCLcW5kdCozp2
L4GzQkReiqiktZ2rC864dbWTwfFm2BK3sgo4+JbD809zEjp0BeSidvotwCXLq8wj2PgJsUD7
7JbJWmyfyzOChV3pGl2GfnZuH9vTKF1s96jTjrQA30Ga/p156Sm6SDzlNrcx3BuhilkLct7Q
EdyeuyOc/KoF9hQ49MYCX1WKONJkVEi4yV2Cm3K0jlBjJ9MepnxY/+8aLS0QETtORHiEPVQg
vOBeOKe3M7Ege95p046UsoqLtdwI+LqfPCDdlsGE9ZY0EpImLuI1SrY2syM0XBm3RpcJ7h2O
7hw7XUqrb8nq7wFpF7KkPO60VqPqo7OuK6XSnifoksHRF/KabArGTHMRB5Fu6walh6dW5tan
UHCJ/HemmxD7UDHejoBNN3AfwblgrNdSW3AUcNdSD7Ph7eA1rbuMPjrs3w7OnLJaNuTQecry
mkVH91PV7v6v/WFU7z49vAzb/ugkIiOLJnhSbTpn4FMRO9lVr6pL1OvUcIO6M3yx7X+C2ei5
y+Wn/d8P9/vRp9eHv4nDq3wp8OTnuiJn9MLqNm5S+rXeqYYEnhjaJNp68dSDV8yVEVeoe71j
qBgcfw7qgZrLAQg5ZW8Xm77c6mkUmdJGdmmBc+1IX28dSGYORA5rAcBZxmFPv/P08APTOp8g
CGHNzcTKcu2+dlVcCestbm1oSM1PWQPeOi0a//Bh7IHAM6wP9ksRiYC/SUTh3M2L/B+bjMdj
L+i+syf43wpuJkyXMihSgh9b8Hf8eXe/txSZiulksrVyyKtgpsFBxEqGJ0XEOXjhDamMWEYA
Bpa2PJzLNYMG7+BVzJYuOgcbioPmPGQuarzrGFf2JNqIvt5jdmNfI+brOkRNRhNR00NRNYwD
+Dli2m8fGw7xgFzHvYbm076o2gxcmGUSm3g0Vbs2q2sLJVZu8fz5dfe6//ROH3dy+iTNI0V9
srdSQ1pzpyZmw43L6OX5z8e9e0AqKvW225CVWIoeO/aqvBHyTjp4Ey/BgYsDlyKfBmrVYRPg
lpYZSS1Czq5VW7fRhahDkbnMquVOApcdPDCHcbaEWCRuAYLx2BUFTrjA4aKDy4h9/JjFHsLN
7OaI6ppNzqhBNde+KXaIFAu1JFDTzgRfW8q5pECId3xg9y6OsKtT1XwS2jwHqG2ID1aVtogr
KkwB6o2tbTfvSeZsjIfK84ZKSkVkAZIkwA1LPTpWKc0S0TQyzhIasAeBbcyj1E8h4YJgG26Y
kGothY/f9oeXl8OXk7qC/caiwTMyqBBu1XFD6WCyJhXARdiQTgqBWtoPH6HGwRR6gozwOsOg
K1Y3PgzmE2Q6iEjplRcOuay8BNak06WXkjm51PB0I+rYSzEV6n+7UxUaJ3sAOFOL6+3WS8nr
tVt5PA/G062jhUqNqi6aeBS2Vj8E06+xgdbRkak8jGwEvV2rW02Zk8k5S9Q0usZbbT1inZk9
woU+XZOV+Fb8QLVWXPV2iT1fKLYlbteyqWOW9w6VBxiO+tTUvyYoOiMX8XsErN0IjfXlQNwq
NERDAmlIVncOk0ANmScLsFyjKaGxkE90WC9wVuHywnwgzkpwFblhdQG9vIeJx2rd10dKaMti
5WOqY/UQZ9kqY2rCLsi1ecIEsQm0GzpRezPUmR19yZ2180Axe00sgzdEoa8MMHNwA8D15A3R
SiZCqyp7RMm5q1RTxSOFRePEsmYRm6XwEa1m2G1CoPf3iPbfX3OXVYHgMxRaaHae2uKQe16G
9SmOwUPp2Rf1Bu1/PT08vx1e94/tl8O/HMY8lqknPR3mBtjRPJYje1+hZNFE0yq+YuUhFqXt
UWQgdf7DTimnzbP8NFE27CQtbU6SSu5EVxloIpTOsYOBWJ0m5VV2hqb64dPUdJM7Z0SIBuGY
mtOLUg4uT9eEZjiT9SbKThONXt0QNUQH3c2QreUAdSPgDs0TeewE6igpRx+rdbIU2OJunq12
2oGiqLA/kA5dVLYZ9Kayn3tf4jZslZ0zgcy88OTjgMSWbUGBdEUYV6k+SOQg4DJKzYxtsT0V
IswQq+vRcpSQ0+OqVYiFgC1ZAhZ4ttAB4BHcBekMDtDUTivTKONH69nudZQ87B8hotHT07fn
/h7Eb4r19242i6/mKgFNnXy4+TBmlliRU0A7lMUmCQATPKXvgFYEViVUxezqygN5OadTD0QV
d4QdAbngtZpJYNdbBPakIFO1HnFfaFBHHxr2CnU1Kptgov7aNd2hrhTZuE3FYKd4Pa1oW3na
mwE9UqbJpi5mXtD3zpsZ3hGufJtGZDfF9WrVIzTYW6SKY7noXtSlnltZdnT1jdMZc87uzAc6
EIxJw6zgRp9fXkd/fHt4PLx7ePYasIyHVW2fLbBfz84ta1wLfMICo2e80HcMvcNmmzZuo6Zu
a6HKXbc8goC2tlP6c7wVO+Pk90SS8NeS5GIbRyedBA8JdGa0S9p3Ew+LkneBA153gaUv8xkh
XU7Cizk5w9Hn5AxLX5WnOHqVTzwqn/yCyj28l1TuS3JB5XaSiyqfXFb55KLKJ5dVPrmo8sll
lU8uqnxyWeWTn1Z54FF58Asq9/BeUrkvyQWV20kuqjy4rPLgosqDyyoPLqo8uKzy4KLKg8sq
D35a5VOPyqe/oHIP7yWV+5JcULmd5KLKp5dVPr2o8ulllU8vqnx6WeXTiyqfXlb59LzKa3k1
n9m61qDJ/hmSljf1kSMhdXwQ8Ach1u4EAuy+EGzexs0tNLjkBJeJrmezqeWcX+D42wNk4qmr
Ahal9k9/gkuHCNXWorFL5NnS0CYuLY6runQ8/h8p+jCS5zNAHMHyXEtGjPOfZZxcL32BVA2f
XqDalcvgUmGxiHUd+7z8Uw59Ez1JLJbwopDwohCwMKR1WZQreWoK6nCcmG06fOOWxAE/K86d
yTh8E584thVSB2Gxohx14Tg6F7NZrL+7TRoXLczdTyrLlwD2HdckpKA79QaLK4TXJB5LT7OJ
wttRD5Vx5DvXgR7ZFe/NTwv1x97yS513JTotNvhFsQEwBz8v9ueqQEudXqqC6a8JnQLv7KeF
/mQFaKlXXQWYFaR9+uUYpO3hvoNHpb3DvzJBpDu3VD+8cKuDNxwjvKula5NX2NLbI21OY4rJ
BlyTZmVBQoQb2Ymocx2TMFyJDG3oJBsd65Vu3nSsojjGzO1rZquqZuBAuRzk6KgcTgm9ZLVE
zjKIlIx2cSAmNJzhcCPIQWCszQnaKVQf8KjpEDoc+6hjaaN6O9gkUINvXuJzYJrGjHXXcJhh
C8XJg5MO6Z0q2VpIGqz6GAC4i+sL0bm6oye+G3wlp2HR6nhBAiiZZ9XN3XxAhlgDEstMh4El
yE4sq1w4jHmOR75eYo1insJnI1Ol/UhlMUlI1SpSEhc87rwP9hvo395cY+OtPp4WCrQ2Un8K
M3U52m8aHG+4ibQWJKHDpQ6IQKLDUp4gmSuyOvydDrJ4nMU5AtpVoUOwsSa2Xk3ZwH5YFtkd
5cEhMq28lIkPZfUHHxzy/Hq63Q4kK4bs193rGz1iqNKYTVhV9cPRrJViGuXGQ6sObN+AG6RH
YwTOdj8cEWG2VI1S0jJZwYCThlhI7ae2RkF3BaXXSUSTS5lEqLHKnJJ1tZWVVTc6FN+TVXIT
kBQCO+oDrn3Tq1n+Xs323iePu7cvo/svD189xzNBb4mghf5fHMW8/8ARrr7fYbpK0+vzyiZc
unSJRdlFEDwGYu4ooeqT75rYiVzoMGYnGC22RVzmagCzGiZ8xiErlq0OiddOzlKDs9Srs9T5
+fdenyVPA7fmxMSD+fiuPJiVGxKnaGCCkybkYsag0TySdgcEuBpomcu9akRm9Rj4wK0GSgtg
oTS3JHVrzXdfv4Ivsk/IPK3b7O5e9Z12ky3BvL7tg0habQ5cH+bOd2LA3t20LwGUDVZ43+c0
Eh9myeLiv14CaNIKuYjJZeLPjuoZIQI8U/UX+zOlOBYxRGSmZMlnwZhHVinVekATKNrI2Wxs
YbDMMzFFab4y1jiqywZXtb225P7x87v7l+fDTnvCVkynT4ArAbB81qseKneA200tTIwuEnGD
8jgNOA9m1dwqVs7TKpgug9m11XHKJphZ3avMnJJWqQOpHxvT8ffKRk2Q9T6yDspKqXHNZGyo
k2COxekRJYCxvd+PeXj76135/I5DYz91sFzXRMkX2HOIcXurJmb5fydXLtqg8LfQMtT8uI05
t9pLh6rhh9PKBYqHN+TpCQmhvnVGem6Zd7dCTnTZOm23IU4SakKpPytwfwzT9nMi1NwfB8gb
cIhBXqplubC/Eko0g5onDss53kjf6hxfZk3FIj0vMgwb3f59XErnV57Mwy+yMY2qMxenlOce
iz9W9rZg0oOvk+vJmO7mDzT1TSYZtyctmpQKKWZjX87BxwGd5BSxm90O7HqE1lM9PUe3qPAn
d7qMnhBsQTsLE4xTf4ZZpVQ6+j/zNxhVPB897Z9eXn/4ezTNRl96q4NTe+ZNagGi5kC13a3M
J9+/u3jHrHdur5adPRMNcEBnsoKYzvDNPmGcq1UmrEBuVywi+99ATGTmJ4CuWplYsmBnXP1N
LGbZ5NPAlQM5X4Uu0G6ytknVp5JCWGiro9QMYRx2jomCsU2DbWSy9OsJEBvF9zZruh41qFMr
E/w/BOds6IFpBarVj0oUSgJCdHIdugODMauzOz8puitYLjgV3PUXGCMry1Kf0yHPOTkkWyb9
KRvCpJbpdcbQmKnWA50n2mOAcgO1C8l9Qdo7KtvO5x9u0LDZE9QAduXIN/ZyfN0xW9KbdR2g
ozfDA87RR/Xheef7fSJebk4PHT1TVmLnFRjVAdNN1Pm5TddHL0t/2qgOUUcET60542jODZNY
8EP5cJIeJDMGBHaZmlz7aM5kAhMjhs5x80hvHSwbHq3xHSoMdxYKeawBSt5Yx0pghwNaE/Xe
093yDLFTliOmZqH4auSQX1wlxTqPzaFmCiUsrCH4rIVyCzBu5ryg1QAw5YQYhXdpzJrj4e3e
tc6oVYlUXS54Z55m63GAD5xHs2C2baOqbLwgtT9hAukto1We3+nvf4BUTdxMA3k1Rsd8WZPH
aoqJ/YGo7j0r5arWsarN5a2Bpq1KvBQFHA5CUqpI3szHAcOhh4XMgpvxeGojeOXQ10OjKGr9
4BLCdEIuA/a4fuMNvjGQ5vx6OkM33iI5uZ6j50aoBs4/zCYIg52+7ppyItnNFZ7qQ7+rSq8m
ntW0D019zAeJsd0NllnFW97UuGKOBO1fCn1HQdeF6lZiNs3cO2DdZhprAjTPOYIzB8ziBcNe
+zs4Z9vr+QeX/WbKt9cedLu9cmG18mznN2kVy8EC1uy/795GAk5df3vaPx/eRm9f4HIccgj+
qFZvo0/qK3j4Cv8ey9bASO9qFj4J2pQJxbR+czsY/D/uRkm1YKPPD69P/6g3jz69/POsXY+b
yEnoOjJcmmKw/q6yXoJ4PuwfR2og1TZUsyIa7vBxkXjgdVl50KOg9OXtcJLId6+ffK85yf/y
9fUFTBMvryN52B32o3z3vPtzD1U9+o2XMv/d3iKB/A3i+s4UNrNbGkwg5ilZEvFtBi5RYu+Q
CUSWrHrDfFnJk2yZ8G37al9pAl8IEdFwTa963O/e9opdLUpf7nU70ibU9w+f9vDzn8P3g7bV
gH/x9w/Pn19GL88jJcBMm/HdyiiGMabyjBdAkhAYHuegXWCX6Pq59fCckck9A7OGh8P8cV2T
yTXiUlJj+v6GyWUrSo4vnQEOt3Pa45U5KDsYrlQN933G+z++/fn54Tuujf5NaEnmTIKUpCgn
B1+6VilFb21xOiUgtsSZR80EVFJTo+5fD9vkie5Va6Rz4GChVml1ZrpcjA4/vu5Hv6n+5K9/
jw67r/t/j3j0TvVOv7vllnjaktYGa1yslBgdUtc+DMI4RyW+R9QLXnhehg0ZumTD2GrhHMwp
jFxh0nhWLhbkjolGpb6MD1tlpIqavs99s3Sl13OudtRMxQsL/dtHkUyexNWHL5k/ga11QHWX
RG6GGlJded+QlRtzTeL4WWiceMk0kN5EkncysWWYRaiTx1UiU/wZI9BjzeipbbTh6u0eDlUR
eGqoH0tb4ea2A8XsGxmk4GohlHNh1yS+ctsBbR3hsCQ9mqql9saF49zDy7IVs9BSRmqBIhpB
/SgPtFVm1yCgUaUGnUYP4PHx5NKRTK98MO3VaeiiYLFQmO8iYrXPOAccpJtFlaEPU+VDWA/+
8nx4fXl8BLP/Pw+HL0rU8zuZJKNnNdT8vT8eTkefDohgKRceHWtY5FsL4fGaWdAWjEcWdlvW
2HueflFnf3/CZVP5Gz5wldV7uwz3394OL08j3YG7+QcJYW76XCNDIX5Bms0quWrfVhahxcNV
ftqN9xRLmQO+9hHAYAm7GdYb8rUF1JwN+wHVz2a/0oqrmQQPIcmQXJTvXp4ff9girHSurjUM
u8tHCjmM8nn3+PjH7v6v0fvR4/7P3b3Pmhe54z++ZJxHcAQrxg5m8kiPqmMHmbiIy3RF9iai
Lq4Yw6vuvLM73BHIieEXmiW89Wxru0O7wc25QTfYPnJtaG+Ex8YRoSpXfPkt8lN4hC3BWmCC
O82ex9jpwHc9W6gpKzyQgdTi037r3CudIF+AwVVI7HdKwVVcS6GqCs7TMOyOTtG0+YcgsmCV
TEsKNqnQm91rNS6VBZmdgxBa7z2ixsxbgqpFDa04oXtCDIFzejgXJCsSIUpRoK0Q4GNc08r0
tByMtthjJyHIxlIKmBAxYk5lkbpOMkZ8vikIrPOND2qTmJPEtt+yruDari8JDDvSC0csxGxH
lTHElsWTtYar1MYCTLBEZDFuhYBVdAYMECgBWR/AThTqdmcZoLRIHPnJzHUsLhlWR8ysC+I4
Hk2mN1ej35KH1/1G/fzuzt8TUcfaL8STjYDIwAMXlotEx6lPLtDyqIhtDwNhWUS0fYN1Ci1G
b1csEx9J5Arba2wTs9xFYB0Se+O8E4a6XBVRXYaiOMmhZhnlyRfo864x6Mp2qXnkgQN4Ictg
fw91qYxT94kANDQeD2VQz4RuedazvektsOMXJVzG1Kmp+k+W1jXBDnN3EXSIOexKRPuBUwgs
UJpa/YOPozWrAn8uKB+K0q51M6jV4oo4m1n7TL+0fWW2j792XaOjDeBh3hgisH8NAGmjAsgs
Dzo/WiJBZilnjNZXlhvcJ2hEb0hpj3Ue/A47Y9RwKoXFOMzq+534w+vDH98O+08jqSYv919G
7FUt5Q/7+8O3V5/vmxnej59p01h/5pLgsJXjJ8Aet48gaxY6hN7re6i6IJkELsEyiHdo3nyY
TccefD2fx9fjazxFgevBelMaPNj7YW8pqcztdnuG1C6yUn2MAW3KwHLL2XzpppS55IPn/LNU
62quj4Nuq7nXW3Wb1oYEtdZiHEYo7E+3M1U20rrd2yfJ2Ue8SwQka5kxQO068MtQPW7RCObP
E/b3oR7AVy63RsIeRsUEJqXCJT0MgeWu1MwEvdI8t0U4n4+tttNtM6MRhnE0IMCT3r5ON2rm
hq0I6HVmVMA3Y0J8M161cKghbPpZkALpR2BjNuYxC9ypuWDuRO4FB4vbOGJKGUR0ZI+ofZ7j
j7r2jgfC9XNbVLKbyoLX+TY+lTyp41iqd6KKgz35JMdDCCDVrdVIAdSZtPCFYEXCau/bwKCS
CY7bYiq2szQKWlpcbXlJYgurxld0IzAtpPV2hVCy+tQSipysC8vDFabMgxl2WYVIOavXMR4B
8/X1FZwUJhnN1zSbOQx2sNpWuYF7RTbFw4mhCk+6qi2bXM/p+3AGVTeJi7WU8/kVSg7Ps4n9
3Oa2D28krrSaXMGD+f9wj90jZi5tn6BT1G1wpchj7xsKpvqxXHhrW/t+Lco89lLn0xtyoyot
ufcNMHnULvcGXtXHfyAePTuA7iv1IPUzYQ7O25G8+3fVqrmBCfFozkipqmq2Dv0pwYdx7S2o
ZLlcESvvdhHGJ5uA/H/CrqTJbVxJ/5U6zhw6nkhqoQ7vQJGUBIubCbJE1YVR3fZMO8ZLh9s9
0f73DwlwyUwA1Qe7xO8DQexIAInMPH/vjqcuklatoVp3ecpO1zb6TFdCh2eudEr3CJfdAYcd
nfe1pO8YytKSMrBo3seb/cDhokmDeLBge/4wuKxTOOez4E7YUInNxk9gXw12yL6KhbOEn/HE
qB7G9irw9LpA7D4H4GDeLCVrSxTxXbwQQcA8j/cdMfayoJFGlx3RCT/1crpx4TyFQ6FEZYez
QyXVw50iZuAEtaRHVTdqykPrb9UuhoKO+kYi1UtFBsLdG4bAwlrboLPxvhJ2vCfRnRJiu3uK
eCz7wY36PzLx1G4SoaCe25x/zvGCa6rTBJPQmuuDWEOSd4Ws5QmXLbtWXGDvyhBGm0SIJ/Xo
VYROSq3UjcS0Sc5jaBdvooFiqnAOIDhzMD44wDF9XCpVNBauF4wsa7McRkOnQsl4LF0puGmv
GJglqgXxt7MmjuJt7AD3Bwqe4c49hUTaFDzxWloYh3vyoHgBp3ZdsAmClBFDR4FJdHCDwebC
iFzW1XgZeHg9vduYWRnZMEytFK600cSExfHeDtjmsM64UVAvYyjS5cFmwIt7tTZR1SxSVoLP
sM+mRHICDmBiU/Us1XDDFv63sqoElONxh2Xshrioaxr6MJ4kNCYGZjnoSeYU5IZyASubhoXS
W5b0FFrBNXGHBAB5raPfr6lnO4jWnM8SSF9uJFsMkmRVFtgTGHD6KglodWKVcU2Ap6OOYXob
Cn6hW0Cg4aTXqNOOyRdMpEmXUuSW3MnUDliTXxLZs1fbrogDrLG1gky/Si2KD2SmB1D9I3L7
nEzQEA0Og484jsEhTmw2zVJm5B8xY46dP2GiSh3EtVdlIPw8EOVJOJisPO6xct2My/Z42Gyc
eOzEVSc87HiRzczRyVyKfbhxlEwFg1fs+AgMgScbLlN5iCNH+FbJEubY3l0ksj9JcKvDFkB2
EMolhRjL3R7fTNNwFR7CDcWMCXEWri1V1+0HiuaNGlzDOI4pfEvD4Lix+8ZL0re8fes0D3EY
BZvR6hFA3pKiFI4Cf6/G2fsdr7e1ORHsfGQOquacXTCwBgMFxX0KamPlzdVKhxR5CxsoPOxz
sXe1q/R6DImcCTvLSPKb7AXfsaFJCLPs32SlmjDwpuvVcslCwndXGtg68L9qA73TPrS5uw4A
s+brDAc2gvV1ZXJgp4Ieb+MVaToYhCcTo45kKS47S9sIrKFOXVrng23cV7P8G8n1ZEXtjlZ2
xt6x/ithguUhuuF4dKVzspeMJ4mJVCWW3jg6WRhlKJhFAduACuyIcWJDNyrPpVXQeP5YIF8G
r/eWOhlpi2NAXX0YxPI9MsG2OeaZuTepA2UfVKnY3wqSYPXMLIVPIBkcJ8xuJ4CCAei6TPDI
lLS7XRiR94PNjT/bHwbQ/siCshLVuLv272kV7fE0MQF2PLS/ljlpFod9utsMtCzwC64dVnxc
sY3M9immRylPFDhp+0gQcNRXvjS/aiqSEO4bHUsQKV3ap/qrGV6ozykbG47awPUxXmyosqGi
sTFs2Bow5lRBIayNAsS1J7YRV+teIDvCCbejnQhf5FStZ4V5gayhdW3BFd/J1DuuDxQKWF+1
rd+wgs2B2rTUN7+/YETSjXqFnJ3I5DHjpCZVlImZZG1ihnvSQBVq64UAmp0u7m6UCpnWbort
SXOqlfiKO8ha+FTVPK/WYX56iLF6JrcTmt3Wmv8AI90aALLdMwGLCXZuZQx42vhwbqx99EKc
1DCFlS9nhKZjQenguMI4jQvKGvWCU5vvCwyaN1BajphmyhvlEoAku7zDCDxYAMvGjHpHVO3C
nQhZpRqFN0HvDt4mdOHaduGAJTz1vNtsyNfa7hAxIIytMBOkfkURPiohzM7PHCI3s/PGtvPE
1le3qr5XnKK2xE2+J3vhTtwZ1u5KiDSX+JwUs8W+Etb0OnGsMZEqNNsw+BW1iI6x3SEDWF8t
QNDJJAt4DNOeQHdywXYCeDEZkDsnmeKzRg8ghmHobWQE2/iS2PYhmcVX+dTDeMTez9pZMZqU
IChtk04EiLcD4du26T0g6x3zbILTKAmDRxgcdSdwpoIQH7OZZ/6uwciXACSCWUGPQe4FPWc3
zzxig9GI9R7WcnBjtBKdlfDyyPAJFnSyl0zlH2UHnoOgvdvIW01Z7zXnVYW+u7qguEvXlonZ
VbgbFSe9xX3/VCbDE6iuff74559Pp+/fXj/8+vr1g32J0vhQEOF2sylxOawoayaYcbpeuOP1
sDb1/wU/UR+OM8LOsAE1EgDFzi0DyMamRojDTFmolW4mw/0uxMdZBTYvD09wt2/NQZE0J7YT
Bo43E4m3uldv9NauIOLOyS0vTk4q6eJ9ew7xNpGLtTszClWqINt3W3cUaRoSCzMkdlKpmMnO
hxCfhQuZofqEp1FsC8rravjJkfH5HQNLEsy1u7y8a21QaybpibSpMfAEeMaXrjUKzWC+6KWe
n/7n46vWDfvzr1/NpUZ84w1eyFp+Ud3Aum5FvXQtQLfFp69//f30++v3D+a+JL0+2IAX9f//
+PSb4l2fuQqZLLc/s19++/3169ePnxcfjnNa0av6jTHv8cEzaFRid1MmTFXDTZHMGF7CtjgW
uihcL93yR4OdaBki6Nq9FRgbuzIQDAlmGo1Npq6f5Ovfs9Lsxw+8JKbI92PEY5KbUz1w8NyK
7qVJBceT53JMAutSz1RYhbSwTOTXQtWoRcg8K05Jj1vinNk0fXDwkrzgBYgBr+Bewko68Xpp
SsUkVxeJWqN912eQa5Mkxffr1LKerCY7JbvbbWM04ywpIb17Qbcylqx5p0lDdC3VWmU2i8iD
6f/IeLIwpciyIqdCIH1PNXvXixM1X0CaCwpgV+/CyVSVjzerFJJTLaqlu13EJSFb5hNgMo+W
3DOuxje3XcyJ1zqyReFYaM8h4BKw/b0y2OycaGCj3H2SHoa/kEc1sTUcKoJaLNq6X/TI5y9H
8wpvLgYk83aFy1o98NQB1BL/foA0xhjFdB37j79+eC/DMg9M+tFI6l8odj6rxV2pXfQxBhSs
ifckA0ttcPJGrMwZpky6VgwTs1ii/AxCksvJyvRS3auhwv7MjINDGXyOwliZtnmu5qZ/B5tw
+3aYx78P+5gGeVc/HJ/On52guRmJyt5nscy8oIb/U61G8DXpM6IkAlSvCG12uzj2MkcX092w
wZEFf98FG7yDjYgw2LuItGjkgTh9Xqhscmbf7uOdgy5u7jRQLRQC67aVu17q0mS/DfZuJt4G
ruIx7c6VsjKO8H43ISIXoabdQ7RzlXSJ7bWsaNOq9YeDqPJ7h5emC1E3eQXLJFdsl7rIzgJU
9+AKkiuE7Op7csc3lhAFvyVxI76SfeWuJPUx/ZYzwhLrJKw5UB1866qgMhy7uk+v5K7UQg+e
pgraImPuSoCaTFSDRBWL+jUax+FRjRJIwF6gMSmw/8wVPz0yF1zUF6H+Ypl6JeWjShp6/OUg
R1kSE85rkPTRUENYKwUCwq2pBb53trJ5AetXrImPvpvD1i2+LINi1ZUhnHGe6xT2dzyRurIw
uRxghZw0IBTDhzhzSsvd8bDlcPpImoSDkEN6f4Hi1DcF45ypfZbDMCTWh5hmnMnYUnWOFKwk
ndHn6QPOQ9Em2YyMSZWoxrS+sBJR5kIz4UDT+oRvUS745RzeXHCLtXYIPJZOphdqGC7x/c+F
0/v+SeqipMjyu6iI9fiF7Eo8ua3RnesW61sygh5ucDLEahgLqYTjVtSuNJTJJS/I9bQ17XDX
tG5PPuqUYI30lYPjfHd+7yJTDw7m5ZpX195Vf9np6KqNpMzT2pXorley/KVNzoOr6cjdBnvI
XQgQbnpnvQ9N4mqEAI/ns6OoNUP3eReukZolO4EOkkRsuo92W4NGJ/Ns9GfSPE3IrdeVEg3s
S7uoS4e3txBxTao70cpF3O2kHpyMpWA2cZOHiHuS1iUa36ZMwVhoJE6UsxWEo78GTsbxPVXM
J5k8xNjMFSUP8eHwBnd8i6MDnIMnlUj4VsnXwRvva3ttJbam7aTHLjp4st0rqVAMqWjdUZz6
UK3qIjcJWqh1lY8ireIIy4gk0CNOu/ISYLsElO862fCb1XYAbyFMvLcQDb/9xy9s/+kTW/83
suS4wZqOhIO5DN+jx+Q1KRt5Fb6U5Xnn+aLqJAV2IWxzluiAg5z7d6KTvZu81HUmPHGLQqgW
4SOpOj2Js69efJm8decwCD39KyczCmU8haqHiPEeb/B+hB3AW91qORIEse9ltSTZketEhCxl
EGw9XF6c4RRaNL4ATKIjRVsO+74YO+lJs6jyQXjKo7wdAk/jVMsi49fGXcJZN5673bDxjIul
uNSegUP/bsXl6ola/74LT9V24A4ginaDP8N9egq2vmp4a0i7Z52+xeCt/rtapgaeFn4vj4fh
DW6zc4+zwAXhG1zk5rQOaF02tRSdp/uUgxyLlmxuUBqfI9GGHESH2DO2a8VZM8Z4E9Yk1Tu8
zuF8VPo50b1B5lr08vNmMPHSWZlCuwk2b3y+NX3NHyDjJ/FWIuCGmxJI/iGiS93VjZ9+Bx5U
0jeKonijHPJQ+MmXB1ynFG/F3SnJIN3uyCqABzLjij+ORD7eKAH9W3ShT4To5Db2dWJVhXoO
84xqig43m+GNed2E8Ay2hvR0DUN6ZqSJHIWvXJqUmL9ATFuOeIMJU1IUxHc65aR/uJJdEEae
4V125dn7QbrRRKi+2nrkDtm3W099KeqsVhCRX0ySQ7zf+eqjkfvd5uAZW1/ybh+Gnkb0wla5
RHSrC3Fqxfh83nmS3dbX0si5OP5p00vIlK+f4rgpY9Xu6orsuBlSSfTB1to7MyitQsKQEpuY
VrzUVaIkRLP7xWkt26uGxmQGw57KhFzHmTbTo2GjctqRLdLp1KGMj9tgbO6tI1OKhLuAz6og
qcG/+QBiOBz2x2hKqkWbaQbidn+7LJN4a6f20oSJjcEtzDxvcisVmupE0Vm73IjP1FI/s99N
ocf6E5gocQQc1HV5yCnYxVXT4ERb7NC9OzrBKZGzsiwt7vqet2ViR/fIjYIfT30ZbKyvtPml
L6C2PLXSqjnWn2PdGcMgfqNMhiZUnaDJreT05hCMt6FUdcB9pJpB2Tu4eHew1vTNvXyrrtu6
S9oHGBxwValZkbk7KXD7yM0Z4W909JDUPnpLsqGIXN1dw+7+bihHhxelVB+xCictk4gsNwjs
+obxZwiVpgaRNrGz3z6He1V3npFF0/vd2/TBR+sbz7oFk8JtS8FX4Bqi7hMBISVjkPLEkPMG
K5ROCJcVNB5mk8FwHj4ILCTkSLSxkC1Hdjay6Otc5wNr8a/6iZvipYnVj/A/NRpj4CZpySGO
QdW8Ro5fDEqU0gw0GRByBFYQ3He1XmhTV+ikcX2wBnP2SYNP8KfMgBBB4+lZrmHvlWZ4RsZK
7naxAy9gmDC6Fb+/fn/97cfH77YuYId9ET9jHdDJ1lnXJpUsEuY+8LmbAyDtl7uNqXArPJ6E
MWe3qlBWYjiqobPDJhHmuw0ecHLTEe72uBDV6gLZvEVqClytYbxIpPSoFV/Ayh2x3WlQSSaQ
LH8u8T0u9XwzwOR07fun18+2vsWUNu2SJsWqKhMRh9R7wwKqDzRtrt192t4bcbgzHITc3By1
OIuIqtXWW+TqhwyzrSpgUeZvBcmHLq8ycrsasWVSPbSDYk+aJ/cOz9SCDA6hPblSNzy0cNRS
sPPzrfRk/JSWYRztEmzrgkR8d+OgPB8P7jgtkyyYVE28uQrcujALhzYVFjkm0mE2t/r29Rd4
BxTIoKlpa3y2jXnzPruRhlG7kxK2wTd+CKOGCuycceJstYyJULJuRKyzENwOT6xCTxi0j4Js
5zBibcgBCyGvo0yF9aKB19dCN+/qONQ0JwK9JSrTtBoaOw1psBcSNtboNM/pN14kx80WS7wq
T6zq0Ke8zZLC/uDk1NfCp9nwXZdcnB114v+Jgxo3YwEfSXCgU9JnLUjyQbALV/+ec+M4D/th
72hMgxwTZwImAxqNdKevBDUC/WFf5S0h7O7Q2h0WBAHVqEw+eVsEG3FF40yHesoH7VxcXIRa
gdf2QKHde9tfLGE5H0Q7R3hiyGkO/pyfend+DOUrh/puDyQK85cbePIxegycAl03YsII9Li1
qXs00epnPF4Wjf2tpiEacNfn1PKnPplITbkdVwFOGK9KTijIUghQtTwV6cjMKyNGdi0RXzRl
DDUZdYQzsROtaWzX1ABSnBl0Bx+uGdbGMB+FtUF9xiZczbx56kyAE3ZCoOQsbot3gaD3gzxZ
5k6WO1BA7zXOF1gTWwltncdJ4Cpuo+N+EU5n3Wi/jAp3RbhNU1A/1zi4BUeyYJdedKJ/EkBI
yx61Ri2AbchNICjxmBnVScG9xirH+cZs1T/XHSefVRrhoH14OJLQRdFLgx1XcYbtcHKW5EGN
g8WDdLoZMX6PjbpomDo0dMl6WeVEK72BI03UA8wlN+K1WWNKfKM6qgo0BsaMPa6/Pv/49Mfn
j3+r+oaPawe7rhSogfVkNjFUlEWRK3nJipQpTa0osWg2w0WXbiN8TDcTTZocd9vAR/xtE8Sw
2QyWxZA22KsHENe8aHIw5NqxwjO6YSRsUlzqk+hsUKUDV9iyZga3Ws6ym4yKklr++eePj1+e
flWvzN68/+vLtz9/fP759PHLrx8/fPj44elfU6hflMQJ3pL+m9WIHoRY8oYB23TRrcU2Gqdh
uIXenSiYQnO0azHLpbhU+mo27b6MtI0usgDGEDUp+PxMhjCA7ASI8sIB1YAaq2e8e9kesK0g
wG55aTUDtVLAinG6ydARU0PdnpgqAqxmKrmAqfbg9FGluUFxqXDcVwC2FYLlQMm9pWp1BStk
Kcou50Fh8D9vGdhXezVjhXdBcXv5g9HxTPHV3TqBjSzHsKI58kLCXk/yv9Vc8lWtkxTxL9XL
VIN//fD6h55gLH14aC2iBn3NnldtVlSsmaxeXG1wLOg5vk5Vfaq7c//yMtZ06ldcl4Bu8TNr
uZ1Qy2eqzgmFIxq4MgPbG1Me6x+/m/FzyiDq1TRzkwoz2Jev8CxmqrNnH3L0Fg3NpgxYL4P7
oHRttOIwbLlwohBLlyuNdZ0aoDKR5tKg2W1pxFP5+idU5uqgyL7noP136TUGEocBa0swoRgR
42LG2ReZ5DU0GD9gauYhpkwBmzYTnCDdYTA4W2Wt4HiVlutnGD/f2yg336nBvgPhs3hQeLYF
TkF7va5LfB4rGX43llopSLqELpzmaGXNLHqsDNBBFhA1yKq/3Jk13Q5QwDu2WFZQUYLJo6Jh
aBPH22BssYmlJUHEdOgEOt1x2764jeVJ9StNPQT3080Hcp06MCv6nnqOBbw23Z6BZaIEOh5F
JxwNA4KOwQZbS9JwK/D0ApDKQBQ6oFG+F3ga0cSQhGDG1DmTQADb7rBGreTJKN1bGZFpEAu5
37DUyCt/Vv3DipCtcjUExbplID2gn6A9g8CvTULU0RY03IzyXCQ8UQtHzxk1NQxHigzazDaF
2JymMd6UYZNVJuoPteEM1Mujel8242VqCcuw2Mw3is34yEZD9Y+I5bpFLt50cuJXFnJS5Ptw
YIMkmx4WSC8yHUEnG/+zKxQcohT0aSylWhaBqboEa8AT5yBX7apwXYmY8yMpmG+yFf78CRzs
onu04Dfyik3zN40kD9Zdx67RYaaPqZ9zrLbcDa+nhQBvAze96qYxT1SRCTwAIMaSLhA3jZ1L
Iv4XPKm9/vj2HafDsF2jkvjtt/9zJFBlJtjFMTgdw+6YKD5mxCgs5SzfBmBAeL/dUBO27KUG
615ATmAsXoq4PrMtoykEbPFTM/tGgLADT64rKTbbVqeovkS3WRejH798+/7z6cvrH3+o1Q+E
sKUo/d5BjWBsPtQ4lzEMyJZJBuyuWAPeYKAfwEGY/W81tghiYL56Mktja043Khz3pOFB8QaQ
Abo2GaxyO3fwZ4P1AnF5OlZchm7p9K5B64zDoNjsgkasYxRTV6d4Lw8Dr8G8eiF60AatqZ8p
AzYp6OSwCKalAms/KZ4r/8PYtTS3jSvrv+LlTNWcGr4fi7OgSEpmTJAMQVGyNyofWzPjuo6d
spNzk39/0QBJAeimcxeJ7e/Di3g2gEa3UpOBad6Ka2viSdCezxVY26W5O86zCOzGZV87//h6
//KIext6bjuhDfpC2Z3tAknUs0skzzJ8jIJaiY0OQizwEtdOWHx+KnNTg2db/OIzlHaW3Tet
ZwAKNARLCdm76Kmn+KluL3ECkxh9GIBhFNotLfX5rEaVSnVJhOpAqfdQcOrapUWa1hK1taRn
ME2XQ1BYuT+sRTH1uFFANrxro7nvJ4ldiK7iLbe7/FEIPYH026Rew/PNx6UwtqATcdCtObmw
ys893P3X/z5Nx1NIGBEh1ZYOTPWI/mSkoTGJRzHsmNMR3AOjCH3hnErFn+8NF+gi8CTFXJe9
mcgkxRiH5wsMhdR1Y00iWSXAflmxMYzMGiF0NWAzarRCeGsxfHeNWI3hn3Ldl5VJrnxUHDkr
RLJKrJQsKXVd5IXZfPZM9zTyhuSUjdrypaC+5PrDPQ2US6i5stosLLAkqXwNX+5l6ECGZGIz
8OtgXMLpIeoh99LQo8kPY4J25dA2Jc1OS9sH3C8+qrcPB3XyTjdFV0o34lJZ8yLaqyxITiUE
JqHrWztvhdrSeAfeL4DXJrNJLMmK/LTJ4DBEk0MnPUXbCeUEWynBHsjGphTBmWWSBmGGmdxU
eZxhe1DoeLKGuyu4h/G63AkBbvQxwzfajgY2O+APxQDnkDCmDJeAFmFeA9lkMZz2oj1ErZkG
Y5aSW8v8XBSBG1rXWngDn8MrHVqiSSx81rU1GxBQ2IeoxBC+3Zf1aZft9aulOQN44BY7AVHU
iSE+blbgxYzVT2a44h0khQmRR5I6REIg1+ji74yb4vclGenMWVPum5MZcj/SzTFqGbtBGBM5
KM2rdgoS6f6ytchSyx0z0kcYZ5sNpkSPCtzwuEKkRJ8AwguJIgIR68eyGhEmVFKiSH5ApDRJ
fzFufdld1HwdECN2NpSCmX4IHapr9IOYWrQyXx+YeZMOBvdH3eu0gqbzd7VFVopi99/AEh2h
igjavRweUvjGSdcFD1bxhMIZvLleI8I1Iloj0hXCp/NIvcChiCE+uiuEv0YE6wSZuSAib4WI
15KKqSrheRyRlWgdHyz4cOyI4AWPPCJfIRGTqU86/8ZTyJnbxm7ihFuaSLztjmJCPw45JuYX
LXRGgxDO90NmuBmfyV0duomuqqsRnkMSYq3NSJhoKXXKob+Xnpnr6jpyfaIuqw3LSiJfgXe6
BesFB69m5iheqEG3Uzyjn/KAKKlYy3rXoxq3rpoy25UEIaclordJIqWSGnIx+xIdBQjPpZMK
PI8oryRWMg+8aCVzLyIyl6/LqQEIRORERCaScYmZRBIRMY0BkRKtIfVUY+oLBRNFPp1HFFFt
KImQ+HRJrOdONRXLO5+cdofceDG4hC+breduWL7WGcXYPBLdt2aRT6HU9CZQOizVDVhMfK9A
ibapWULmlpC5JWRu1EirGTkIxJJComRuYp/mE9UtiYAaSZIgitjlSexT4wKIwCOK3wy5Oqao
+GAqjE58PoiuTpQaiJhqFEGIHQnx9UCkDvGdDc98alKSR4mp9v2dqeG0hKNhWPA9qoRV74ce
1e1r5glpmxAq5GRH9ipFXB7+6YqqSxA/oaa9aeahxll29JyYmkNhLAcBJayAfB8lRBGF4BmI
vQXRIPu8SB2HSAsIjyLu6silcHg0SK6A/HqgPl3AVP0L2P9BwjkleLDSjX2iT5dCVAgcos8K
wnNXiOhgWF9f8mY8D2L2AUMNdMVtfGo65vl1GEmFfUbOoZKnhqokfKJ38mHgZG/hjEXUyiam
addLioQWxbnrUG0mbTV5dIw4iSm5U9RqQrVz1WTGNZWOU+uHwH1ywA55TAyf4Zrl1Ao5sM6l
JiaJE71C4tSIYl1A9RXAqVKOA9jtx/ghEQKtSwjmQKSrhLdGEJ8gcaIxFQ5jFlTs8aQl+DpO
woGYVhUVNYTsLijRc68JeV8xJUnZhl9gNTIsKClADNJSbG8beLo3HTCKHWud3Z4Y/7djB1YC
yk8bbrcYA4/qYOwMnP3qFjRnfva+s2tH8MTanQ4VN3wtUQG3WdWrl2ekLWgqinSpJw3z/b+j
TIfWdd3msPgQijVzLLNM+CPtjyNoUBaT/9H0pfg0b5UVByrZXj0O1d6Bwrtm1CFAcxaBn9u+
+oxhsfXNegzPukYEk5PhARWdz8fUTdXfHNq2wEzRzndAOjppDOLQ8D7e03B5ApPlXXVVNYMf
OMcr0NH8Qj0NZcONHVE67Xh4/bIeadIuxCWRTqS5neBw/nH/flW9vH97+/5FKqyspjxU8jk8
HsmXxrxoa7Cs82k4oOEQw0WfxaGn4eom9P7L+/eXv9fLWR5vm5YT5RSdvCW6mDxyBN2ioWSd
6MqZobWgXRxYVff5+/2zaIoP2kImPcCUeEnw7uilUYyLsTzx+Wkjlu7sAjftIbttdRvpC6Ve
L53kbYryG1sQoWZ9GOUd5v7bwz+Pr3+v2gTn7XYgHiIZ8KnrS9BpMko1HSXhqJIIV4jIXyOo
pNSNOYIvW1vMye5wJIjp3gcT04tBTNxVVQ/Xj5jJuNgyRg7FDKnbs1Q6RyJJnrGUykzgWVgE
BDOp81Jx/FxsOamcigMBKm1dgpA6pFSzjFWTU8/U+iYcIjehirRvjlQMUL3w4eaoH6hWa/Z5
SlaZ0rkhidgjPwaOVujPVPcTHpWaWKU8MHunfSJYfyHSaI/weNQIyqt+C1Mo9dWg3USVHtSL
CFxOLUbis9/7zYYcCEBSuPKKRzXq/N6U4CZNLLLn1hmPqZ4gJlKecbvuFNjfZQY+PWPEqSzv
QqicfS/rYjBjZlZ6HkJL6pDSGTIxsTwG8DTaBuViaoNS824dRa5AcxY7fmJGqNiuE4uK2YYd
FFaVdonNxig4Ro7d2s0p81wT3LNar5hZZedf/7l/Pz9eZvjcdLwjQnS5HW0J3L2dvz19Ob9+
/3a1exUrwsuroaWDJ34QC3U5mgqiS7tN23aEiPuraPLBLbGomQWRqeNF1g5lJcbBIGPLebWp
F/cz/PXl6eH9ij89Pz28vlxt7h/+5+vz/ctZWyD19ziQxAbUWo1XzuBZvKhacCuqZ0DQFlrV
xpNjwNSDV0ubRLnetYq+CI3vX88PT389PVxlbJMZImNmdOQMF12istxc9y8l4Un73QTn4oHf
15w1KywuvOEcSb4C/ev7y8O3J1Hnk6cYLCdvC0tGAgQrYwCqLOnsOuO+SQaXJiy2dQmq+BR1
Xed2HOkgwNHPNmRwS+Xgglnm+beE6wgNXA1tuZMFbfhJjcKogEnuMh5Pzbh+47VgPsIMVQuJ
GQqYgEzSdt1luv8dYOBq72hXzgSan6AT6KMJm6wK9sSWgSP8uooCMS9CrSAiDI8WAT7VRcmr
3Pp2W6sUMGWs0KHA0CobUo6YUCFV6AqkFzT1EZqkjp3AEBlnjxKbJVxNers7KgtrRqtbmiUA
UcqagINEYyJYYWWxQWc0wIKaaiaT1qv1RlcOR2nGALWVrQOhMG6595XoTaIf5klIyZxWRlUQ
R7Y9FUmwUD/1WyBrbpL4zW0i2lUbANnmGM7fZQadFIjV0jGwp4e31/Pz+eHb27SMAC+23ZNP
KWK3BQHw2LX19AAzDDyj8WCrQk8xat10IGixuI6uW6OUnQ0788imqEwJKUUvqKEVM+dqqWBr
sKGErSWSEKihV62jePZYGDThgL/Y2Cd6RM38UHa+RYSRCbGqJcQUOblPCuw/CRCXaCZQgXIe
xLUXmMkcWAin2wjTH14oLEnTmMAShMG5LIHhzrZooBsd+xAk7tEGme8pEyK602R8e3axpGk7
RF6IbXUEw19tPRiaC5cAYItkrwzj8L3xfOoSBk4t5aHlh6HQhH6hQHhI9B5pUqZcoXFF6KcJ
yTTZoEuiGjP1h7po3Y94MUmCuisZxJI4LgyWUC6ctSJobWPpZJpMtM74K4znkpUsGfKbt1kT
+mFI1r+5tGi2WeXCv86MoU+WQskFFFPxOvUdshCCirzYJTuBmD8in0wQ5uKYLKJkyIqVipwr
qZmTqcnQlYdmWo0act9w0WdSURxRFBZ1TC5M1qIlUUBmJqmIbCokFVkU3WklFZN9E4tkNpeu
xzPUITRuEmQtC60Gb7gKMKkkpVMVsh89VoDx6OQsefHCdJsqo2bq09qUgAVAjdvu70qXnke7
MUkcujEllaxTKU3p73cu8HKIT5GWjKgRtqSoUZYEemGwFKhxanU8jYzl1OImpJHQjXwyLhbC
TM7z6XpUIhjdA7DQZnN035ecu15OU7hDHFmjigvWy2JIddp6L++yCcK+YjYYQ3DJy9wajoA0
7VBtK13rXZ6Gykcl6oH15Qziy/nx6f7q4fWNcFSrYuUZA1OFc2QrTeVM7zSMawHgtHUAK4yr
IcQWWFoCJkle9Kvx8jUGKuEDSn/fNaHqgX6N6+zCnIpRexw1VkUJpuI1iwIKGoNayOT7DVj8
y3TB80LbUbJitMVGRSiRkVUNjNus2el+yVSIYd/oIqDMnJXME/+swgEjj7PAWdwpr41TDcUe
GuN9ksxhs9/CVSGBjkxejhNMwVS9VTuKHDcY9axefMHFh7S6jYIL81Eu3nrpVESu3weMGyt7
QBrD/d3Q5RUyAgTBwBpfVmTdADsEN9Ip8OkFR1iy/bSWk1wJBtN4mYMCwaluOQcHp8vpoByU
6Diwtwe7AAxfz30+Oy/Q7UdXutHOqpfACUKZcFMusQ28z8MVPCLxTyOdDm+bW5rImlvK64JS
/uhIhokd0M2mILkjI+LIqgFzlVrN9LnmtcFI4mI+7oJVhvqbKoNpqapHJs160/oj1FoJtl59
8zMNO/+w2PZlxu4MVwIi/13bd/V+Z+dZ7faZ/shcQMMgAlVWcx11lTv5PTv7b2kY/qeFXWOo
0Z0RTZhodoRBk2MQGhWj0AkQKvoegUVGE85GXIyPUcYhKrMD6DZeoJrhotZELGd4C6RswbNq
GPQVAmg9C7VigMOjy+Kj7pzO/3m4/4LNe0JQNY9b87FFGM7Pf+qBdlyZRtQgFhpmhWRxhtGJ
9L2yjFonuty1pHbalM1nCs/BYC9JdFXmUkQx5NwQNi9UObSMUwQYFO0qMp9PJegpfCKpGvw0
bfKCIm9Ekrq3XI0B31cZxbCsJ4vH+hTeUpFxmkPikAVvx1B/mGEQuia9RZzIOF2We/pW0WBi
3257jXLJRuKloR2qEU0qctJVaG2O/Fgx6KvjZpUhmw/+Cx2yNyqKLqCkwnUqWqforwIqWs3L
DVcq43O6Ugog8hXGX6m+4cZxyT4hGNeweq1TYoAndP3tG7FqkH1ZbBHJsTm0hitSndibDn41
akxCn+x6Y+4Yxlg0Row9RhHHqgfbYmICJ0ftXe7bk1l3yBFgy9QzTE6m02wrZjLrI+563zTf
pibUm0O5QaXnnqefTqk0BTGM854re7l/fv37ahilxRG0IKgY3dgLFm0TJti29mSSxCZloaA6
wFKfxV8XIgRR6rHihgU9RcheGDnoPYDB2vCujQ1PeTpqXn0ZTN1mhhBnR5MV7pwMy6Cqhv98
fPr76dv98y9qOts7xhsBHVVbtZ8k1aNKzI+e2LMf7aQmeD3CKat5thYL75VOA4uMNzA6SqY1
USopWUPFL6oG9idGm0yAPZ4WuNqAlyn9pnamMuMWQosgBRUqi5k6SYWYWzI3GYLITVBOTGW4
Z8PJuPObifxIfihoLx6p9HfVMGJ87GJHf96m4x6Rzq5LOn6D8aYdxUR6Msf+TEqZnsCLYRCi
zx4T4MpcF8uWNtmmhktLE0e7oZnu8mEMQo9gioNnvFNZKleIXf3u9jSQpRYiEdVU277Sb0GW
wt0JoTYmaqXMr5uKZ2u1NhIYfKi7UgE+hTe3vCS+O9tHEdWpoKwOUda8jDyfCF/mrv46d+kl
Qj4nmq9mpRdS2bJj7bou32KmH2ovOR6JPiJ+8ptbjN8VrmFdizOuwvdW9994uTfpFnV40rBZ
agbJuOo82kbpD5iafrs3JvLfP5rGS+YleO5VKHniNlHUfDlRxNQ7MfI0ZdKc++ubNEP/eP7r
6eX8ePV2//j0ShdUdoyq551W24Bdi51rvzUxxisvvNiZg/SuC1Zd5WU+W/K2Uu72NS8TONw0
U+qzqhH77aI9mJyok8WQ4aSyhiSKWfd57KqtmPq4CH9rV4kRBvyM7tEp3qlgURBEp9xQMpsp
PwxJhl+fxnZvo8z34NoYwXvU9mC9N/6BUpXX+UVuGF9t8+mIm8IIc4+TIMACPxadq9uiD7bt
Ieroaejs48qZGQdUC/IZz1ghKUvp7VW6bdhpsQfT17XZzstxMN3MeVugMQAvlsaiRfiiVP2p
K9FnLOTY4YabOVZ06/GsM8mZnk+zpZ+Z2vAzMzVrxsQuRzRb2J12+nNDTFMF13m2xQU4emLo
s6zrUdHnmJOq4I7jfixaZAODhyKuR1TDE6wmSizlA12U9UDGk8SJyU9ci4e8yVyGW4labVZi
3xa63RWT+4Qbe4mWo6+eqZHjFAeYRlDbKpS+HpF3o2PZ7O37ARWrYFgYBsvb1KDh1uQojaGt
jJixYiiNsTIMFGmgnHhRCkDA1YF04hMFKAPPumZYn6zhQutXU7new3M8xGSnEwsMzcGUiVm4
l/tVtnL6EtzibIerG0axUjKW/wka4cR6BrIGUKawoS4Jl5uWnyY+lFkYGzfM6k6xCmLnaJ4A
TNgSUvn6MLFLbPuAxMaWKrCJOVkduyQbWecJrE/s06+Cb3oU9Trrb0jQOrS4KUvdY4MSBUCy
b6xzHZalupyn1aZu6GHKKMvi2ImucfBtlBjKUxJWaoj/Xn2OCXzy42rLpouuq9/4cCUfdGhe
dy5JJUfci7ZPb+cDmDT9rSrL8sr10+D3qwz1KBhA26ovC3vzNoHqRAjf/MIBh+aBVGYO7yJB
Y18V+fUr6O8jeRT274GLRIBhtG8Q89uuLzmHgjDTBYUtPn8gWNs+Q2D8VFkjVi7jgy+4fhhx
QVeWHHk1rKQW7W7y/uXh6fn5/u3nxZfSt+8v4ucfV+/nl/dX+OXJexB/fX364+qvt9eXb+eX
x/ffbeUCuCjvR+kdipc1nNLb+gXDkOXXdqHgKshb5HCwJ12+PLw+yvwfz/NvU0lEYR+vXqXD
mH/Oz1/FD3DttFjKz76D0H6J9fXtVUjuS8QvTz+MzjQ3ZbYv9I3qBBdZHPhouyHgNAnwqU2Z
RYEb4gUJcA8FZ7zzA3z2k3Pfd9AZVs5DP0BnkYDWvofXxXr0PSercs9H2559kbl+gL7pwBLD
ns4F1e1DTX2o82LOOjQg5JXwZtieFCeboy/40hh2rYsZKFJ2wWXQ8enx/LoaOCtGMOeGZGcJ
+xQc6dZ+DJhaxIFKcL1MMBVjMyQuqhsB6pYkFzBC4A13DMPvU6+ok0iUMUJEVoQJ7kTFIY1d
9JkwtbsuCqxgPI+BPmUcoDocxi50A2LaE3CIez8cjzl4rBy8BLfDcEgNa6Aaiupp7I6+shqn
9RIYyvfGSCc6V+zG1AluqMaultr55YM0cBtJOEGDRXbFmO6heGgB7ONKl3BKwqGLhOwJpvtz
6icpGv7ZTZIQXeCaJ97liCK//3J+u58m3NXDdrGSNrCjru3U2tGL8PQIaIjGSzuGZFiBoiqT
KGqNVgwXKoU4wm3RjmmEu247un4Sojl35FHkoa7LhpQ5eE0A2MUNJODOMPu5wIPjUPDokImM
RJa8d3yny330PU3bNo5LUixkbY32Vzy8iTK8CQUU9USBBmW+w5N/eBNusq0Nl0NS3qCq5WEe
+2yRJrfP9+//rPYzsV2NQjwiuB8ZrxsUDI9i8IUVqLoHkTnon74I0eC/Z5BeFwnCXCm7QnQs
30V5KCJZii9Fjj9VqkKg/Pom5A14B0qmCoteHHrXfJF/i/5KClt2eNiTgS02NXkoae3p/eH8
DM95X7+/2+KPPaJjH0+xLPSULUaV9SRRfYdn16LA768Ppwc19pUcOAtVGjFPCtjsxnLIVrHj
/1F2ZU1u40j6r+hpYyY2ZpuHDnI2+gEiKYk2LxOkSuUXRnVb3eOIsstbVZ4Z769fJHgImUhU
9z740PeROBMJgEBmesgz1o3Sowd9isYcdpKJuA67z8Wcb97JxdzZC3hOqxMXRbxcmtQO2Tkg
KkaaCFM7B9W+26wrvmYwIfq33mryN7v8KP0tspfVK+75Sumo2L+/vD59+fy/V/gkP67w6RJe
Pw/hMhvTS77JqeVvFJi33i0SWd9h0les72TjyHRyiUi9iXW9qUnHm6XMkcQhrguwWTThto5a
ai50coG52iOcHzrK8qHz0aGmyV3IzR3MbdARMubWTq68FOpF09exze46B5us1zLyXC0ASguZ
SVoy4Dsqc0g8NAFaHC/fI+cozpSj483M3UKHRK0VXa0XRa2Eo3hHC3W9iJ1iJ/PA3zjENe9i
P3SIZKsWaa4euRSh55tHT0i2Sj/1VROtl6O5SRO8XFfpeb86zDv6WeFrY4KXV7XMfnj+tPrL
y8OrmnY+v17/etv84w8ystt7UWws7yZwax0Lw+Wm2Pu3BW7VjoWgqpFTGfq3uD6kWL8+/PJ4
Xf3n6vX6rObc1+fPcLDoKGDaXsgZ/ayNkiBNSWlyLL+6LFUUrXcBBy7FU9Df5J9pLbULWfv0
1FeDpn2MzqELfZLpx0K1qemJ8wbS9t+cfPTlYW7/IIrsnvK4ngrsPtU9xfWpZ7Vv5EWh3ege
suaZHw3o8fg5k/4lpu9PgyT1reKO1Ni0dq4q/Qt9XtjSOb6+5cAd1120IZTkXGg+Uilv8pwS
a6v8ECBO0KzH9tJT5iJi3eovf0biZaNmU1o+wC5WRQLrns0IBow8hQRUA4sMn0Lt0iKfq8ea
ZF1dOlvslMhvGJEPN6RT54tKex5OLBhCLpUs2lhobIvXWAMycPTtE1KwLGGVXri1JCgNlEZv
GXTtZwTWtz7ofZMRDFgQ9g+MWqPlh/saw4F8qh4vjIBZS036drzsNL6wCGQyqWKnKMJQjugY
GBs0YAWFqsFRFe2WHVcnVZ7V0/PrP1ZCbUs+//rw9af3T8/Xh6+r7jY0fkr0BJF2Z2fJlAQG
Hr0dVrcb7DN3Bn3a1vtE7TepNiyOaReGNNEJ3bDoVlA4QPcul9HnEXUs+mgTBBw2WOcjE35e
F0zC/qJicpn+eR0T0/5TYyfiVVvgSZQFnin/4/+Vb5eA6f+ympnvQBqvqv3s449pj/NTUxT4
ffTV6jZ5wJVDj+pMgzK2zlkyx+mdP1ysflP7Yr0EsFYeYXy5f0d6uNqfAioM1b6h7akx0sFg
1b+mkqRB+vYIksEE2zc6vpqACqCMjoUlrAqk05vo9mqdRjWTGsZqC03Wc/kl2HgbIpV6JR1Y
IqOv75FSnuq2lyEZKkImdUcvMp6yYjwyHU8rn54eX1av8LH4n9fHp2+rr9d/OdeJfVneG/rt
+Pzw7R/gCseyp03Na0Tqx1DmEFNeGkaigKaNGngXHR4JXVHXnI55VJaDzIoDjloN9PtSQk0a
NBdM+GE/UyjFg7ZUZZwZ38j6nLWjNaVStCYN97MHtWtIbweY6PWuIxU+QjBs8LfGFATK6OJ0
gLXlrG/6+r56sg70jFfgAD45qbl6i4swHswXKKTojFeXRn9UiG+HzSJpVn8ZjwiTp2Y+Gvyr
+vH1t8+/f39+gANgnPP5mJFq92mBgfFmxJ2+V4EZcAID4YTNazWAN6LKFk/B6eeXb48PP1bN
w9frI6m3fnAozqlkErA+09yYvKrqQold4+3ij6ZN1O2Rd2k+FJ1Sd2Xm4U8IRgbTVZMijVHs
OKNoijyuN6bfihup/hZgJpQM5/PF9w5euK7ezkhus/BkGm2wj0RC8KloC9Pig6+2wr68mPtv
6yHprcPOLzLHQ3nXgn2TWnnsdlF8xs/s2zw9ktExvrcwqGdvDrT2z58//X4lnTxa3avMRHXZ
oauvWkn0pVpBHcWQigQzIBZDVhHbWK2KsqMAD+MQniFtLuDn45gN+2jjncPhcIcfhnHSdFW4
3lqN2oo0GxoZbQPSJWrMqT95hIJ1jUQe42vyoDlqecr3YjpqROtgYPOhOzQoctk8hK1zL0IM
47n9D5ZW0wfWttwIncBBnPYDuQRg0nkgOVq0SXPsaXWqezQ3TMA0P+xzjlFb8vADUf8FdOI9
rgNEkm9FldaLCj08P3y5rn75/ttvSnOm9AzlYOyOZq2udfwtWTWTJGUKMbgQpt1O3JseUhWY
pgkbHkBR2om3WugvviUYL2SQ1QEu6hRFi2w4JyKpm3tVQGEReSmO2b7QBmRmpsC1ak5r8ktW
gGHtsL/vMj5neS/5nIFgcwbClXPT1vDZXQ2rDn72VSmaJgO3aZng8z/UbZYfKzVg01xUqK33
dXe64ahV1T8j4Wp3VbSuyJiHSM2R3wXoyuyQta0qsR4/ZopSKRslZ64MS5FA5FvJ5wW220V+
PHWogvDCtBSQiOjyQreuGglHVqL/8fD8abyVT0+coPuLRuIrDdAVIIQIqRtQkm2GG0D6KXH8
CeUpzSE+AYNIkqwoUMGJP0aNyKQ/kLKYSwWQ471aNV26NbKOVbgdZvOwHyYPcggrM5iX6jJD
6L5VSzd5yjIs2KKvh/d+7F1Y1GNRUicJXwJQ2NCpe4ciSW2vLACONu6je4bbi8AU64Pa2q+D
zjw50EQplf47Hsw9jca7c7jxPpwxmhd5HJhTzAyiOF4AdmkdrEuMnY/HYB0GYo1h2zJAV3Cb
bcOSpEoXQ4CptUm4jQ9Hc3061UzJyfsDrfHpEoUbtl355rvxU6QHtktmj5AWg9xR3WDqQc94
oYzitT/cFWbozhtNfRrdGJE2UYRjRSNqx1K23y5Uq23oCScVs0wTIW95N8b2i3XjuDC5S7sj
f35GTudN4O2KhuP26dbnR49aOFySqjInFaV1JcQQZfSqPuDldahe0UyKU+1iXp4elaqclp3T
tVZ756r3uOqHrE2/3ghW/xZ9WcmfI4/n2/pO/hxsFmXRilLtZg8H+AY/pfzlDXKKv6zmUTXD
tsYqh3u2rTuynVXr5Rr/giiiasOjL0VzhGpef8sySdF3genQVNZ9ZYa5gp8DOPXBDnQxDr7a
1djNjeDhEqVSpaPrUQw1SWkBQ1akKBUN5lkSbyKMp6XIqqNautnpnO7SrMGQzD5YigXwVtyV
eZpjMKnL8TZzfTjAlwHMvgMXLT8oMhnjo+8ccmwj+CSBwVKt2Vqg7Pq7wAE85uSVtBtnbFnc
Ng6HTTpvobpdtGpXHQaohcYJbVDTMXb3pfNp62Q4kJTO4JpaZpp0c3nVkeai17xnaH7JruKl
7SvutXOp9AatvOrqHoKstIwEwDC24PFpu+XhDRCOITuDa3+Ws1G1dLGJsunXnj/0okWLaS0g
TRHqvY16mV13Tg+tuYeMR0QS76gzLN201KRFg3ZDiALFYdDNy1ala8SZQhJF09Qtob0N9f52
g24TLW1BhFxJXimq4EKS0ZUaA5hJcSaSQcg59sTP3jglnNK/6c9dxkUt0A2pIL7HZjS7dA5G
KQP91VDt8D9mhpGTLvkFQidC35AGoENIdLswCcwDMxMdOgghr6bNvGvVRPgzRMrxzAfBJPkH
Aei2fIZ74dMG1mbbIhcfHDC1HFmSkn4QFPZLW7A4seFTfhBUxe6TFH/jnh+GfffWhps6ZcET
A3d1lU2eIAlzFkoALxiHMt/lLRGjGbX7MLWmi/pifkECJJd6x2fnU7fviT7ZZ/t6z5dIe2RA
h3GI7YRELloQWdZmLICZsvtB1okFjGMIovj+oMwcVg3Ps9Zj81xpM8LSjiM4iIv+qOQmZZPm
B4YuYcw3dACBBbFVtwUemtRJSfkmjcwu7TffpikV+yMjyvgIwY7AvMR3vQ/eUT2qCs0kLps/
SEFvpVJ3m6B4DuMoHeMoAc12TnJ/RPapgE9RyKzWz7RnSorONvZsFiZZJkLbGU++CZLJsgkO
GQ/P1+vLrw9qpZ80/XJnKxlt2W6PTuZszCt/x9OB1OuaYhCyZUYHMFIwYqwJ6SJ48QUqY1MD
K3VY5lgSNZNKzyPXAFrxlHPDk2aa9jyk7p//q7ysfnmCuFJME0BiIHTmJVqTy2QUohDEBieP
XbGxFPzCuhtDjPd9WyKm8L35lG8D37Ol5N3H9W7t2aJ1w996Z/iQD8V+S0q6xBG1UjWZKXxo
uPOGlE71uqpHWwOCk0eojen3gHIQh5El4QSiKNSAdT6hm9aZ+Mi6k88l2CPm9aC9EVQQ4FaQ
+k9rG3bi+IBiMM2ojjE0JE3vouxvTZjPmw+Rt724aAG0v7Vp2bGJTs8Pcs9UYY5F+/YQkt+/
XZ9P9pCRp7WSYmY0QwhBHuXWa5gb7MXM8kAv6Z5W1ztfis/GsAmDlXpushizPofckgHjeFY5
jRQ7JUxvgaC1t3Pmx8d/ff4KRh1We5J8dZwvZseiiOiPiOng0OLX3NJCww5FdOkOzVHw9dPH
adNafr48DJkzJiCzvBXFWD5udTPFoLGIu3I49XvmDUWIlOt1AceZHttE85rOxaV+FDLjR+Fx
yAy7EceOfQmHwueYXMRMCSLdhch9540Q/dB3ecGuC0Xvh7vQwezoZufGXJzM9g3GVaWJdTQG
sJEz1ejNVKO3Uo3N2B2Uefs9d57YYtVgzhHdhtwIvnZnZA5xI6SPrFAX4v3ap4vbCd+YXstM
fMM/v6Xb6RlfcyUFnKuzwnfs85sw4oZKkWy2AZcxECGTw74bZMLo/eSD58XhmemhRIabgktq
JJjMR4JpppFg2jWR66DgGkQTG6ZFJoIXqpF0Jsc0pCa4UQ3E1lHiHaNUNO4o7+6N4u4cow64
y4VZ7k6EM8XQjJdj4DpoGEOADwOuPpfAW3M9My1lHbq9YJoyFTsU2gnhrueZmmucqZzCkU/c
Gw6Bwxk8rwM/4AhrRwro6K6Hr24mdz4n8LBX4ZaIrj3MiPN9OnGslBzBISkjdSe1jia3bZYF
hZYRblzDpbWhfR963OScS7HPisL+njMU5Tpeb5h+LMVFzb8RU92RiRmZmBimczQTbnbM4mWk
uNGnmQ2n6TWzZSY1TcSceEwM0zgT40qNfn2+5c8RUu2B1XbhDo61uQUmeWYKCWM/pHbz/pZb
DACxi5kBMxG8GM4kK4eKDD2P6WkgVCmYTpsZZ24j68oO4h3yqW784N9OwpmbJtnM2kLNtEwz
Kjxcc+LYdgE3Zys4Zlqo7TYbnxFQhW85FQI4W5wOu3JAOCPNgHMTrMYZLQs4J68aZ0a/xh35
chOoxpkRNOJ817g/+1C/YDf8WPL7mZnhJWRh2+yIosTcHlj2z465wrH5k7IMNtx0B8SWWyBP
hKNJJpKvhSzXG07pyU6wUyjgnPZS+CZghAS+58S7LftxRG1/BbOx6oQMNtyaTRE48JhJ7Hym
tJoImOJ2BxFHO6a8hqumN0m+Oc0H2M64PcBVYyaxq3Obtk4kLfoPiqcfebuA3L57JNVKglvr
dzIUQbBj1gOjiysmPU1wG/LFuR3FwVkF93zpg6f67Myor7vSPuib8IDHsetshDNSOUXVZfBo
48I54QKcbYsy2nHfJgDnFhgaZ7QHdzSz4I50uB0r4JwG0Dhfrx2n3jXOjALAI7ado4hbt404
L/ATx0q6Ps7iyxVznw64468Z56ZZwLnNhj7RcDzPff9xnYAAzq1wNe4o546Xizhy1DdylJ9b
wuvgiY56xY5yxo58Y0f5uW2Axnk5imNermNu2XVXxh63OAacr1e889jyqG5h+yvecZvbj/ok
Ld4iq8KZVFupaOPYRey2ro0Ut2qywscuRBFsfe5LQAX2qZxkAxFxqk0TrqQibgfVNWLrh56g
Vdd2VPoYjv38eqNZQiY9Q45rsWMrmtMfsPb7y42E6Yv7KU/t84OT6aNY/Rj2AiKa3euIc9Wx
M/xsKhYFg+utd2/XisdzlG/XX8GKFjK2PvTD82INwRpwGiJJ+q7ubbg1z20XaDgcUAkH0SBr
tgUyo7JpUJpn8Brp4XYSaY2seG+eC45YVzeQL0KTU9a29xTLE4h3h8G6lYKWpmnrNH+f3ZMi
JdoBC8GaAHmq0tjoNBaDqreOddXmEhn3zZjVcBkYhJJKgf9V83RyxGoCfFQFp4JQ7vOWSseh
JUmd6gLFjhp/WyU7dtsoJA2msmSk5P096fo+AaO4BIN3okDx6XUe9+14OxiheSJSkmLeEaC7
y6uTqGjxKpmr4UMTLBJ9zY6AWUqBqj6TVoZ62KNlRof0nYNQPxqjrgtuNjKAbV/ui6wRaWBR
R7VWsMC7UwbWUrSvSqGau6x7SVqpFPc61CxB86StZX3oCFzDSToVqrIvupzp9KrLKdCaEWAB
qlssaDDkRNWpMVvUppwaoFW1JqtUxSpS1ibrRHFfEd3UqIFfJCkLgj3dDw5nzJ5MGtLjiSyV
PAPhLjFRiEobpSZEWegr9KQSLdgJUflv6yQRpA2UPrOad7K0JSDShtrXL21l2WQZGBLS5DoQ
NzW7ZKTgVjA7XciSiMSxzbJKSFOXLpBdhFK03bv6HqdrotYrXU7Hq9IwMqMDuzsppVBSrO1l
N92/XhgTtXLrYSIeGhnilO6Epazv8hzHbgLwkitBxtDHrK1xdWfEyvzjvdpMt1SxSaXw6haO
31k8UZWBqPX6F5l2i2ZZoui4NtwyZbwSa40nY0BMT4xmAyix/dPT66p5fnp9+hUcbdCFiHaO
vydRQmcNtjgYYEsF1xpQqXSQrVOSY4NKEm2A2rvpq8Nj6DyEiRbUt5DDKcH1JI9VldJKSTZU
2Z0R0Jjx9QkNYrmnH2Mw6fveA1gF5ZIUzWUCoevaHYe7kxr8hfUaUPtCazTZablANOisAfT0
Ucm3AvB1nbELKgzcoWBSMzIkyGMsghdLh5s8PL28gokT+Fx5BFtnThqS7e7iebrZUboX6Fke
tW9jLRQK93xDz6poDA5uIDCcsblqtAVzadW0Q0caX7NdByIh1cI0ZdgTa6+oe+7SB753auxM
c9n4/vbCE+E2sImDEgO4f2gRarIJ14FvEzVb3RkdJBX/+u3K9H7IFEsWkc/kvcCqQjXOpo3A
F43aUFkvzQFs1P9P0qZPd4IBE31PWNiopDIOoA49A6Z7pExmzqYKHS34V8njw8sLr/BEQtpJ
mxdlRPTuUvJUVy6bu0pNK39f6VbrarWryFafrt/AFQ44AZaJzFe/fH9d7Yv3oJMGma6+PPyY
bxk/PL48rX65rr5er5+un/579XK9opRO18dv+srgl6fn6+rz19+ecOmn50jnjSAX4nWmYH9n
Batd3hOdOIg9Tx7UYgFNriaZyxR9lTU59X/R8ZRM09aL3Zz5Ic7k3vVlI0+1I1VRiD4VPFdX
GVk/m+x7uJjLU3NQEtVEiaOFlCwO/X4bbEhD9AKJZv7l4ffPX3/no+GVaWIFvdFbBNppeUMM
jEbszOmOG67vhMqfI4as1NJFLYl9TJ1q2Vlp9aadw4gxIld2PazOFtuzGdNpstZpyxNHkR4z
zvPF8kTai0Ip/yKz82TLovVIqu/l4+w08WaB4K+3C6TXCEaBdFc3jw+vagB/WR0fv19XxcMP
7QecvgbxS7focOSWojRdTSxwf7FigmtclGG4AQdZebGE/S21KiyF0iKfroaLaq3u8lqNhuKe
LHXuEhLcCZChL7QJGmoYTbzZdPqJN5tOP/EHTTcuUOYAR2TZBu/X6CB3gcf4cwwBH5nA2Iuh
iLADGFCRAcyq9+jk7OHT79fXn9LvD49/ewYLcmj21fP1f75/fr6Oi8/xkeVS+KueBK5fwcHi
p+lSMM5ILUjz5gSuxtxNGLiGw8jZw0HjloXswnQtGCGXuZQZbFgP0pWqLl2d5glZyp9ytTHJ
iCad0aE+OAjQK2xCoxriqUk0yVJstyVjZAKtncRE+FPmqAOWd1TuunWdkj4/OQq79SzzpCX0
IB1aJtjVSi8lOhH/P86upblxHEn/FUefeiK2d0RSpKjDHCiQkrgiRZqgZLkujBqXusrR5UfY
7p3x/vpFAiSVCSTt3r1Y5pd4I/FKJDL1uqPfxXLYKIZ+Z2i2tTNESnK1+V5NEZtdQGz6Ipot
JEYksQ3wfSSi6GPTNnM2B4YK+lDGYkzmnoyGtGu1s7a90Pekfr0uY5aclcTnJaKs2zRXbVSx
xGNODu+Iktf4PSwm8OEzxSiT9RqIHZbr4TLGno81/ygpDPgm2ajdzUQn5fUNjx8OLA5TaJ3s
u9rZZxH6h3HLumH5c6AfZOLHn4ewPRlyQZK/EGb1WRhv+WmIzwvjLW8+D3L9V8Lkn4WZf56V
ClLwk8SukDzr7aoV2KwTPOOWou0OU6ypjS7xlEouJqY3Q/NCeMPmCo1QGOJKDtNOh8lxtk+O
5QSX1oVPHL8gUtXmEXF0hGjXIjnwo+9aTfgg42KJshZ1fLJPOz0tWfMTMhBUs6SpLd0YJ/qs
aRJ4112QmzEc5LZcVfwSMjH1iNtV1mizJRz1pBYQ54zYz/Y3Ey1tfEnypHKf7zO+7yCamIh3
AimrOgzwBcnlduVs/4YGkQfPOcj2HdjybH2o00W8ni0CPprZfqHzH5VIsqt9VuaRlZmCfGvt
TdJD6zLbUdoLm9qiOUeGIttULb2H07AtphmWUXG7EFFg0+CiyOrtPLWuvgDUa2pW2Aygb6Ud
J+q6GrlUP8eNvboMMJg0sqSpVsHVHnYvsmO+apLWXrLz6iZpVKtYMLUcrBt9K9VuTsue1vmJ
+nE3mzm4q1pba+etCmd1S/ZFN8PJ6lQQXKpfP/ROtsxL5gL+CUJ7Ehooc+KYUTdBvt91qim1
Kx+7KmKbVJLcUuseaO3BCndPjIREnEDXwJJrZMmmyJwkTgcQ+JSY5esf76/3d19/mmMwz/P1
Fh1FhyPaSBlz2Fe1yUVkOTIdM5x+K7jbKyCEQ1PJUBySAXNk3XGFr33aZHusaMgRMkeB1e1o
H8Y5SgQza7NbyhJuAiionXTHJy+ildOtqs4zap+Z3birnTldWBUwJw7mkNdT2GMejgXWPDP5
EZ0nQqt1Wh/GZ6iDUGx/KDtjhkyicONqMhpPu/DK+eX++cf5RXHL5ZaCssoaBoY9ow1ydls4
1W0aFxtk2RZK5NhupAvZGpPwuH1hDfny6KYAWGBL3PeMJE+jKroW6VtpQMGteWSVij4zKj9h
ZSZqQfX9hZVCD2qzEVxnn3I1u1g1NIbsHMl+ka/AkEsliZ6I7iJX6L5WK25XWINyYA8bzWC9
sUHrHXyfKBN/3VUre15ed3u3RJkL1dvK2YeogJlbm8NKugGbvVrlbLAEIwSsHH8NQ85CDonw
OGwwb+ySfAc7CqcMxCSXwZzr3DV/NbLuWruhzL924Qd06JV3lphgi0CEoruNJ+0nI2UfUYZu
4gOY3pqInE0l27MITyR9zQdZq2HQyal8184sjEiaNz4iOjaw3TD+JFHzyBRxaysg4FSPtjjv
Qhs4aore2t0HyhjW3oUO/H6iom2BQLYN1IxibdTaLdf/ADtdv3EnD5OfM3oPewFnnGlcF+R9
gsaUB1FZUd/03NK3iDERZ5HYaVNbLWR3Hvy0IFJjyIuZ/2H3tssTG1QjX+2SbFTrurEg1yAD
Sdgi5I07n226dLWBGwMiwjVob2RyQnjbh+HmsU13k62MYbXLVubpX9qY/E/Y7r5rb9vt+/P5
N8boR3tb47dm+rM7CFsIo45LWmWE5q23jGQPe7hZkQ+4g6dA7s3jGdrQl9jTl/qwd5T1TQMW
KTMSrgdlGi+wx9EBtr2fqlRXRYVFByM06NaMN5MS9Lp7G5cocH+wMbdbpfi7TP8OIT/XaoHI
Mt2KnKanoa63TS4lUe+50OuiXZdcxGqtbalxJNCb3YuMI63hF4sLUEnAAColwPVWt5UUdC2d
6zRqq3rpjf3N1UWh9mVZD+8CK4Mt/ODnj4AeD3Q7DthBboWNpNs8UqczK2SvkkBPXJoxjBk5
ChJdoku7nbI9lgWVWSnbnHBaj1CFqPL88PTyLt/u7/5wx+QY5bDXorYmk4cSLd+lVJ3lcLQc
ESeHz5l0yJFtE9Buo4quWoVMm+e7hLpgnaVurCmrBkQWe5DpbG9AKrDfaPGhLqwK4TaDiSbK
iFg8uKChjYpa4AtZjWmb6jMODFyQmFbRYNmq3O2QKptlGNhBe9QYGqctRW2Pm9zqYDmfM2Bo
p1vUYXg6DWqKLg37DLuATu0UGLlJx8QLwgASuwKXymGD7CMaBTZqjMvDU972YPOHbbG+B4Xn
z+UMPx8z6WOz9xppsg24xsISNMMQqR/PnOq1Qbi0G8J52GTUIUUShdjUu0ELES7J41qTRHJa
LCInZeAq7DJNg1VL9JNM/Gy/9r0VXtA0vmtTP1ratchl4K2LwFvaxegJ/mk0W3YZRlqF658/
7x//+NX7m94TNJuVpqsNxp+P4K+LeSJ09etFU/pv9kAEKZ/dHQd5cU4Oibcv99+/u4O4Vza1
J5BBB9Uyq05o6mxDFa4IVW3HdhOJlm06QdlmanVfketfQr+8BeDpYBmQT5kZ52NJez1fPYR1
e90/v4H2xevVm2m0S8/sz2+/3/98A09q2q/Z1a/Qtm9fX76f3+xuGduwSfYyJya+aaG1S/oJ
Yp3s8UbebEnyVV7kLTq3JJ53q6bxJC+0ewHL+0DTCm18mABmiSDQVrSVvOXBwbr9Ly9vd7Nf
cAAJYtmtoLF6cDqWtZUEaH8ss9H3kgKu7gffXohpIaDaja8hh7VVVI3rzZILE8P5GO0OedZR
E/q6fM2RbCxB2x3K5CyFQ+A4rkti2mwgJKtV+CXDrxIulBMbY9UIteavXEIqqcsYiqvFm2jL
W1ShOPCA3UFgOn5mTPHuJm3ZOBGWFw749raMw4ipq5qYI/JIGxHiJVcpM5VjIxEDpdnF2B7N
CMtQBFyhcll4PhfDEHwmyknhoQvXYk2f/BPCjKu4pkwSYq6p5l4bcy2lcb4/VteBv3OjSLWZ
WmIHLwNhXQZewOTRKI70eDzET6pxeJ9pqKwMZj7Tqc0xJtb8xoKG4ylZHVg+HmnQDsuJdltO
8PGM6WONM2UHfM6kr/GJ0bfkOTtaehz/LolJyUtbzifamLqFJ/w+Z9jajDWmxorlfI9j31LU
i6XVFIx1UugakFh8OhmmMiAaHxSfmqhM8ViuUR24FEyChjImSK86Pimi53OTi8KJW0SMhzxX
RHHYrZMyL26nyFiLkFCWrPogCrLw4/DTMPO/ECamYXAIUwPtg0Vt4q1Ftafq5ZYjD0Vge9uf
z7gBaZ00MM7NlLLdeYs24Th9HrdcJwIeMEMbcGwAa8RlGflcFVbX85gbSU0dCm4MAzsyQ9V2
6jXWrM7wwyY0ECyfXQNlfxDsKvrldn9djpamnx5/Uzvhj/k/keXSj5ikekv5DCHfwAPaiimw
DIQLGuv9TBs1c4/Dkzbwk3oxYzdR7dJrVIG5ugMNnBa4FMeh21iENg65pORhf2JqXh6ZXI29
9pgp7LpV/7HLqKi2y5kXBAznyLasOU5IGBQOyieuCY0BTxcvauHPuQiK0J9S7YzLmM2hzTYN
s5+Q+6NkylmdiLR2xNsoWDLz/GmTYR2scbAtAm6safPgTBv3bTZa95Dnx1d15P5wNKBHuXDk
vaSaql4eX486mH2MQZQjESrCgwvHg2wib/eia09dtgcNay150x6Sb/JWbEmqnfGRQrHei+UQ
j5YQVOov58NTDhh2bQg3nqukU2dGdBXS86cX06Rsthqw2MLoqwvtpEOdTE9WKDXIIjTIeicf
RMtA+7Ig7irAp0CZCurDAq6aClAvS7B/qF1AQ5VlDR5JUPKAtBRRzFehq8fyJGmJ9qt63bfi
JWVjoJ6EUzMhjDTT2iOqeGpFo7Y6qQ4MPahuaHBQU/MR0IOCRv5yot9adWgL7dCVG6zieCGg
LrjRhbMUTHoUDades4XWbqv99XSrhLhRMyiKq722s8lpJRFCkYf+exwo4uf9+fGNGyikMOqD
6rRdxolh68vYWx3W7htvnSgoOqGa3GgUDZzDaVBGHDE13BpqxSKd07EAzJpIkedUeXLbetEO
r+rGVz39HJWaZxbcVLqsIYWNtL4rMymJHkHvMx2eQg+0X0YBzIFox4BbiH61zJtrSkjLrGQJ
dXPANwwwFbmu4ADVWel+ON6/qB5w52ATSvFUUVRYjN7jxnOYjZbEmzACB4/b7qP/u5en16ff
366278/nl9+OV9//PL++IfMF47Z5e1tnsNRJUcO7W8YfXptsjFflobWbXJY+vZpRQy/DmjTm
214xRtTIIBWjai9w3W71D382jz8Ipk55OOTMClrm4EPK7pGeuKr2qVMyOph6cOBGGzf33T6x
8j+QpNor7msHz2UyWaBaFMQ0IoKxFTIMRyyMhRoXOPbcYmqYTSTGJltHuAy4oiRlXRiz47MZ
1HAigNp/BdHH9Chg6YqxyetfDLuVShPBouoQV7rNq/BZzOaqY3AoVxYIPIFHc644rU98PSCY
4QENuw2v4ZCHFyyMjeAOcKkW8MTl7nURMhyTgE5AXnl+5/IH0PK8qTqm2XJgn9yf7YRDEtEJ
jlWVQyhrEXHsll57vjPJdHtFabvE90K3F3qam4UmlEzeA8GL3ElC0YpkVQuWa9QgSdwoCk0T
dgCWXO4KPnANAro814GDy5CZCfQuo59qXFZYctPBXseKQoYxFZ4eXOYx8DphZk1D0oalHdqx
3MWzk5tc7IdufyvQ5XEAO6b5d+aX+JlkpqmPpih+ipjkAo7Q4s5r2oIUx3yrbe9t3aqlWtAz
L6a1u3ySdpNRUrzwA+zgqIkXnn/A314cZwiAry6pLeshxzaKtCMRc+OTV1evb71dhnHPYnxQ
3d2df55fnh7Ob2Qnk6i9ohf5mIUGKHChpQPpM5bJ4fHrz6fv8Bb82/33+7evP+G+URXBzm8R
zSKcDHx32knq6OBtgkz0ghSFbGDVN1kb1beHb7/Vtx/bhR1K+s/7377dv5zvYLs9Uex2EdDk
NWCXyYDGHLDZn319/nqn8ni8O/+FpiGTof6mNVjMx75OdXnVj0lQvj++/Ti/3pP0lnFA4qvv
+SW+ifj9Xe0x756ez2q3CEIHhzdm0dhq+/Pbv55e/tCt9/4/55f/uMofns/fdOUEW6NwqQ8P
5kr//vuPNzeXVhb+vxf/HntGdcJ/gzGB88v39yvNrsDOucDJZgti7dkAcxuIbWBJgdiOogBq
ynkA0YVCc359+gk6EZ/2pi+XpDd96ZGpzCDe2LqDusPVbzCIH78pDn08D+NXPp+//vHnM2T1
CjYZXp/P57sf6GRYZ8nugD0BGAAOh+22S8S+xdOvS8Uzo0WtqwKbCrWoh7Rumynqai+nSGkm
2mL3ATU7tR9Qp8ubfpDsLrudjlh8EJHatrRo9Y46lSTU9lQ30xWxvI6bU1pnrMWOszzca4Ea
4QxfnRV5I9xjnUa/5MZPSz/VfXt5uv+GRRPbEiuBF23WbdJSHRXQCj/6UraVqdc3bXsLJ7mu
rVp4YaxN8Vw8gF/o2oCyIQfjm6my1ddze7imK1t/iXUwEUkd9vIsE0isUZCXK/ClM6mT26JS
OzhvBraqI0KXWbGmJ8TiANaQybuUHqpWqU4vrxRX9k+9/hGrFcQKZ96SZaca7MIeQQCaCaw0
ZEJp/ZkiUQ2bNQ1opF5U+WUH3hVByoHadNW1a+e7Szal50fzndrbO7RVGoG/l7lD2J7UJD9b
7XnCImXxMJjAmfBqs7b08J0XwgN8k0TwkMfnE+GxrQ2Ez+MpPHLwWqRq6nYbqEnieOEWR0bp
zE/c5BXueT6Dbz1v5uYqZer58ZLFydU9wfl0uFbTeMAUB/CQwdvFIggbFo+XRwdv8/0tkQAO
eCFjf+a25kF4kedmq2CiMDDAdaqCL5h0brS18qqlo2Bd4KdufdD1Cv72aloj8SYvhEd8bQyI
VsLnYLx1G9HtTVdVK5DTY9k6sR8GX1QUneRlJ0BfiyBqDgKf9xSU1QELsgA6zgtsBjwt1amn
tBCyLQGACO52ckEu7zZNdkteUvRAl0nfBe0nSD0Mc1SDzSAMBLU2lDcJrv9AIY9UBtDSfhxh
7ErsAlb1iphlGCiWue0Bhoe7Dui+lx/r1OTpJkvpe+WBSBUuB5S0/FiaG6ZdJNuMhM0GkD4J
GVHcp2PvNGqNucBwC3bM06yiHDg4qz+KbX49AQ8Wa0FBX208aiz1Vgm6LxX6syuoHArRZCg7
/akYoZbIDO7/+8VQ14oateeI4St9A5o301gUslUsmo02Q7GYtqnglaS++yBDcyDUarpBuvXb
G9h8WM8gkrxYVejZ5pBPV27xYb13m92VJDA8fmgSA9pJWqJrfeuV1ALcfVtXnnUqrCTyqiwP
drtv4MR0f3eliVf11+9nrRDsPp42seE+ZNNqK1nvUxRQGDgu5KcBLhutQRP2/PD0dn5+ebpj
rq8zsGPdvw8yoZ8fXr8zAetSohlCf+rOtDHddhttNqKpLzp5lbj6Vb6/vp0frqrHK/Hj/vlv
cGq6u/9dtZLzZgU0MrRTkMsl3Orl6eu3u6cH7bXb9W+twg5ar32E+/8sT3zgst2pFWXdJGKN
9uqASlET/WXtY85cPCLwVgp47K/O/wGLhhy6WHKoOpNyqMeiPovOWZQtwzJCuldgfk7gi2MT
jkDjUNo0a9oqgx36ETSPvbpaLZeVGlV7iVXWtFpMJ5uk5C7DwPkLfg0MxigvvY9CfWnxBWUJ
U+G6ya7He1rzebV5Uv39SMQaPanbVMfBM4w61GRlgld/HKjOGphZEvLajgSAFVAmxwkyvJ6Q
dTIZO5FwYBmGx1BydyCoya1vWW0Zo6/wg9sIXXaEFwLvdm4aHtLYV6J2C0SC1HWJ5lJ16hcX
xcjs3293T4+DHWSnsCZwl6gpkhpzGgi2g/kBP9U+diPVw3QD0IPqVOjNQ+zV6EIIAizrvuDW
g56eoCcqWZfmptMhN228XARuYWUZhnjJ6+HB7gtaWtTUirXl+xHTlcIZNBI2dpd5HaeSw8W2
Of++u1iHzQYDvFvna02kcP8YRG21+rQI1fyLj+EoDs1W/QvvFRsJA2QM4uMg8sY5DfTwEHyi
aIaBHz6WkK/KxMOCZvXt++RbeOHM2HjkUbqFJBSyOUwTn6gnJQE+WaWl2sfgk6IBlhaAjwFI
Rcxkh4/5uonagZCccjlBA+nTR3RVB5u+O8l0aX3SuhqINMzuJP5r582wm7JSBD59YZyoxS50
AJrQAFrviJMF8RirgHiO5esKWIahZ2kZ9agN4EKexHyGj/YKiMglmBRJQN1Atrs4IH7aFLBK
wv/zZYrxBKvYv2jRxAF3HRG9C/GXnvVNpOOL+YKGX1jxF1b8xZLI3xcxfjmvvpc+pS/xg0Oz
2UrKJEx9mLwRRU3Ms5OLxTHFYGOsH5lTWCtSUihNljDcNjVBs/0xK6oa5HltJshRsZ8hSXDQ
rSsaWGYIDDp/5ckPKbrN4zk+RW1PRE8k3yf+yapiXp4WKYWKWnixHa7XfbXAVvhz8toVAKys
CksYedsCgEcsBBokpkCAxXzg3I+IekpRBz7WfAZgjp8maTE2PAAv20itoKCaRps123dfPLtv
98lhQRRF9Lp5TIxFDvJs+bKi5iSJC34kuFZep7kZhUiTOB73I46gw36e25zUgoaDmMUeg+Hb
twGbyxmWJhrY870gdsBZLInT1yFsLMlbhB6OPBlhvQINqwSwxonB1F59ZmNxFFsFMObm7Lq2
hZiHWDp7XEfejAY75jVYawPxP8GN9a2u5wEzzz08/1QHMmtWi4NovOcUP84P2uiedK4n2yIB
g0WOM6I8uaZ9efwS4+lHbyt6IYCJK63OZ0IM5dnefxvUs+G6XTw9PDw9XgqFllyze6EMa5HZ
/Ukpx1Khi2Qp6yFfO0+9Gssa1QUytZfrMQBx7dSv5DRDnkaWU4vWN5/psac/H9+Q6sFw06wW
s69mWePXsnAWkfvYMIhm9Jve94dz36Pf88j6Jhe+Ybj0m/9t7MqaG8d9/FdJ9dNu1c60rzjJ
wzzIkmyrrSs6Eicvqkza0+2aTtIVJ/vv3k+/AKkDAKGeqUqVox8gkuIBgiQIWDtgiQpgLoAJ
L9dytih4baDEXfKz9nN2DxSeL6hGgM/LqXjmucgZd84NMi6ZPVqQZxVa0rnzBwOT5WxOiwky
/XzK54XzyxmX8YsLegyBwBULFmsNyj1HpAaOwbYVFcFgOY0D6PP709PPdseEd2nrmC+82YSp
6Hd2OS3OTiXFavclX00whn6VYwqzxggHh+fHn70txf/hYXwQlB/zOO46s//t5fFvu9n28Pby
+jE4nt5ej3++o+UIM72wd27t7b6vD6fDbzG8ePh8Fr+8fD/7L0jxv8/+6nM8kRxpKuvFfNAJ
/73FBh8nCLGbsx20lNCMD7h9US7O2UpnM106z3J1YzA2OojQ29wVGVuFJHk9n9BMWkCVRPZt
dSliSOMrFUNWFipRtZlbowwr3A8P396+kqmmQ1/fzoqHt8NZ8vJ8fONVvg4XCzY0DbBgg2o+
kcoWIrM+2/en4+fj20+lQZPZnE7gwbaiKtgWtQSqgrEIfujzrKKxN6tyRge3fRZHSxbj7VfV
9LUyumDLHXye9VUYwch4Q08gT4eH0/vr4enw/Hb2DrXmdNPFxOmTC77QjkR3i5TuFjndbZfs
l0zfvsFOtTSdim10UALrbYSgTXpxmSyDcj+Gq123oznp4Yc3zOCQokJGjZhQecEnaHa2W+DF
IOjpNXovD8or5t/KICwa/Go7vTgXz7RFfJDrU3oEjwCdT+CZOTiC5yXtKvi8pItpqmiZc0I8
lSE1u8lnXg69y5tMyAZTr62U8exqQlctnEI9PRlkSqcyujtCg3wTnBfmU+mBok7v7+XFhHlM
6rJ3XEJVBbPSBQEAMoI2RpZX0DiEJYe8ZhOOldF0uqAjr9rN53TDp/LL+YJanhuAOpjoSohW
d8zHgwEuObA4p5YGdXk+vZwR2X3jpzH/ipswiZeTC4rEy+lgdpk8fHk+vNktNqUb7y6vqEWL
eaZK025ydUU7ebuVlnibVAXVjTdD4FtD3mY+Hdk3Q+6wypIQw7eyiSvx5+czar/SjnSTvj4L
dWX6FVmZpLo22yb++SV18yAI/HMlkdgwJu/f3o7fvx1+MHXDrD3q3v1T9Pz47fg81lZ0IZP6
sK5Tqojw2P3ZpsiqLjj3L00eSYm2hXHopC+VjOPOos4rnWwV0V+8X6HIQZuDkfeNB4GBxNSw
7y9vMLUdnf3iAG+W8H2Tc2a3ZAGqdYNOPZ0LrZsNvSqPqb4giwB1R6fXOMmvWlMYq3++Hk44
FSsjbpVPlpNkQwdJPuOTMD7LgWQwZyrrBPnKKzK1F8jY9jmrpzyeUlXHPoudXYvx0ZvHc/5i
ec73qcyzSMhiPCHA5heyB8lCU1Sd6S2FpVydMw1xm88mS/Life7BLLp0AJ58B5JxbNSBZzSP
dlu2nF+ZTci2B7z8OD6hhom2GJ+PJ2uQ7rwVR4FXYMzjsLmhk0axpgptub9irgiQfNkP6cPT
d1wbqf0Nun6UNCbeRuZnNfOrSu+zh/SSRRLvryZLNqsl+YSehJhn0nIVDFw6b5pnOnOh+Q7x
CZtI11wI+XFeXkypCwyD2j7OQdwHXtMAEghuo9VNxSHjpnHOMTQCwPu6Am13TjlqPCTSnV4E
eaBhg7QXl6u85gThfqCFcmoCUlzj6TuxYSmSZoNBo719kxZDXM1PuLRvPOrorSpBfZ807N5t
eJ/mJSZAssgxLB+zTuvDd2V+RW2NYSSFVRcNI6Yng5biVVtq5GDBVVjAdCTRTZhEaSRR3HeX
WLtxIuEkLDMn1TwqKw8qLJOEMvPRzNeBTQNI0HjwEGAVGesIundoCXUa5duIzlQWv9/PrpbO
x6CTFXZzNcGDZkPaRnN2HiaIS3vIOTjmscVCXxrNKk9yxaxiTX1QwkOz9nYhM71CEKbTG25R
DuBtgTInRLughFMG8y0rybZ3Z+X7nydj2DMIltZtCg+/gqFSum0wPC1n8UuQKJx3IGSa+dLG
s1EozWYfKzT/bpOiVaAfCUu9XZZ6hp9bHOI7SE5LJbGBMOeEtJyJLDrUXh4MRDoFOszw6MEg
wrZpua1h6xnm4hxhH43Y0Y+jrMt87zWzyzQxwXt4oj1JqRtzJseyQ9gc0Vy77AavTYSgUYLM
3ThvgLacK5XWGxS5NdeThNtypLWngEFubSNVYhKZGDdjZJMhq8DOIKP96n5sDS8tTOwXIKvO
sQjffjr7N3zns3M3PVqiyp6Jgfo5we+RnXSgL0bo0XYxueDNa1x9t1Lb7fcYK7i99tShaNiE
HnAGyzhqZQIPOIyJwPN6Fwnu5ZY0KDJqxdUCzSpKA+jUaF45RuvcL3z484ieU//n63/sPx/G
02rms1VEbaA8MnF03kHpI045oCkSITfAoBJVuSR0AkzKRk5VXsQza5Ei6hrhmsU1s8NwzdPu
B4ZgtgmjfFKLavf3BamkehE8uFeejE184Q9eYDWa4mbXutKhoS86pNmoaKmiMCQVNKdRG3qU
OUjCCR6vPf51/PIOCi9eHXWMbY0S8ESf0CEdCxRowGQD3ckPF2JV1dM6fWKU0nh0RPXU9shV
TxSVA62E1kiezA7WjjnHHi+OdBySCPXU5p/j4LIKf7+OXpcR8WTS5U9DRcADhgCpHF/IhMAO
MxEvWeTGKuyPJeFfxTAXr+hDqfZDucjmiMaPR+Kbi6sZ9dYEoLDEA6T1FGC/9YhXNo3OcqIf
ixbVVO6F+2rGbqe1QLP3KnofpYMxPAkUyI9dUhn6dcE8LgNlLhOfj6cyH01lIVNZjKey+EUq
YWouGERUr+5eGaUJZzSfVgFRIPBJcmDcopXvsUsURRjBmgND9JQKKO709bixN4rSdabQ3Dai
JKVuKNmtn0+ibJ/0RD6NviyrCRlx7w0DM5CF7l7kg8/XdVZ5nEXJGmEa2W/vZoqQV8JXVrAS
qGiomc265P28BRq8foEXYIOYTJwg4QR7hzTZjGoKPdzbQjetMqvwYHWUMhN7vRNkyw6vJ6lE
uk+wqmQn6hCtynqa6WBmVtnwlus5ijoFbTEForko4mQpatqCtq6JwhHFsuLWM1FeA2BVsO9q
2WSX7mDl2zqS2xsNxX6xloU20C3NuI2K0k+hL6glV7PGZA/ejqE5dkgbqifLaWmiOOw6H1FD
QeHDe2R3I3RefDINpVkVrUkdBBKILGBDHgzpeZKvQ1ov+mgDnURlGWUpKbwYsOYRLx+aKzFm
ixzdZ5ClCAaHatluvSJl32Rh0b8sWLErYNfrpGpuphKgdnv4ll+RRvHqKluXfP5AFZMBPtM5
s5uwiL07Pvx7DMRoEBXQQxr4IeN1YEBVfd+pbP7D49cDm4HFxNACUjh08BbkZ7YpvMQlObOO
hbMV9t8mjkoyMA3Jxit9cjHH39pAofnbDwp+A839Y3ATGB3DUTGiMrtaLid8LsniiIb4uwcm
FpYvEJEb4TmN+yVXkJUfQZx/TCs9y7WVJYMuVMIbDLmRLPjc+YnzsyBEH3Z/LOYXGj3KcEcI
wx5+OJ5eLi/Pr36bftAY62pNfIKmlRB8BhA1bbDitvvS/HR4//xy9pf2lUYXYFvGCOyMdswx
3MajY8CA+IVNkoGApy48DQkWXnFQhETg7cIiXfPrYPSxSnLnUZOIltCJ9MF1YL0BUbFqRhwH
2h9RecZTn+mSdzC70luXWYHOHQW7F+iAresOWwum0EhWHWo9RDLJtRXvw3Me12OYOkHLghtA
zrWymI6qJifdDmlTmji42f2Ud24GKrpOBIHGJgZLLWEt7RUO7M7cPa4qkZ1GpGiSSMI4iHio
BbNOG+a+lCz3LFiLxeL7TELmLNcB65XZr+97ZJsrerpq0izVeiVlyTEGui22mgS6nFS3ySjT
2rvJ6gKKrMVMXEWijTsEOvINXtgLbB0RGdoxsEroUV5dFvawbsitXvmOpij1RLfpfJglaJHL
69ortxpilRs7EdIblIxsZ1ntLmXHhgv8JIfaTjexnlDLYdbcaoOonKjzoGv4X2QtOnuP82ru
4fh+oaKZgu7vFXCB8fduVvHO9C2FIUxWYRCEgUJaF94mwduPraKBCcz7mVGux/DQas91n0SK
uVwA1+l+4UJLHZJBjZzkLYI+GfAu310bFpAGhxAMSRXokR1kQlm11cI7GDaQNF1G3dyIcWDp
Br55Nk3cCyharJYOrdqT9b3yjm+h8nEuX8apanFzVZ3OzzdcXEjxYQetEftkMLvNEe4zOdsY
RLCximm9jujTcyq1IHimKrt5nstnPl8YbMF5ylu6jWU5mqmDkLPPPO1ECyjqzH2WoYh4lJY7
DvfqG11+jbHuxtFlrIyaKGivhv/x4e/D6/Ph2+8vr18+OG8lEajUXLC2tE6souvIMJbV2IlM
AuJaxkZohjWfqHepbK7LgH1CAC3h1HSAzSEBjWshgJypjAYyddrWHaeUfhmphK7KVeKvKygY
X8FDdaO3R1BpMlIFWDr5KL8Lv7yfI1n7tzdthkFYpwVz9Waemw215WkxlEltQAL5vujYgMAX
YyLNrlidOynJlVuYb/nS1gKi47Soppv5EXs9cvetBmwmwNvQ2zX5bbPF8KmcVOe+F4ts5Cxq
MFMkgTkFdD67x2SR7A4a+p5BT3zyK4KxkpXJCo2bOeiOPz/nss03CyGccCq8J8v3OSzVuntz
NnYssayKzEWxs7GhbdAMNEkXLRP4mCBz8DR2oHBfFdTnB6yDPb5mkmsot+I9rVqueK2YR41F
636W4CqXvPxx2S26tTU5krtFfbOgBnaMcjFOoTbAjHJJjdUFZTZKGU9trAQsgq2gTEcpoyWg
ZteCshiljJaaXtQWlKsRytV87J2r0Rq9mo99z9ViLJ/LC/E9UZlh76DOyNkL09lo/kASVW3C
VejpT3V4psNzHR4p+7kOL3X4QoevRso9UpTpSFmmojC7LLpsCgWrOYbBVEDR9lIX9kNYc/ka
nlZhTQ17e0qRgfakpnVXRHGspbbxQh0vQmr02MERlIo5zukJaR1VI9+mFqmqix0LOI8Es1XY
I3j2RB/4qf/OKJJnXx8e/z4+fyH+oIyq3cU86W/sfH89Pr/9bc1unw6nL25MF7Mbv2v4zohv
FxroJC8Ob8K4F7D9nmgbIsXl6D2xmkgqbepByOLBBHepl0Q+/zL/5en78dvht7fj0+Hs8evh
8e+TKfejxV/dordBoPAMAZKCtZPvVXTR29KTuqzkoSosgxP75h/TyawvM0y5UY6B3qA+6bql
CL3Aehwryd57nYK+HSDrKqMzkhEY2W1Kt9Pcw7stpIl+ZUTJLGNpdVbc6Uw8FtpKUuznZ2l8
J78uz8zhi1OGDA1arHYmA0AnHhrbQl8qrlWw3922VfvH5MdU42qd9oqMcR/ZqLjWzuHw9PL6
8yw4/Pn+5Qvryqb6QB9BF7dUpbapIBXj3vijhK7dux7J2wVqpcy4LsbxJs3as89RjvuwyGT2
9qSlHIGpjzqVvsbDrRGauUEymrLx8TlCK/za9LMxut3bguFeaz2l4xL12Td5GderjpUufRAW
y4S2V1doe13zuEuWdJO4CPx5QlXsScVKAfPNOvY2TrbWMRUI68ip/nIbFYM7NeyLZ3hR+v27
lT3bh+cv9H4CKOp1PjhtGT45W1ejRBSEGFogoWwiSvk4T3PjxXU4VLpNv9miIWfllazp7djs
Sabj4Lp5Opu4GQ1so2URLLIot9foN9ffBhkbTMiJW/XshJvBMiFL7Erbl7WEpg/kIsOC3AzG
YKLHWT7b48I00MUsZrkLw9yKA3upBS/Y91Lp7L9O34/PeOn+9D9nT+9vhx8H+Ofw9vj777+T
6PU2NXRtWlfhPnQ7fufaU/ZMnd2rMpweyxiKJmmdiYqXR71QIQkY2wLofqBohMLJ5+2tzU9x
TG6qqSrYMb2ZV0CgwjRXhmEAlVmADpQ543RnxcQIDFNnHLLYd+1nRK4shC/SYLq3aRFjlBAp
MtEvoKApaIPDeTGIQDbJDDuvRXYTooRU9lv1CkRpim4SFXj8BVGrCIXXbhhMU3wYUHZ2LsS8
bMnWcARmRtz+p7thbXWgt3dzIbLb8Rp27dfQDr/iZvu36HD3H7jGrWO8KC5jb8URO4GKadsQ
Em8Hnxte12xuNCRzQdKKBPFO4o+8ssb+TTFWSkUJkxzDQMDNYTYnoj/91L+rMjIkLaffDm3b
396fjQZbHU5vTK2Jd0HFTENLa5cBEp9uJhqcQ6u+TDhyZR9boV2MAI3iB0K2UWjtnM9BK3GW
C0U22FCSGB5yKV4yRd2Ge+PMWXwAKNIp6rhxziJKGeIOqBW92W5Qs1RYC3AVVSxCqgHrmvr5
NlCBu4zWn7AonkdXWzYjvG6SypbYybZB2yjQjPI7WaRcFtJ1SG7rwFpeiFTtUkjWFqxgfbsn
ObS6UauawKs8NG3Ga8ZspNmqTMwW9nCEiZGpQv2oCT15g1RPQaOrV6DhoqKb1nGsHjkDnSx0
DLsXR5s0YT5X23RqumUaQlfdN7tN4GhoaRIJrQBV7XBjB2GFtmVpdxBinU4dHt9f8dqos/Tj
27fYx2Ak4bEmELDnUYOUAk0wA1G77XF2h/8kCTfBtskgSU+YGvSHDQEsfs2NNOjkdOJzNyr7
V/CszSjR2yzbKWmutXw6B/6jlGa/LhKFnHsVWTHGsJxN8E4MVG/jBUHxx/L8fN7HKtl6MA2a
W24p1Ab2euz0Vlh7TJF1mH5BMhK/zGkvans5cqDNg5UN/0C2n/Lh4+nP4/PH99Ph9enl8+G3
r4dv38m9mP67QXhFab1XaqSlDLrtv+GRaqrDGUQld2HucoTG0eMvOLwbXy6WHB6ju8KMhy7j
20JNXObE87W+YnC8VJFuarUghg49Sk54gsPLc9Sj8STDi7XSwhSS3WWjBHOHF+1Ic9w+qIo7
Fq5VZa6DqDKBGtgejeCEiasiFtcYjkf9Cig/CP7sV6R/0fQ9Kz/W0unuFoTLJ5c3OkNrXK1V
u2BsN+Y0TqyanN5NlpR2pR8oHHceDSSt2I73kO0hqA1rRNAmkiREwSkE78BCBHbB9mlIKtgz
CIGVDWN6w5ID1fHcB9U12EP/oVQUiEUdh8xgAwl48R/VPGVGRDIuhlsO+WYZbf7p7W553ifx
4fj08NvzYBBAmUzvKbfeVGYkGWbny3/Iz3TUD6evD1OWk73ynGdx5N/xysPNTpUAPQ3UQLqW
o6gmW02ljjYnELvp2lqW23PS1j6nBnEEXRI6dolLl4AZGuK7qxjEktGg1aSxTzf788kVhxHp
ZhVYx3/8+/Dz9PEHgtAcv9Prluzj2oLxPaSQ7lrBQ4NH2LC+MAoqI5jj1VaQmoPuktOVwiI8
XtjD/z6xwnatrcyFff9xebA8qq7osFph++94O4n077gDz1d6sGSDHnz4dnx+/9F/8R7lNa68
SrlWEXcEDZaEiU9VeYvuqd9WC+XX+tIH1883klT1OgC8h3MGOkEfmtBhwjI7XDbMTafz+q8/
v7+9nD2+vB7OXl7PrKozKL5tTBwv3nh5JNNo4ZmLhyzK8AC6rKt450f5lk6hkuK+JGw8BtBl
Leg4HTCVsZ8/naKPlsQbK/0uz11uAN0U0A5PKU7pNBksFBwo9IOtU1xYbnobpUwt7mZm7uWM
pNJ3JrHz13Jt1tPZZVLHzutmlaaBbva4triuwzp0KObH7UrJCO7V1RZWWg7O9xG6qks3Udrf
l/Xe376id6nHh7fD57Pw+RHHBV5y/s/x7euZdzq9PB4NKXh4e3DGh+8nTvobBfO3HvzNJjDd
3fEAeS1DGV5HN25R4SWYCnq3ICvjlhXXJie3KCvfrcbKbV48PHLzWTlYXNw6WI6ZSHCvJAgz
5W1hNlSs58+H09exYieem+QWQfkxey3zm2TwsxscvxxOb24OhT+fuW8aWEOr6SSI1m6H51s8
XY2MNWgSLBTs3B2bEbRxGOOvw18kGGVRhZlLmx6enS81mMWr7DqcVfocEJNQ4POpW1cAz90h
tylYQOpuqOc2BTv3HL9/ZffN+5nClTOANdRJQQen9Spy+51X+G61w+x9u46UxusIjv/yrjN4
SRjHkacQ8Kh/7KWycrsDom7bBKH7CWvz646orXevTK4lrJE9pXk7gaMImlBJJSxyGyNGyk/3
26vbTK3MFh+qpbe2QL98zG90//Vrs0BxJA+9otBilwu3T+EFBwXbDiHXHp4/vzydpe9Pfx5e
O2/WWkm8tIwaP0edwWmiYmUiONQ6RZVUlqLpKobiV+4UjQQnh08RRgvFTQy2PUwmbzyeGyU0
qsTqqWWnwoxyaPXRE1VdzywX+aFpR7mlSwQSchG9yvmel/RtAWnDuNCUdfJW63xIbTEgl+e5
insVjOhRdYFwKANzoFbauB3IIBdV6rXv9nXEo2RThb7eWkh3veQR4k1UVNQnB9/4MD6f2EKh
I+b1Km55ynrF2cxy0A8LPJpDWyc8ymA3uPOdX170tlk61Z4bhNQHjl3b5qG9EGHu+mH6ZCfe
R6/afxll7HT2F6xLTscvz9bLojHVYiddSRbUsVkym3w+PMLLp4/4BrA1sIb9/fvhadi9tZZr
o9sELr3844N8266vSdU47zsc9u7TYnLV74T3+wz/WJhfbD04HGbomXNtKHU/0FZRihnZQy86
pFrPmn++Prz+PHt9eX87PlPtzC5G6SJ1FVVFCG1Wsk0ps6tvTnwGunbdybQy843RurLDCOZ1
FdGd3t7LnR9JzzEdaRSm+/xJ3gVMI8IIlu1+VLGpzp+ymRlW0o4GCElXdcPfmrMFDTwqh5ot
DiMtXN1dcglIKAt1C6Nl8YpbsdsnOKDmVWHJVSGf2OjG0crVin0aKstsibfVSottCabBcf3q
9Uxqo6MdB62Xvr5gOh+usz1R1N6J5Li5/QazSswGnEE7HWI4hSI34ThKUib4QimHUSJ0XE1l
f4+wfG72l0sHM07gcpc38pYLB/ToqduAVds6WTmEEsSwm+7K/+RgvIsOH9Rs7iNmB9QTVkCY
qZT4nm4xEQK9Ucr4sxF84Q5n5WywCNHaKouzhPsMHVBM9VJ/AUlT0iYrn0y4K9OlU3tw71Gr
3Apkehlin9ewZsetEnp8lajwuiS4MargJw+9PQWdtsvMBxUgMrK28NiBqPGuRR3wIWQ3/obN
UDxVQP/fWa6f2SMDahKSoSNfU8kdZyv+pAi+NObXrPombQ1AyBAs6ka4J/Hj+6aitj1+VgR0
fYyHy0OlFde4DCclTPKIX4l2T4qAvg6IQEJ/h+iyrqzo9v86Syv3Hh6ipWC6/HHpILSrGWj5
g17vMtDFj+lCQOiIMlYS9KAWUgXHy9PN4oeS2URA08mPqXy7rFOlpIBOZz9mNDAyWhfG9FSi
RJeWWcxmB+zg2P9sDOso1a6zo+lTEOYZTaq1vxlUSGE7A/pLEjYpyD1r5vP/4guHHM+lAgA=
--n8g4imXOkfNTN/H1--
^ permalink raw reply [flat|nested] 79+ messages in thread
* Re: [PATCH v7 08/11] mm: zero reserved and unavailable struct pages
@ 2017-08-30 23:12 ` kbuild test robot
0 siblings, 0 replies; 79+ messages in thread
From: kbuild test robot @ 2017-08-30 23:12 UTC (permalink / raw)
To: sparclinux
[-- Attachment #1: Type: text/plain, Size: 1768 bytes --]
Hi Pavel,
[auto build test ERROR on sparc/master]
[also build test ERROR on v4.13-rc7 next-20170829]
[cannot apply to mmotm/master]
[if your patch is applied to the wrong git tree, please drop us a note to help improve the system]
url: https://github.com/0day-ci/linux/commits/Pavel-Tatashin/complete-deferred-page-initialization/20170831-041021
base: https://git.kernel.org/pub/scm/linux/kernel/git/davem/sparc.git master
config: cris-allmodconfig (attached as .config)
compiler: cris-linux-gcc (GCC) 6.2.0
reproduce:
wget https://raw.githubusercontent.com/01org/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross
chmod +x ~/bin/make.cross
# save the attached .config to linux build tree
make.cross ARCH=cris
All errors (new ones prefixed by >>):
In file included from mm/page_alloc.c:18:0:
include/linux/mm.h:1974:33: error: expected '=', ',', ';', 'asm' or '__attribute__' before 'zero_resv_unavail'
static inline void __paginginit zero_resv_unavail(void) {}
^~~~~~~~~~~~~~~~~
mm/page_alloc.c: In function 'free_area_init':
>> mm/page_alloc.c:6863:2: error: implicit declaration of function 'zero_resv_unavail' [-Werror=implicit-function-declaration]
zero_resv_unavail();
^~~~~~~~~~~~~~~~~
cc1: some warnings being treated as errors
vim +/zero_resv_unavail +6863 mm/page_alloc.c
6858
6859 void __init free_area_init(unsigned long *zones_size)
6860 {
6861 free_area_init_node(0, zones_size,
6862 __pa(PAGE_OFFSET) >> PAGE_SHIFT, NULL);
> 6863 zero_resv_unavail();
6864 }
6865
---
0-DAY kernel test infrastructure Open Source Technology Center
https://lists.01.org/pipermail/kbuild-all Intel Corporation
[-- Attachment #2: .config.gz --]
[-- Type: application/gzip, Size: 42983 bytes --]
^ permalink raw reply [flat|nested] 79+ messages in thread
* [PATCH v7 08/11] mm: zero reserved and unavailable struct pages
@ 2017-08-30 23:12 ` kbuild test robot
0 siblings, 0 replies; 79+ messages in thread
From: kbuild test robot @ 2017-08-30 23:12 UTC (permalink / raw)
To: linux-arm-kernel
Hi Pavel,
[auto build test ERROR on sparc/master]
[also build test ERROR on v4.13-rc7 next-20170829]
[cannot apply to mmotm/master]
[if your patch is applied to the wrong git tree, please drop us a note to help improve the system]
url: https://github.com/0day-ci/linux/commits/Pavel-Tatashin/complete-deferred-page-initialization/20170831-041021
base: https://git.kernel.org/pub/scm/linux/kernel/git/davem/sparc.git master
config: cris-allmodconfig (attached as .config)
compiler: cris-linux-gcc (GCC) 6.2.0
reproduce:
wget https://raw.githubusercontent.com/01org/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross
chmod +x ~/bin/make.cross
# save the attached .config to linux build tree
make.cross ARCH=cris
All errors (new ones prefixed by >>):
In file included from mm/page_alloc.c:18:0:
include/linux/mm.h:1974:33: error: expected '=', ',', ';', 'asm' or '__attribute__' before 'zero_resv_unavail'
static inline void __paginginit zero_resv_unavail(void) {}
^~~~~~~~~~~~~~~~~
mm/page_alloc.c: In function 'free_area_init':
>> mm/page_alloc.c:6863:2: error: implicit declaration of function 'zero_resv_unavail' [-Werror=implicit-function-declaration]
zero_resv_unavail();
^~~~~~~~~~~~~~~~~
cc1: some warnings being treated as errors
vim +/zero_resv_unavail +6863 mm/page_alloc.c
6858
6859 void __init free_area_init(unsigned long *zones_size)
6860 {
6861 free_area_init_node(0, zones_size,
6862 __pa(PAGE_OFFSET) >> PAGE_SHIFT, NULL);
> 6863 zero_resv_unavail();
6864 }
6865
---
0-DAY kernel test infrastructure Open Source Technology Center
https://lists.01.org/pipermail/kbuild-all Intel Corporation
-------------- next part --------------
A non-text attachment was scrubbed...
Name: .config.gz
Type: application/gzip
Size: 42983 bytes
Desc: not available
URL: <http://lists.infradead.org/pipermail/linux-arm-kernel/attachments/20170831/cef71a4d/attachment-0001.gz>
^ permalink raw reply [flat|nested] 79+ messages in thread
* [PATCH v7 00/11] complete deferred page initialization
@ 2017-09-14 22:35 ` Pavel Tatashin
0 siblings, 0 replies; 79+ messages in thread
From: Pavel Tatashin @ 2017-09-14 22:35 UTC (permalink / raw)
To: linux-kernel, sparclinux, linux-mm, linuxppc-dev, linux-s390,
linux-arm-kernel, x86, kasan-dev, borntraeger, heiko.carstens,
davem, willy, mhocko, ard.biesheuvel, will.deacon,
catalin.marinas, sam, mgorman, Steven.Sistare, daniel.m.jordan,
bob.picco
Changelog:
v8 - v7
- Added Acked-by's from Dave Miller for SPARC changes
- Fixed a minor compiling issue on tile architecture reported by kbuild
v7 - v6
- Addressed comments from Michal Hocko
- memblock_discard() patch was removed from this series and integrated
separately
- Fixed bug reported by kbuild test robot new patch:
mm: zero reserved and unavailable struct pages
- Removed patch
x86/mm: reserve only exiting low pages
As, it is not needed anymore, because of the previous fix
- Re-wrote deferred_init_memmap(), found and fixed an existing bug, where
page variable is not reset when zone holes present.
- Merged several patches together per Michal request
- Added performance data including raw logs
v6 - v5
- Fixed ARM64 + kasan code, as reported by Ard Biesheuvel
- Tested ARM64 code in qemu and found few more issues, that I fixed in this
iteration
- Added page roundup/rounddown to x86 and arm zeroing routines to zero the
whole allocated range, instead of only provided address range.
- Addressed SPARC related comment from Sam Ravnborg
- Fixed section mismatch warnings related to memblock_discard().
v5 - v4
- Fixed build issues reported by kbuild on various configurations
v4 - v3
- Rewrote code to zero sturct pages in __init_single_page() as
suggested by Michal Hocko
- Added code to handle issues related to accessing struct page
memory before they are initialized.
v3 - v2
- Addressed David Miller comments about one change per patch:
* Splited changes to platforms into 4 patches
* Made "do not zero vmemmap_buf" as a separate patch
v2 - v1
- Per request, added s390 to deferred "struct page" zeroing
- Collected performance data on x86 which proofs the importance to
keep memset() as prefetch (see below).
SMP machines can benefit from the DEFERRED_STRUCT_PAGE_INIT config option,
which defers initializing struct pages until all cpus have been started so
it can be done in parallel.
However, this feature is sub-optimal, because the deferred page
initialization code expects that the struct pages have already been zeroed,
and the zeroing is done early in boot with a single thread only. Also, we
access that memory and set flags before struct pages are initialized. All
of this is fixed in this patchset.
In this work we do the following:
- Never read access struct page until it was initialized
- Never set any fields in struct pages before they are initialized
- Zero struct page at the beginning of struct page initialization
==========================================================================
Performance improvements on x86 machine with 8 nodes:
Intel(R) Xeon(R) CPU E7-8895 v3 @ 2.60GHz and 1T of memory:
TIME SPEED UP
base no deferred: 95.796233s
fix no deferred: 79.978956s 19.77%
base deferred: 77.254713s
fix deferred: 55.050509s 40.34%
==========================================================================
SPARC M6 3600 MHz with 15T of memory
TIME SPEED UP
base no deferred: 358.335727s
fix no deferred: 302.320936s 18.52%
base deferred: 237.534603s
fix deferred: 182.103003s 30.44%
==========================================================================
Raw dmesg output with timestamps:
x86 base no deferred: https://hastebin.com/ofunepurit.scala
x86 base deferred: https://hastebin.com/ifazegeyas.scala
x86 fix no deferred: https://hastebin.com/pegocohevo.scala
x86 fix deferred: https://hastebin.com/ofupevikuk.scala
sparc base no deferred: https://hastebin.com/ibobeteken.go
sparc base deferred: https://hastebin.com/fariqimiyu.go
sparc fix no deferred: https://hastebin.com/muhegoheyi.go
sparc fix deferred: https://hastebin.com/xadinobutu.go
Pavel Tatashin (11):
x86/mm: setting fields in deferred pages
sparc64/mm: setting fields in deferred pages
mm: deferred_init_memmap improvements
sparc64: simplify vmemmap_populate
mm: defining memblock_virt_alloc_try_nid_raw
mm: zero struct pages during initialization
sparc64: optimized struct page zeroing
mm: zero reserved and unavailable struct pages
x86/kasan: explicitly zero kasan shadow memory
arm64/kasan: explicitly zero kasan shadow memory
mm: stop zeroing memory during allocation in vmemmap
arch/arm64/mm/kasan_init.c | 42 ++++++++
arch/sparc/include/asm/pgtable_64.h | 30 ++++++
arch/sparc/mm/init_64.c | 31 +++---
arch/x86/mm/init_64.c | 9 +-
arch/x86/mm/kasan_init_64.c | 66 ++++++++++++
include/linux/bootmem.h | 27 +++++
include/linux/memblock.h | 16 +++
include/linux/mm.h | 26 +++++
mm/memblock.c | 60 +++++++++--
mm/page_alloc.c | 207 ++++++++++++++++++++----------------
mm/sparse-vmemmap.c | 15 ++-
mm/sparse.c | 6 +-
12 files changed, 406 insertions(+), 129 deletions(-)
--
2.14.1
^ permalink raw reply [flat|nested] 79+ messages in thread
* [PATCH v7 00/11] complete deferred page initialization
@ 2017-09-14 22:35 ` Pavel Tatashin
0 siblings, 0 replies; 79+ messages in thread
From: Pavel Tatashin @ 2017-09-14 22:35 UTC (permalink / raw)
To: linux-arm-kernel
Changelog:
v8 - v7
- Added Acked-by's from Dave Miller for SPARC changes
- Fixed a minor compiling issue on tile architecture reported by kbuild
v7 - v6
- Addressed comments from Michal Hocko
- memblock_discard() patch was removed from this series and integrated
separately
- Fixed bug reported by kbuild test robot new patch:
mm: zero reserved and unavailable struct pages
- Removed patch
x86/mm: reserve only exiting low pages
As, it is not needed anymore, because of the previous fix
- Re-wrote deferred_init_memmap(), found and fixed an existing bug, where
page variable is not reset when zone holes present.
- Merged several patches together per Michal request
- Added performance data including raw logs
v6 - v5
- Fixed ARM64 + kasan code, as reported by Ard Biesheuvel
- Tested ARM64 code in qemu and found few more issues, that I fixed in this
iteration
- Added page roundup/rounddown to x86 and arm zeroing routines to zero the
whole allocated range, instead of only provided address range.
- Addressed SPARC related comment from Sam Ravnborg
- Fixed section mismatch warnings related to memblock_discard().
v5 - v4
- Fixed build issues reported by kbuild on various configurations
v4 - v3
- Rewrote code to zero sturct pages in __init_single_page() as
suggested by Michal Hocko
- Added code to handle issues related to accessing struct page
memory before they are initialized.
v3 - v2
- Addressed David Miller comments about one change per patch:
* Splited changes to platforms into 4 patches
* Made "do not zero vmemmap_buf" as a separate patch
v2 - v1
- Per request, added s390 to deferred "struct page" zeroing
- Collected performance data on x86 which proofs the importance to
keep memset() as prefetch (see below).
SMP machines can benefit from the DEFERRED_STRUCT_PAGE_INIT config option,
which defers initializing struct pages until all cpus have been started so
it can be done in parallel.
However, this feature is sub-optimal, because the deferred page
initialization code expects that the struct pages have already been zeroed,
and the zeroing is done early in boot with a single thread only. Also, we
access that memory and set flags before struct pages are initialized. All
of this is fixed in this patchset.
In this work we do the following:
- Never read access struct page until it was initialized
- Never set any fields in struct pages before they are initialized
- Zero struct page at the beginning of struct page initialization
=====================================
Performance improvements on x86 machine with 8 nodes:
Intel(R) Xeon(R) CPU E7-8895 v3 @ 2.60GHz and 1T of memory:
TIME SPEED UP
base no deferred: 95.796233s
fix no deferred: 79.978956s 19.77%
base deferred: 77.254713s
fix deferred: 55.050509s 40.34%
=====================================
SPARC M6 3600 MHz with 15T of memory
TIME SPEED UP
base no deferred: 358.335727s
fix no deferred: 302.320936s 18.52%
base deferred: 237.534603s
fix deferred: 182.103003s 30.44%
=====================================
Raw dmesg output with timestamps:
x86 base no deferred: https://hastebin.com/ofunepurit.scala
x86 base deferred: https://hastebin.com/ifazegeyas.scala
x86 fix no deferred: https://hastebin.com/pegocohevo.scala
x86 fix deferred: https://hastebin.com/ofupevikuk.scala
sparc base no deferred: https://hastebin.com/ibobeteken.go
sparc base deferred: https://hastebin.com/fariqimiyu.go
sparc fix no deferred: https://hastebin.com/muhegoheyi.go
sparc fix deferred: https://hastebin.com/xadinobutu.go
Pavel Tatashin (11):
x86/mm: setting fields in deferred pages
sparc64/mm: setting fields in deferred pages
mm: deferred_init_memmap improvements
sparc64: simplify vmemmap_populate
mm: defining memblock_virt_alloc_try_nid_raw
mm: zero struct pages during initialization
sparc64: optimized struct page zeroing
mm: zero reserved and unavailable struct pages
x86/kasan: explicitly zero kasan shadow memory
arm64/kasan: explicitly zero kasan shadow memory
mm: stop zeroing memory during allocation in vmemmap
arch/arm64/mm/kasan_init.c | 42 ++++++++
arch/sparc/include/asm/pgtable_64.h | 30 ++++++
arch/sparc/mm/init_64.c | 31 +++---
arch/x86/mm/init_64.c | 9 +-
arch/x86/mm/kasan_init_64.c | 66 ++++++++++++
include/linux/bootmem.h | 27 +++++
include/linux/memblock.h | 16 +++
include/linux/mm.h | 26 +++++
mm/memblock.c | 60 +++++++++--
mm/page_alloc.c | 207 ++++++++++++++++++++----------------
mm/sparse-vmemmap.c | 15 ++-
mm/sparse.c | 6 +-
12 files changed, 406 insertions(+), 129 deletions(-)
--
2.14.1
^ permalink raw reply [flat|nested] 79+ messages in thread
* [PATCH v7 00/11] complete deferred page initialization
@ 2017-09-14 22:35 ` Pavel Tatashin
0 siblings, 0 replies; 79+ messages in thread
From: Pavel Tatashin @ 2017-09-14 22:35 UTC (permalink / raw)
To: linux-kernel, sparclinux, linux-mm, linuxppc-dev, linux-s390,
linux-arm-kernel, x86, kasan-dev, borntraeger, heiko.carstens,
davem, willy, mhocko, ard.biesheuvel, will.deacon,
catalin.marinas, sam, mgorman, Steven.Sistare, daniel.m.jordan,
bob.picco
Changelog:
v8 - v7
- Added Acked-by's from Dave Miller for SPARC changes
- Fixed a minor compiling issue on tile architecture reported by kbuild
v7 - v6
- Addressed comments from Michal Hocko
- memblock_discard() patch was removed from this series and integrated
separately
- Fixed bug reported by kbuild test robot new patch:
mm: zero reserved and unavailable struct pages
- Removed patch
x86/mm: reserve only exiting low pages
As, it is not needed anymore, because of the previous fix
- Re-wrote deferred_init_memmap(), found and fixed an existing bug, where
page variable is not reset when zone holes present.
- Merged several patches together per Michal request
- Added performance data including raw logs
v6 - v5
- Fixed ARM64 + kasan code, as reported by Ard Biesheuvel
- Tested ARM64 code in qemu and found few more issues, that I fixed in this
iteration
- Added page roundup/rounddown to x86 and arm zeroing routines to zero the
whole allocated range, instead of only provided address range.
- Addressed SPARC related comment from Sam Ravnborg
- Fixed section mismatch warnings related to memblock_discard().
v5 - v4
- Fixed build issues reported by kbuild on various configurations
v4 - v3
- Rewrote code to zero sturct pages in __init_single_page() as
suggested by Michal Hocko
- Added code to handle issues related to accessing struct page
memory before they are initialized.
v3 - v2
- Addressed David Miller comments about one change per patch:
* Splited changes to platforms into 4 patches
* Made "do not zero vmemmap_buf" as a separate patch
v2 - v1
- Per request, added s390 to deferred "struct page" zeroing
- Collected performance data on x86 which proofs the importance to
keep memset() as prefetch (see below).
SMP machines can benefit from the DEFERRED_STRUCT_PAGE_INIT config option,
which defers initializing struct pages until all cpus have been started so
it can be done in parallel.
However, this feature is sub-optimal, because the deferred page
initialization code expects that the struct pages have already been zeroed,
and the zeroing is done early in boot with a single thread only. Also, we
access that memory and set flags before struct pages are initialized. All
of this is fixed in this patchset.
In this work we do the following:
- Never read access struct page until it was initialized
- Never set any fields in struct pages before they are initialized
- Zero struct page at the beginning of struct page initialization
==========================================================================
Performance improvements on x86 machine with 8 nodes:
Intel(R) Xeon(R) CPU E7-8895 v3 @ 2.60GHz and 1T of memory:
TIME SPEED UP
base no deferred: 95.796233s
fix no deferred: 79.978956s 19.77%
base deferred: 77.254713s
fix deferred: 55.050509s 40.34%
==========================================================================
SPARC M6 3600 MHz with 15T of memory
TIME SPEED UP
base no deferred: 358.335727s
fix no deferred: 302.320936s 18.52%
base deferred: 237.534603s
fix deferred: 182.103003s 30.44%
==========================================================================
Raw dmesg output with timestamps:
x86 base no deferred: https://hastebin.com/ofunepurit.scala
x86 base deferred: https://hastebin.com/ifazegeyas.scala
x86 fix no deferred: https://hastebin.com/pegocohevo.scala
x86 fix deferred: https://hastebin.com/ofupevikuk.scala
sparc base no deferred: https://hastebin.com/ibobeteken.go
sparc base deferred: https://hastebin.com/fariqimiyu.go
sparc fix no deferred: https://hastebin.com/muhegoheyi.go
sparc fix deferred: https://hastebin.com/xadinobutu.go
Pavel Tatashin (11):
x86/mm: setting fields in deferred pages
sparc64/mm: setting fields in deferred pages
mm: deferred_init_memmap improvements
sparc64: simplify vmemmap_populate
mm: defining memblock_virt_alloc_try_nid_raw
mm: zero struct pages during initialization
sparc64: optimized struct page zeroing
mm: zero reserved and unavailable struct pages
x86/kasan: explicitly zero kasan shadow memory
arm64/kasan: explicitly zero kasan shadow memory
mm: stop zeroing memory during allocation in vmemmap
arch/arm64/mm/kasan_init.c | 42 ++++++++
arch/sparc/include/asm/pgtable_64.h | 30 ++++++
arch/sparc/mm/init_64.c | 31 +++---
arch/x86/mm/init_64.c | 9 +-
arch/x86/mm/kasan_init_64.c | 66 ++++++++++++
include/linux/bootmem.h | 27 +++++
include/linux/memblock.h | 16 +++
include/linux/mm.h | 26 +++++
mm/memblock.c | 60 +++++++++--
mm/page_alloc.c | 207 ++++++++++++++++++++----------------
mm/sparse-vmemmap.c | 15 ++-
mm/sparse.c | 6 +-
12 files changed, 406 insertions(+), 129 deletions(-)
--
2.14.1
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 79+ messages in thread
* [PATCH v7 00/11] complete deferred page initialization
@ 2017-09-14 22:35 ` Pavel Tatashin
0 siblings, 0 replies; 79+ messages in thread
From: Pavel Tatashin @ 2017-09-14 22:35 UTC (permalink / raw)
To: linux-arm-kernel
Changelog:
v8 - v7
- Added Acked-by's from Dave Miller for SPARC changes
- Fixed a minor compiling issue on tile architecture reported by kbuild
v7 - v6
- Addressed comments from Michal Hocko
- memblock_discard() patch was removed from this series and integrated
separately
- Fixed bug reported by kbuild test robot new patch:
mm: zero reserved and unavailable struct pages
- Removed patch
x86/mm: reserve only exiting low pages
As, it is not needed anymore, because of the previous fix
- Re-wrote deferred_init_memmap(), found and fixed an existing bug, where
page variable is not reset when zone holes present.
- Merged several patches together per Michal request
- Added performance data including raw logs
v6 - v5
- Fixed ARM64 + kasan code, as reported by Ard Biesheuvel
- Tested ARM64 code in qemu and found few more issues, that I fixed in this
iteration
- Added page roundup/rounddown to x86 and arm zeroing routines to zero the
whole allocated range, instead of only provided address range.
- Addressed SPARC related comment from Sam Ravnborg
- Fixed section mismatch warnings related to memblock_discard().
v5 - v4
- Fixed build issues reported by kbuild on various configurations
v4 - v3
- Rewrote code to zero sturct pages in __init_single_page() as
suggested by Michal Hocko
- Added code to handle issues related to accessing struct page
memory before they are initialized.
v3 - v2
- Addressed David Miller comments about one change per patch:
* Splited changes to platforms into 4 patches
* Made "do not zero vmemmap_buf" as a separate patch
v2 - v1
- Per request, added s390 to deferred "struct page" zeroing
- Collected performance data on x86 which proofs the importance to
keep memset() as prefetch (see below).
SMP machines can benefit from the DEFERRED_STRUCT_PAGE_INIT config option,
which defers initializing struct pages until all cpus have been started so
it can be done in parallel.
However, this feature is sub-optimal, because the deferred page
initialization code expects that the struct pages have already been zeroed,
and the zeroing is done early in boot with a single thread only. Also, we
access that memory and set flags before struct pages are initialized. All
of this is fixed in this patchset.
In this work we do the following:
- Never read access struct page until it was initialized
- Never set any fields in struct pages before they are initialized
- Zero struct page at the beginning of struct page initialization
==========================================================================
Performance improvements on x86 machine with 8 nodes:
Intel(R) Xeon(R) CPU E7-8895 v3 @ 2.60GHz and 1T of memory:
TIME SPEED UP
base no deferred: 95.796233s
fix no deferred: 79.978956s 19.77%
base deferred: 77.254713s
fix deferred: 55.050509s 40.34%
==========================================================================
SPARC M6 3600 MHz with 15T of memory
TIME SPEED UP
base no deferred: 358.335727s
fix no deferred: 302.320936s 18.52%
base deferred: 237.534603s
fix deferred: 182.103003s 30.44%
==========================================================================
Raw dmesg output with timestamps:
x86 base no deferred: https://hastebin.com/ofunepurit.scala
x86 base deferred: https://hastebin.com/ifazegeyas.scala
x86 fix no deferred: https://hastebin.com/pegocohevo.scala
x86 fix deferred: https://hastebin.com/ofupevikuk.scala
sparc base no deferred: https://hastebin.com/ibobeteken.go
sparc base deferred: https://hastebin.com/fariqimiyu.go
sparc fix no deferred: https://hastebin.com/muhegoheyi.go
sparc fix deferred: https://hastebin.com/xadinobutu.go
Pavel Tatashin (11):
x86/mm: setting fields in deferred pages
sparc64/mm: setting fields in deferred pages
mm: deferred_init_memmap improvements
sparc64: simplify vmemmap_populate
mm: defining memblock_virt_alloc_try_nid_raw
mm: zero struct pages during initialization
sparc64: optimized struct page zeroing
mm: zero reserved and unavailable struct pages
x86/kasan: explicitly zero kasan shadow memory
arm64/kasan: explicitly zero kasan shadow memory
mm: stop zeroing memory during allocation in vmemmap
arch/arm64/mm/kasan_init.c | 42 ++++++++
arch/sparc/include/asm/pgtable_64.h | 30 ++++++
arch/sparc/mm/init_64.c | 31 +++---
arch/x86/mm/init_64.c | 9 +-
arch/x86/mm/kasan_init_64.c | 66 ++++++++++++
include/linux/bootmem.h | 27 +++++
include/linux/memblock.h | 16 +++
include/linux/mm.h | 26 +++++
mm/memblock.c | 60 +++++++++--
mm/page_alloc.c | 207 ++++++++++++++++++++----------------
mm/sparse-vmemmap.c | 15 ++-
mm/sparse.c | 6 +-
12 files changed, 406 insertions(+), 129 deletions(-)
--
2.14.1
^ permalink raw reply [flat|nested] 79+ messages in thread
end of thread, other threads:[~2017-09-14 22:39 UTC | newest]
Thread overview: 79+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2017-08-29 2:02 [PATCH v7 00/11] complete deferred page initialization Pavel Tatashin
2017-08-29 2:02 ` Pavel Tatashin
2017-08-29 2:02 ` Pavel Tatashin
2017-08-29 2:02 ` Pavel Tatashin
2017-08-29 2:02 ` [PATCH v7 01/11] x86/mm: setting fields in deferred pages Pavel Tatashin
2017-08-29 2:02 ` Pavel Tatashin
2017-08-29 2:02 ` Pavel Tatashin
2017-08-29 2:02 ` Pavel Tatashin
2017-08-29 2:02 ` [PATCH v7 02/11] sparc64/mm: " Pavel Tatashin
2017-08-29 2:02 ` Pavel Tatashin
2017-08-29 2:02 ` Pavel Tatashin
2017-08-29 2:02 ` Pavel Tatashin
2017-08-30 1:09 ` David Miller
2017-08-30 1:09 ` David Miller
2017-08-30 1:09 ` David Miller
2017-08-30 1:09 ` David Miller
2017-08-29 2:02 ` [PATCH v7 03/11] mm: deferred_init_memmap improvements Pavel Tatashin
2017-08-29 2:02 ` Pavel Tatashin
2017-08-29 2:02 ` Pavel Tatashin
2017-08-29 2:02 ` Pavel Tatashin
2017-08-29 2:02 ` [PATCH v7 04/11] sparc64: simplify vmemmap_populate Pavel Tatashin
2017-08-29 2:02 ` Pavel Tatashin
2017-08-29 2:02 ` Pavel Tatashin
2017-08-29 2:02 ` Pavel Tatashin
2017-08-30 1:08 ` David Miller
2017-08-30 1:08 ` David Miller
2017-08-30 1:08 ` David Miller
2017-08-30 1:08 ` David Miller
2017-08-29 2:02 ` [PATCH v7 05/11] mm: defining memblock_virt_alloc_try_nid_raw Pavel Tatashin
2017-08-29 2:02 ` Pavel Tatashin
2017-08-29 2:02 ` Pavel Tatashin
2017-08-29 2:02 ` Pavel Tatashin
2017-08-29 2:02 ` [PATCH v7 06/11] mm: zero struct pages during initialization Pavel Tatashin
2017-08-29 2:02 ` Pavel Tatashin
2017-08-29 2:02 ` Pavel Tatashin
2017-08-29 2:02 ` Pavel Tatashin
2017-08-29 2:02 ` [PATCH v7 07/11] sparc64: optimized struct page zeroing Pavel Tatashin
2017-08-29 2:02 ` Pavel Tatashin
2017-08-29 2:02 ` Pavel Tatashin
2017-08-29 2:02 ` Pavel Tatashin
2017-08-30 1:12 ` David Miller
2017-08-30 1:12 ` David Miller
2017-08-30 1:12 ` David Miller
2017-08-30 1:12 ` David Miller
2017-08-30 13:19 ` Pasha Tatashin
2017-08-30 13:19 ` Pasha Tatashin
2017-08-30 13:19 ` Pasha Tatashin
2017-08-30 13:19 ` Pasha Tatashin
2017-08-30 17:46 ` David Miller
2017-08-30 17:46 ` David Miller
2017-08-30 17:46 ` David Miller
2017-08-30 17:46 ` David Miller
2017-08-29 2:02 ` [PATCH v7 08/11] mm: zero reserved and unavailable struct pages Pavel Tatashin
2017-08-29 2:02 ` Pavel Tatashin
2017-08-29 2:02 ` Pavel Tatashin
2017-08-29 2:02 ` Pavel Tatashin
2017-08-30 21:22 ` kbuild test robot
2017-08-30 21:22 ` kbuild test robot
2017-08-30 21:22 ` kbuild test robot
2017-08-30 23:12 ` kbuild test robot
2017-08-30 23:12 ` kbuild test robot
2017-08-30 23:12 ` kbuild test robot
2017-08-30 23:12 ` kbuild test robot
2017-08-29 2:02 ` [PATCH v7 09/11] x86/kasan: explicitly zero kasan shadow memory Pavel Tatashin
2017-08-29 2:02 ` Pavel Tatashin
2017-08-29 2:02 ` Pavel Tatashin
2017-08-29 2:02 ` Pavel Tatashin
2017-08-29 2:02 ` [PATCH v7 10/11] arm64/kasan: " Pavel Tatashin
2017-08-29 2:02 ` Pavel Tatashin
2017-08-29 2:02 ` Pavel Tatashin
2017-08-29 2:02 ` Pavel Tatashin
2017-08-29 2:02 ` [PATCH v7 11/11] mm: stop zeroing memory during allocation in vmemmap Pavel Tatashin
2017-08-29 2:02 ` Pavel Tatashin
2017-08-29 2:02 ` Pavel Tatashin
2017-08-29 2:02 ` Pavel Tatashin
2017-09-14 22:35 [PATCH v7 00/11] complete deferred page initialization Pavel Tatashin
2017-09-14 22:35 ` Pavel Tatashin
2017-09-14 22:35 ` Pavel Tatashin
2017-09-14 22:35 ` Pavel Tatashin
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.