All of lore.kernel.org
 help / color / mirror / Atom feed
* [PATCH -v4 0/14] mm: make movable onlining suck less
@ 2017-05-15  8:58 ` Michal Hocko
  0 siblings, 0 replies; 74+ messages in thread
From: Michal Hocko @ 2017-05-15  8:58 UTC (permalink / raw)
  To: Andrew Morton
  Cc: linux-mm, Mel Gorman, Vlastimil Babka, Andrea Arcangeli,
	Jerome Glisse, Reza Arbab, Yasuaki Ishimatsu, qiuxishi,
	Kani Toshimitsu, slaoub, Joonsoo Kim, Andi Kleen, David Rientjes,
	Daniel Kiper, Igor Mammedov, Vitaly Kuznetsov, LKML,
	Balbir Singh, Dan Williams, Heiko Carstens, Martin Schwidefsky,
	Michal Hocko, Tobias Regnery, Yasuaki Ishimatsu

Hi,
The last version of this series has been posted here [1]. The timing wasn't
all that great so this is mostly a resubmit. I've added one additional patch
to fix another pfn walker noticed by Joonsoo (this is patch 11) and also
added a clarification for pfn_valid() and offline pages.

There is still a lot of work on top - namely this implementation doesn't
support reonlining to a different zone on the zones boundaries but I
will do that in a separate series because this one is getting quite
large already and it should work reasonably well now.

Joonsoo had some worries about pfn_valid and suggested to change its
semantic to return false on offline holes but I would be rally worried
to change a established semantic used by a lot of code and so I have
introuduced pfn_to_online_page helper instead. If this is seen as a
controversial point I would rather drop pfn_to_online_page and related
patches as they are not stictly necessary because the code would be
similarly broken as now wrt. offline holes.

This is a rebase on top of the current mmotm tree (mmotm-2017-05-12-15-53)
and the full series is in
git://git.kernel.org/pub/scm/linux/kernel/git/mhocko/mm.git try
attempts/rewrite-mem_hotplug branch.

Motivation:
Movable onlining is a real hack with many downsides - mainly
reintroduction of lowmem/highmem issues we used to have on 32b systems -
but it is the only way to make the memory hotremove more reliable which
is something that people are asking for.

The current semantic of memory movable onlinening is really cumbersome,
however. The main reason for this is that the udev driven approach is
basically unusable because udev races with the memory probing while only
the last memory block or the one adjacent to the existing zone_movable
are allowed to be onlined movable. In short the criterion for the
successful online_movable changes under udev's feet. A reliable udev
approach would require a 2 phase approach where the first successful
movable online would have to check all the previous blocks and online
them in descending order. This is hard to be considered sane.

This patchset aims at making the onlining semantic more usable. First of
all it allows to online memory movable as long as it doesn't clash with
the existing ZONE_NORMAL. That means that ZONE_NORMAL and ZONE_MOVABLE
cannot overlap. Currently I preserve the original ordering semantic so
the zone always precedes the movable zone but I have plans to remove this
restriction in future because it is not really necessary.

First 3 patches are cleanups which should be ready to be merged right
away (unless I have missed something subtle of course).

Patch 4 deals with ZONE_DEVICE dependencies down the __add_pages path.

Patch 5 deals with implicit assumptions of register_one_node on pgdat
initialization.

Patches 6-10 deal with offline holes in the zone for pfn walkers. I
hope I got all of them right but people familiar with compaction should
double check this.

Patch 11 is the core of the change. In order to make it easier to review
I have tried it to be as minimalistic as possible and the large code
removal is moved to patch 14.

Patch 12 is a trivial follow up cleanup. Patch 13 fixes sparse warnings
and finally patch 14 removes the unused code.

I have tested the patches in kvm:
# qemu-system-x86_64 -enable-kvm -monitor pty -m 2G,slots=4,maxmem=4G -numa node,mem=1G -numa node,mem=1G ...

and then probed the additional memory by
(qemu) object_add memory-backend-ram,id=mem1,size=1G
(qemu) device_add pc-dimm,id=dimm1,memdev=mem1

Then I have used this simple script to probe the memory block by hand
# cat probe_memblock.sh
#!/bin/sh

BLOCK_NR=$1

# echo $((0x100000000+$BLOCK_NR*(128<<20))) > /sys/devices/system/memory/probe

# for i in $(seq 10); do sh probe_memblock.sh $i; done
# grep . /sys/devices/system/memory/memory3?/valid_zones 2>/dev/null 
/sys/devices/system/memory/memory33/valid_zones:Normal Movable                                                                                                                                                     
/sys/devices/system/memory/memory34/valid_zones:Normal Movable                                                                                                                                                     
/sys/devices/system/memory/memory35/valid_zones:Normal Movable                                                                                                                                                     
/sys/devices/system/memory/memory36/valid_zones:Normal Movable                                                                                                                                                     
/sys/devices/system/memory/memory37/valid_zones:Normal Movable                                                                                                                                                     
/sys/devices/system/memory/memory38/valid_zones:Normal Movable                                                                                                                                                     
/sys/devices/system/memory/memory39/valid_zones:Normal Movable

The main difference to the original implementation is that all new
memblocks can be both online_kernel and online_movable initially
because there is no clash obviously. For the comparison the original
implementation would have

/sys/devices/system/memory/memory33/valid_zones:Normal                                                                                                                                                     
/sys/devices/system/memory/memory34/valid_zones:Normal                                                                                                                                                     
/sys/devices/system/memory/memory35/valid_zones:Normal                                                                                                                                                     
/sys/devices/system/memory/memory36/valid_zones:Normal                                                                                                                                                     
/sys/devices/system/memory/memory37/valid_zones:Normal                                                                                                                                                     
/sys/devices/system/memory/memory38/valid_zones:Normal                                                                                                                                                     
/sys/devices/system/memory/memory39/valid_zones:Normal Movable

Now
# echo online_movable > /sys/devices/system/memory/memory34/state                                                                                                                                      
# grep . /sys/devices/system/memory/memory3?/valid_zones 2>/dev/null                                                                                                                                   
/sys/devices/system/memory/memory33/valid_zones:Normal Movable                                                                                                                                                     
/sys/devices/system/memory/memory34/valid_zones:Movable                                                                                                                                                            
/sys/devices/system/memory/memory35/valid_zones:Movable                                                                                                                                                            
/sys/devices/system/memory/memory36/valid_zones:Movable                                                                                                                                                            
/sys/devices/system/memory/memory37/valid_zones:Movable                                                                                                                                                            
/sys/devices/system/memory/memory38/valid_zones:Movable
/sys/devices/system/memory/memory39/valid_zones:Movable

Block 33 can still be online both kernel and movable while all
the remaining can be only movable.
/proc/zonelist says
Node 0, zone   Normal
  pages free     0
        min      0
        low      0
        high     0
        spanned  0
        present  0
--
Node 0, zone  Movable
  pages free     32753
        min      85
        low      117
        high     149
        spanned  32768
        present  32768

A new memblock at a lower address will result in a new memblock (32)
which will still allow both Normal and Movable.
# sh probe_memblock.sh 0
# grep . /sys/devices/system/memory/memory3[2-5]/valid_zones 2>/dev/null
/sys/devices/system/memory/memory32/valid_zones:Normal Movable
/sys/devices/system/memory/memory33/valid_zones:Normal Movable
/sys/devices/system/memory/memory34/valid_zones:Movable
/sys/devices/system/memory/memory35/valid_zones:Movable

and online_kernel will convert it to the zone normal properly
while 33 can be still onlined both ways.
# echo online_kernel > /sys/devices/system/memory/memory32/state
# grep . /sys/devices/system/memory/memory3[2-5]/valid_zones 2>/dev/null
/sys/devices/system/memory/memory32/valid_zones:Normal
/sys/devices/system/memory/memory33/valid_zones:Normal Movable
/sys/devices/system/memory/memory34/valid_zones:Movable
/sys/devices/system/memory/memory35/valid_zones:Movable

/proc/zoneinfo will now tell
Node 0, zone   Normal
  pages free     65441
        min      165
        low      230
        high     295
        spanned  65536
        present  65536
--
Node 0, zone  Movable
  pages free     32740
        min      82
        low      114
        high     146
        spanned  32768
        present  32768

so both zones have one memblock spanned and present.

Onlining 39 should associate this block to the movable zone
# echo online > /sys/devices/system/memory/memory39/state

/proc/zoneinfo will now tell
Node 0, zone   Normal
  pages free     32765
        min      80
        low      112
        high     144
        spanned  32768
        present  32768
--
Node 0, zone  Movable
  pages free     65501
        min      160
        low      225
        high     290
        spanned  196608
        present  65536

so we will have a movable zone which spans 6 memblocks, 2 present and 4
representing a hole.

Offlining both movable blocks will lead to the zone with no present
pages which is the expected behavior I believe.
# echo offline > /sys/devices/system/memory/memory39/state
# echo offline > /sys/devices/system/memory/memory34/state
# grep -A6 "Movable\|Normal" /proc/zoneinfo 
Node 0, zone   Normal
  pages free     32735
        min      90
        low      122
        high     154
        spanned  32768
        present  32768
--
Node 0, zone  Movable
  pages free     0
        min      0
        low      0
        high     0
        spanned  196608
        present  0

Any thoughts, complains, suggestions?

As a bonus we will get a nice cleanup in the memory hotplug codebase.
 arch/ia64/mm/init.c            |  11 +-
 arch/powerpc/mm/mem.c          |  12 +-
 arch/s390/mm/init.c            |  32 +--
 arch/sh/mm/init.c              |  10 +-
 arch/x86/mm/init_32.c          |   7 +-
 arch/x86/mm/init_64.c          |  11 +-
 drivers/base/memory.c          |  79 +++----
 drivers/base/node.c            |  58 ++----
 include/linux/memory_hotplug.h |  40 +++-
 include/linux/mmzone.h         |  57 +++++-
 include/linux/node.h           |  35 +++-
 kernel/memremap.c              |   6 +-
 mm/compaction.c                |   5 +-
 mm/memory_hotplug.c            | 455 ++++++++++++++---------------------------
 mm/page_alloc.c                |  13 +-
 mm/page_isolation.c            |  26 ++-
 mm/sparse.c                    |  48 ++++-
 mm/vmstat.c                    |   4 +-
 18 files changed, 417 insertions(+), 492 deletions(-)

Shortlog says:
Michal Hocko (14):
      mm: remove return value from init_currently_empty_zone
      mm, memory_hotplug: use node instead of zone in can_online_high_movable
      mm: drop page_initialized check from get_nid_for_pfn
      mm, memory_hotplug: get rid of is_zone_device_section
      mm, memory_hotplug: split up register_one_node
      mm, memory_hotplug: consider offline memblocks removable
      mm: consider zone which is not fully populated to have holes
      mm, compaction: skip over holes in __reset_isolation_suitable
      mm: __first_valid_page skip over offline pages
      mm, vmstat: skip reporting offline pages in pagetypeinfo
      mm, memory_hotplug: do not associate hotadded memory to zones until online
      mm, memory_hotplug: replace for_device by want_memblock in arch_add_memory
      mm, memory_hotplug: fix the section mismatch warning
      mm, memory_hotplug: remove unused cruft after memory hotplug rework

[1] http://lkml.kernel.org/r/20170421120512.23960-1-mhocko@kernel.org

^ permalink raw reply	[flat|nested] 74+ messages in thread

* [PATCH -v4 0/14] mm: make movable onlining suck less
@ 2017-05-15  8:58 ` Michal Hocko
  0 siblings, 0 replies; 74+ messages in thread
From: Michal Hocko @ 2017-05-15  8:58 UTC (permalink / raw)
  To: Andrew Morton
  Cc: linux-mm, Mel Gorman, Vlastimil Babka, Andrea Arcangeli,
	Jerome Glisse, Reza Arbab, Yasuaki Ishimatsu, qiuxishi,
	Kani Toshimitsu, slaoub, Joonsoo Kim, Andi Kleen, David Rientjes,
	Daniel Kiper, Igor Mammedov, Vitaly Kuznetsov, LKML,
	Balbir Singh, Dan Williams, Heiko Carstens, Martin Schwidefsky,
	Michal Hocko, Tobias Regnery, Yasuaki Ishimatsu

Hi,
The last version of this series has been posted here [1]. The timing wasn't
all that great so this is mostly a resubmit. I've added one additional patch
to fix another pfn walker noticed by Joonsoo (this is patch 11) and also
added a clarification for pfn_valid() and offline pages.

There is still a lot of work on top - namely this implementation doesn't
support reonlining to a different zone on the zones boundaries but I
will do that in a separate series because this one is getting quite
large already and it should work reasonably well now.

Joonsoo had some worries about pfn_valid and suggested to change its
semantic to return false on offline holes but I would be rally worried
to change a established semantic used by a lot of code and so I have
introuduced pfn_to_online_page helper instead. If this is seen as a
controversial point I would rather drop pfn_to_online_page and related
patches as they are not stictly necessary because the code would be
similarly broken as now wrt. offline holes.

This is a rebase on top of the current mmotm tree (mmotm-2017-05-12-15-53)
and the full series is in
git://git.kernel.org/pub/scm/linux/kernel/git/mhocko/mm.git try
attempts/rewrite-mem_hotplug branch.

Motivation:
Movable onlining is a real hack with many downsides - mainly
reintroduction of lowmem/highmem issues we used to have on 32b systems -
but it is the only way to make the memory hotremove more reliable which
is something that people are asking for.

The current semantic of memory movable onlinening is really cumbersome,
however. The main reason for this is that the udev driven approach is
basically unusable because udev races with the memory probing while only
the last memory block or the one adjacent to the existing zone_movable
are allowed to be onlined movable. In short the criterion for the
successful online_movable changes under udev's feet. A reliable udev
approach would require a 2 phase approach where the first successful
movable online would have to check all the previous blocks and online
them in descending order. This is hard to be considered sane.

This patchset aims at making the onlining semantic more usable. First of
all it allows to online memory movable as long as it doesn't clash with
the existing ZONE_NORMAL. That means that ZONE_NORMAL and ZONE_MOVABLE
cannot overlap. Currently I preserve the original ordering semantic so
the zone always precedes the movable zone but I have plans to remove this
restriction in future because it is not really necessary.

First 3 patches are cleanups which should be ready to be merged right
away (unless I have missed something subtle of course).

Patch 4 deals with ZONE_DEVICE dependencies down the __add_pages path.

Patch 5 deals with implicit assumptions of register_one_node on pgdat
initialization.

Patches 6-10 deal with offline holes in the zone for pfn walkers. I
hope I got all of them right but people familiar with compaction should
double check this.

Patch 11 is the core of the change. In order to make it easier to review
I have tried it to be as minimalistic as possible and the large code
removal is moved to patch 14.

Patch 12 is a trivial follow up cleanup. Patch 13 fixes sparse warnings
and finally patch 14 removes the unused code.

I have tested the patches in kvm:
# qemu-system-x86_64 -enable-kvm -monitor pty -m 2G,slots=4,maxmem=4G -numa node,mem=1G -numa node,mem=1G ...

and then probed the additional memory by
(qemu) object_add memory-backend-ram,id=mem1,size=1G
(qemu) device_add pc-dimm,id=dimm1,memdev=mem1

Then I have used this simple script to probe the memory block by hand
# cat probe_memblock.sh
#!/bin/sh

BLOCK_NR=$1

# echo $((0x100000000+$BLOCK_NR*(128<<20))) > /sys/devices/system/memory/probe

# for i in $(seq 10); do sh probe_memblock.sh $i; done
# grep . /sys/devices/system/memory/memory3?/valid_zones 2>/dev/null 
/sys/devices/system/memory/memory33/valid_zones:Normal Movable                                                                                                                                                     
/sys/devices/system/memory/memory34/valid_zones:Normal Movable                                                                                                                                                     
/sys/devices/system/memory/memory35/valid_zones:Normal Movable                                                                                                                                                     
/sys/devices/system/memory/memory36/valid_zones:Normal Movable                                                                                                                                                     
/sys/devices/system/memory/memory37/valid_zones:Normal Movable                                                                                                                                                     
/sys/devices/system/memory/memory38/valid_zones:Normal Movable                                                                                                                                                     
/sys/devices/system/memory/memory39/valid_zones:Normal Movable

The main difference to the original implementation is that all new
memblocks can be both online_kernel and online_movable initially
because there is no clash obviously. For the comparison the original
implementation would have

/sys/devices/system/memory/memory33/valid_zones:Normal                                                                                                                                                     
/sys/devices/system/memory/memory34/valid_zones:Normal                                                                                                                                                     
/sys/devices/system/memory/memory35/valid_zones:Normal                                                                                                                                                     
/sys/devices/system/memory/memory36/valid_zones:Normal                                                                                                                                                     
/sys/devices/system/memory/memory37/valid_zones:Normal                                                                                                                                                     
/sys/devices/system/memory/memory38/valid_zones:Normal                                                                                                                                                     
/sys/devices/system/memory/memory39/valid_zones:Normal Movable

Now
# echo online_movable > /sys/devices/system/memory/memory34/state                                                                                                                                      
# grep . /sys/devices/system/memory/memory3?/valid_zones 2>/dev/null                                                                                                                                   
/sys/devices/system/memory/memory33/valid_zones:Normal Movable                                                                                                                                                     
/sys/devices/system/memory/memory34/valid_zones:Movable                                                                                                                                                            
/sys/devices/system/memory/memory35/valid_zones:Movable                                                                                                                                                            
/sys/devices/system/memory/memory36/valid_zones:Movable                                                                                                                                                            
/sys/devices/system/memory/memory37/valid_zones:Movable                                                                                                                                                            
/sys/devices/system/memory/memory38/valid_zones:Movable
/sys/devices/system/memory/memory39/valid_zones:Movable

Block 33 can still be online both kernel and movable while all
the remaining can be only movable.
/proc/zonelist says
Node 0, zone   Normal
  pages free     0
        min      0
        low      0
        high     0
        spanned  0
        present  0
--
Node 0, zone  Movable
  pages free     32753
        min      85
        low      117
        high     149
        spanned  32768
        present  32768

A new memblock at a lower address will result in a new memblock (32)
which will still allow both Normal and Movable.
# sh probe_memblock.sh 0
# grep . /sys/devices/system/memory/memory3[2-5]/valid_zones 2>/dev/null
/sys/devices/system/memory/memory32/valid_zones:Normal Movable
/sys/devices/system/memory/memory33/valid_zones:Normal Movable
/sys/devices/system/memory/memory34/valid_zones:Movable
/sys/devices/system/memory/memory35/valid_zones:Movable

and online_kernel will convert it to the zone normal properly
while 33 can be still onlined both ways.
# echo online_kernel > /sys/devices/system/memory/memory32/state
# grep . /sys/devices/system/memory/memory3[2-5]/valid_zones 2>/dev/null
/sys/devices/system/memory/memory32/valid_zones:Normal
/sys/devices/system/memory/memory33/valid_zones:Normal Movable
/sys/devices/system/memory/memory34/valid_zones:Movable
/sys/devices/system/memory/memory35/valid_zones:Movable

/proc/zoneinfo will now tell
Node 0, zone   Normal
  pages free     65441
        min      165
        low      230
        high     295
        spanned  65536
        present  65536
--
Node 0, zone  Movable
  pages free     32740
        min      82
        low      114
        high     146
        spanned  32768
        present  32768

so both zones have one memblock spanned and present.

Onlining 39 should associate this block to the movable zone
# echo online > /sys/devices/system/memory/memory39/state

/proc/zoneinfo will now tell
Node 0, zone   Normal
  pages free     32765
        min      80
        low      112
        high     144
        spanned  32768
        present  32768
--
Node 0, zone  Movable
  pages free     65501
        min      160
        low      225
        high     290
        spanned  196608
        present  65536

so we will have a movable zone which spans 6 memblocks, 2 present and 4
representing a hole.

Offlining both movable blocks will lead to the zone with no present
pages which is the expected behavior I believe.
# echo offline > /sys/devices/system/memory/memory39/state
# echo offline > /sys/devices/system/memory/memory34/state
# grep -A6 "Movable\|Normal" /proc/zoneinfo 
Node 0, zone   Normal
  pages free     32735
        min      90
        low      122
        high     154
        spanned  32768
        present  32768
--
Node 0, zone  Movable
  pages free     0
        min      0
        low      0
        high     0
        spanned  196608
        present  0

Any thoughts, complains, suggestions?

As a bonus we will get a nice cleanup in the memory hotplug codebase.
 arch/ia64/mm/init.c            |  11 +-
 arch/powerpc/mm/mem.c          |  12 +-
 arch/s390/mm/init.c            |  32 +--
 arch/sh/mm/init.c              |  10 +-
 arch/x86/mm/init_32.c          |   7 +-
 arch/x86/mm/init_64.c          |  11 +-
 drivers/base/memory.c          |  79 +++----
 drivers/base/node.c            |  58 ++----
 include/linux/memory_hotplug.h |  40 +++-
 include/linux/mmzone.h         |  57 +++++-
 include/linux/node.h           |  35 +++-
 kernel/memremap.c              |   6 +-
 mm/compaction.c                |   5 +-
 mm/memory_hotplug.c            | 455 ++++++++++++++---------------------------
 mm/page_alloc.c                |  13 +-
 mm/page_isolation.c            |  26 ++-
 mm/sparse.c                    |  48 ++++-
 mm/vmstat.c                    |   4 +-
 18 files changed, 417 insertions(+), 492 deletions(-)

Shortlog says:
Michal Hocko (14):
      mm: remove return value from init_currently_empty_zone
      mm, memory_hotplug: use node instead of zone in can_online_high_movable
      mm: drop page_initialized check from get_nid_for_pfn
      mm, memory_hotplug: get rid of is_zone_device_section
      mm, memory_hotplug: split up register_one_node
      mm, memory_hotplug: consider offline memblocks removable
      mm: consider zone which is not fully populated to have holes
      mm, compaction: skip over holes in __reset_isolation_suitable
      mm: __first_valid_page skip over offline pages
      mm, vmstat: skip reporting offline pages in pagetypeinfo
      mm, memory_hotplug: do not associate hotadded memory to zones until online
      mm, memory_hotplug: replace for_device by want_memblock in arch_add_memory
      mm, memory_hotplug: fix the section mismatch warning
      mm, memory_hotplug: remove unused cruft after memory hotplug rework

[1] http://lkml.kernel.org/r/20170421120512.23960-1-mhocko@kernel.org


--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

^ permalink raw reply	[flat|nested] 74+ messages in thread

* [PATCH 01/14] mm: remove return value from init_currently_empty_zone
  2017-05-15  8:58 ` Michal Hocko
@ 2017-05-15  8:58   ` Michal Hocko
  -1 siblings, 0 replies; 74+ messages in thread
From: Michal Hocko @ 2017-05-15  8:58 UTC (permalink / raw)
  To: Andrew Morton
  Cc: linux-mm, Mel Gorman, Vlastimil Babka, Andrea Arcangeli,
	Jerome Glisse, Reza Arbab, Yasuaki Ishimatsu, qiuxishi,
	Kani Toshimitsu, slaoub, Joonsoo Kim, Andi Kleen, David Rientjes,
	Daniel Kiper, Igor Mammedov, Vitaly Kuznetsov, LKML,
	Michal Hocko, Balbir Singh, Yasuaki Ishimatsu

From: Michal Hocko <mhocko@suse.com>

init_currently_empty_zone doesn't have any error to return yet it is
still an int and callers try to be defensive and try to handle potential
error. Remove this nonsense and simplify all callers.

This patch shouldn't have any visible effect

Acked-by: Balbir Singh <bsingharora@gmail.com>
Acked-by: Vlastimil Babka <vbabka@suse.cz>
Reviewed-by: Yasuaki Ishimatsu <isimatu.yasuaki@jp.fujitsu.com>
Signed-off-by: Michal Hocko <mhocko@suse.com>
---
 include/linux/mmzone.h |  2 +-
 mm/memory_hotplug.c    | 23 +++++------------------
 mm/page_alloc.c        |  8 ++------
 3 files changed, 8 insertions(+), 25 deletions(-)

diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h
index 087255b98dbc..8363dd27b8af 100644
--- a/include/linux/mmzone.h
+++ b/include/linux/mmzone.h
@@ -771,7 +771,7 @@ enum memmap_context {
 	MEMMAP_EARLY,
 	MEMMAP_HOTPLUG,
 };
-extern int init_currently_empty_zone(struct zone *zone, unsigned long start_pfn,
+extern void init_currently_empty_zone(struct zone *zone, unsigned long start_pfn,
 				     unsigned long size);
 
 extern void lruvec_init(struct lruvec *lruvec);
diff --git a/mm/memory_hotplug.c b/mm/memory_hotplug.c
index 717c5e301aa8..17dd511614ff 100644
--- a/mm/memory_hotplug.c
+++ b/mm/memory_hotplug.c
@@ -348,27 +348,20 @@ static void fix_zone_id(struct zone *zone, unsigned long start_pfn,
 		set_page_links(pfn_to_page(pfn), zid, nid, pfn);
 }
 
-/* Can fail with -ENOMEM from allocating a wait table with vmalloc() or
- * alloc_bootmem_node_nopanic()/memblock_virt_alloc_node_nopanic() */
-static int __ref ensure_zone_is_initialized(struct zone *zone,
+static void __ref ensure_zone_is_initialized(struct zone *zone,
 			unsigned long start_pfn, unsigned long num_pages)
 {
 	if (!zone_is_initialized(zone))
-		return init_currently_empty_zone(zone, start_pfn, num_pages);
-
-	return 0;
+		init_currently_empty_zone(zone, start_pfn, num_pages);
 }
 
 static int __meminit move_pfn_range_left(struct zone *z1, struct zone *z2,
 		unsigned long start_pfn, unsigned long end_pfn)
 {
-	int ret;
 	unsigned long flags;
 	unsigned long z1_start_pfn;
 
-	ret = ensure_zone_is_initialized(z1, start_pfn, end_pfn - start_pfn);
-	if (ret)
-		return ret;
+	ensure_zone_is_initialized(z1, start_pfn, end_pfn - start_pfn);
 
 	pgdat_resize_lock(z1->zone_pgdat, &flags);
 
@@ -404,13 +397,10 @@ static int __meminit move_pfn_range_left(struct zone *z1, struct zone *z2,
 static int __meminit move_pfn_range_right(struct zone *z1, struct zone *z2,
 		unsigned long start_pfn, unsigned long end_pfn)
 {
-	int ret;
 	unsigned long flags;
 	unsigned long z2_end_pfn;
 
-	ret = ensure_zone_is_initialized(z2, start_pfn, end_pfn - start_pfn);
-	if (ret)
-		return ret;
+	ensure_zone_is_initialized(z2, start_pfn, end_pfn - start_pfn);
 
 	pgdat_resize_lock(z1->zone_pgdat, &flags);
 
@@ -481,12 +471,9 @@ static int __meminit __add_zone(struct zone *zone, unsigned long phys_start_pfn)
 	int nid = pgdat->node_id;
 	int zone_type;
 	unsigned long flags, pfn;
-	int ret;
 
 	zone_type = zone - pgdat->node_zones;
-	ret = ensure_zone_is_initialized(zone, phys_start_pfn, nr_pages);
-	if (ret)
-		return ret;
+	ensure_zone_is_initialized(zone, phys_start_pfn, nr_pages);
 
 	pgdat_resize_lock(zone->zone_pgdat, &flags);
 	grow_zone_span(zone, phys_start_pfn, phys_start_pfn + nr_pages);
diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index beb2827fd5de..c1670f090107 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -5513,7 +5513,7 @@ static __meminit void zone_pcp_init(struct zone *zone)
 					 zone_batchsize(zone));
 }
 
-int __meminit init_currently_empty_zone(struct zone *zone,
+void __meminit init_currently_empty_zone(struct zone *zone,
 					unsigned long zone_start_pfn,
 					unsigned long size)
 {
@@ -5531,8 +5531,6 @@ int __meminit init_currently_empty_zone(struct zone *zone,
 
 	zone_init_free_lists(zone);
 	zone->initialized = 1;
-
-	return 0;
 }
 
 #ifdef CONFIG_HAVE_MEMBLOCK_NODE_MAP
@@ -5995,7 +5993,6 @@ static void __paginginit free_area_init_core(struct pglist_data *pgdat)
 {
 	enum zone_type j;
 	int nid = pgdat->node_id;
-	int ret;
 
 	pgdat_resize_init(pgdat);
 #ifdef CONFIG_NUMA_BALANCING
@@ -6077,8 +6074,7 @@ static void __paginginit free_area_init_core(struct pglist_data *pgdat)
 
 		set_pageblock_order();
 		setup_usemap(pgdat, zone, zone_start_pfn, size);
-		ret = init_currently_empty_zone(zone, zone_start_pfn, size);
-		BUG_ON(ret);
+		init_currently_empty_zone(zone, zone_start_pfn, size);
 		memmap_init(size, nid, j, zone_start_pfn);
 	}
 }
-- 
2.11.0

^ permalink raw reply	[flat|nested] 74+ messages in thread

* [PATCH 01/14] mm: remove return value from init_currently_empty_zone
@ 2017-05-15  8:58   ` Michal Hocko
  0 siblings, 0 replies; 74+ messages in thread
From: Michal Hocko @ 2017-05-15  8:58 UTC (permalink / raw)
  To: Andrew Morton
  Cc: linux-mm, Mel Gorman, Vlastimil Babka, Andrea Arcangeli,
	Jerome Glisse, Reza Arbab, Yasuaki Ishimatsu, qiuxishi,
	Kani Toshimitsu, slaoub, Joonsoo Kim, Andi Kleen, David Rientjes,
	Daniel Kiper, Igor Mammedov, Vitaly Kuznetsov, LKML,
	Michal Hocko, Balbir Singh, Yasuaki Ishimatsu

From: Michal Hocko <mhocko@suse.com>

init_currently_empty_zone doesn't have any error to return yet it is
still an int and callers try to be defensive and try to handle potential
error. Remove this nonsense and simplify all callers.

This patch shouldn't have any visible effect

Acked-by: Balbir Singh <bsingharora@gmail.com>
Acked-by: Vlastimil Babka <vbabka@suse.cz>
Reviewed-by: Yasuaki Ishimatsu <isimatu.yasuaki@jp.fujitsu.com>
Signed-off-by: Michal Hocko <mhocko@suse.com>
---
 include/linux/mmzone.h |  2 +-
 mm/memory_hotplug.c    | 23 +++++------------------
 mm/page_alloc.c        |  8 ++------
 3 files changed, 8 insertions(+), 25 deletions(-)

diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h
index 087255b98dbc..8363dd27b8af 100644
--- a/include/linux/mmzone.h
+++ b/include/linux/mmzone.h
@@ -771,7 +771,7 @@ enum memmap_context {
 	MEMMAP_EARLY,
 	MEMMAP_HOTPLUG,
 };
-extern int init_currently_empty_zone(struct zone *zone, unsigned long start_pfn,
+extern void init_currently_empty_zone(struct zone *zone, unsigned long start_pfn,
 				     unsigned long size);
 
 extern void lruvec_init(struct lruvec *lruvec);
diff --git a/mm/memory_hotplug.c b/mm/memory_hotplug.c
index 717c5e301aa8..17dd511614ff 100644
--- a/mm/memory_hotplug.c
+++ b/mm/memory_hotplug.c
@@ -348,27 +348,20 @@ static void fix_zone_id(struct zone *zone, unsigned long start_pfn,
 		set_page_links(pfn_to_page(pfn), zid, nid, pfn);
 }
 
-/* Can fail with -ENOMEM from allocating a wait table with vmalloc() or
- * alloc_bootmem_node_nopanic()/memblock_virt_alloc_node_nopanic() */
-static int __ref ensure_zone_is_initialized(struct zone *zone,
+static void __ref ensure_zone_is_initialized(struct zone *zone,
 			unsigned long start_pfn, unsigned long num_pages)
 {
 	if (!zone_is_initialized(zone))
-		return init_currently_empty_zone(zone, start_pfn, num_pages);
-
-	return 0;
+		init_currently_empty_zone(zone, start_pfn, num_pages);
 }
 
 static int __meminit move_pfn_range_left(struct zone *z1, struct zone *z2,
 		unsigned long start_pfn, unsigned long end_pfn)
 {
-	int ret;
 	unsigned long flags;
 	unsigned long z1_start_pfn;
 
-	ret = ensure_zone_is_initialized(z1, start_pfn, end_pfn - start_pfn);
-	if (ret)
-		return ret;
+	ensure_zone_is_initialized(z1, start_pfn, end_pfn - start_pfn);
 
 	pgdat_resize_lock(z1->zone_pgdat, &flags);
 
@@ -404,13 +397,10 @@ static int __meminit move_pfn_range_left(struct zone *z1, struct zone *z2,
 static int __meminit move_pfn_range_right(struct zone *z1, struct zone *z2,
 		unsigned long start_pfn, unsigned long end_pfn)
 {
-	int ret;
 	unsigned long flags;
 	unsigned long z2_end_pfn;
 
-	ret = ensure_zone_is_initialized(z2, start_pfn, end_pfn - start_pfn);
-	if (ret)
-		return ret;
+	ensure_zone_is_initialized(z2, start_pfn, end_pfn - start_pfn);
 
 	pgdat_resize_lock(z1->zone_pgdat, &flags);
 
@@ -481,12 +471,9 @@ static int __meminit __add_zone(struct zone *zone, unsigned long phys_start_pfn)
 	int nid = pgdat->node_id;
 	int zone_type;
 	unsigned long flags, pfn;
-	int ret;
 
 	zone_type = zone - pgdat->node_zones;
-	ret = ensure_zone_is_initialized(zone, phys_start_pfn, nr_pages);
-	if (ret)
-		return ret;
+	ensure_zone_is_initialized(zone, phys_start_pfn, nr_pages);
 
 	pgdat_resize_lock(zone->zone_pgdat, &flags);
 	grow_zone_span(zone, phys_start_pfn, phys_start_pfn + nr_pages);
diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index beb2827fd5de..c1670f090107 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -5513,7 +5513,7 @@ static __meminit void zone_pcp_init(struct zone *zone)
 					 zone_batchsize(zone));
 }
 
-int __meminit init_currently_empty_zone(struct zone *zone,
+void __meminit init_currently_empty_zone(struct zone *zone,
 					unsigned long zone_start_pfn,
 					unsigned long size)
 {
@@ -5531,8 +5531,6 @@ int __meminit init_currently_empty_zone(struct zone *zone,
 
 	zone_init_free_lists(zone);
 	zone->initialized = 1;
-
-	return 0;
 }
 
 #ifdef CONFIG_HAVE_MEMBLOCK_NODE_MAP
@@ -5995,7 +5993,6 @@ static void __paginginit free_area_init_core(struct pglist_data *pgdat)
 {
 	enum zone_type j;
 	int nid = pgdat->node_id;
-	int ret;
 
 	pgdat_resize_init(pgdat);
 #ifdef CONFIG_NUMA_BALANCING
@@ -6077,8 +6074,7 @@ static void __paginginit free_area_init_core(struct pglist_data *pgdat)
 
 		set_pageblock_order();
 		setup_usemap(pgdat, zone, zone_start_pfn, size);
-		ret = init_currently_empty_zone(zone, zone_start_pfn, size);
-		BUG_ON(ret);
+		init_currently_empty_zone(zone, zone_start_pfn, size);
 		memmap_init(size, nid, j, zone_start_pfn);
 	}
 }
-- 
2.11.0

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

^ permalink raw reply	[flat|nested] 74+ messages in thread

* [PATCH 02/14] mm, memory_hotplug: use node instead of zone in can_online_high_movable
  2017-05-15  8:58 ` Michal Hocko
@ 2017-05-15  8:58   ` Michal Hocko
  -1 siblings, 0 replies; 74+ messages in thread
From: Michal Hocko @ 2017-05-15  8:58 UTC (permalink / raw)
  To: Andrew Morton
  Cc: linux-mm, Mel Gorman, Vlastimil Babka, Andrea Arcangeli,
	Jerome Glisse, Reza Arbab, Yasuaki Ishimatsu, qiuxishi,
	Kani Toshimitsu, slaoub, Joonsoo Kim, Andi Kleen, David Rientjes,
	Daniel Kiper, Igor Mammedov, Vitaly Kuznetsov, LKML,
	Michal Hocko, Yasuaki Ishimatsu

From: Michal Hocko <mhocko@suse.com>

the primary purpose of this helper is to query the node state so use
the node id directly. This is a preparatory patch for later changes.

This shouldn't introduce any functional change

Acked-by: Vlastimil Babka <vbabka@suse.cz>
Reviewed-by: Yasuaki Ishimatsu <isimatu.yasuaki@jp.fujitsu.com>
Signed-off-by: Michal Hocko <mhocko@suse.com>
---
 mm/memory_hotplug.c | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/mm/memory_hotplug.c b/mm/memory_hotplug.c
index 17dd511614ff..6290d34b6331 100644
--- a/mm/memory_hotplug.c
+++ b/mm/memory_hotplug.c
@@ -945,15 +945,15 @@ static int online_pages_range(unsigned long start_pfn, unsigned long nr_pages,
  * When CONFIG_MOVABLE_NODE, we permit onlining of a node which doesn't have
  * normal memory.
  */
-static bool can_online_high_movable(struct zone *zone)
+static bool can_online_high_movable(int nid)
 {
 	return true;
 }
 #else /* CONFIG_MOVABLE_NODE */
 /* ensure every online node has NORMAL memory */
-static bool can_online_high_movable(struct zone *zone)
+static bool can_online_high_movable(int nid)
 {
-	return node_state(zone_to_nid(zone), N_NORMAL_MEMORY);
+	return node_state(nid, N_NORMAL_MEMORY);
 }
 #endif /* CONFIG_MOVABLE_NODE */
 
@@ -1087,7 +1087,7 @@ int __ref online_pages(unsigned long pfn, unsigned long nr_pages, int online_typ
 
 	if ((zone_idx(zone) > ZONE_NORMAL ||
 	    online_type == MMOP_ONLINE_MOVABLE) &&
-	    !can_online_high_movable(zone))
+	    !can_online_high_movable(pfn_to_nid(pfn)))
 		return -EINVAL;
 
 	if (online_type == MMOP_ONLINE_KERNEL) {
-- 
2.11.0

^ permalink raw reply	[flat|nested] 74+ messages in thread

* [PATCH 02/14] mm, memory_hotplug: use node instead of zone in can_online_high_movable
@ 2017-05-15  8:58   ` Michal Hocko
  0 siblings, 0 replies; 74+ messages in thread
From: Michal Hocko @ 2017-05-15  8:58 UTC (permalink / raw)
  To: Andrew Morton
  Cc: linux-mm, Mel Gorman, Vlastimil Babka, Andrea Arcangeli,
	Jerome Glisse, Reza Arbab, Yasuaki Ishimatsu, qiuxishi,
	Kani Toshimitsu, slaoub, Joonsoo Kim, Andi Kleen, David Rientjes,
	Daniel Kiper, Igor Mammedov, Vitaly Kuznetsov, LKML,
	Michal Hocko, Yasuaki Ishimatsu

From: Michal Hocko <mhocko@suse.com>

the primary purpose of this helper is to query the node state so use
the node id directly. This is a preparatory patch for later changes.

This shouldn't introduce any functional change

Acked-by: Vlastimil Babka <vbabka@suse.cz>
Reviewed-by: Yasuaki Ishimatsu <isimatu.yasuaki@jp.fujitsu.com>
Signed-off-by: Michal Hocko <mhocko@suse.com>
---
 mm/memory_hotplug.c | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/mm/memory_hotplug.c b/mm/memory_hotplug.c
index 17dd511614ff..6290d34b6331 100644
--- a/mm/memory_hotplug.c
+++ b/mm/memory_hotplug.c
@@ -945,15 +945,15 @@ static int online_pages_range(unsigned long start_pfn, unsigned long nr_pages,
  * When CONFIG_MOVABLE_NODE, we permit onlining of a node which doesn't have
  * normal memory.
  */
-static bool can_online_high_movable(struct zone *zone)
+static bool can_online_high_movable(int nid)
 {
 	return true;
 }
 #else /* CONFIG_MOVABLE_NODE */
 /* ensure every online node has NORMAL memory */
-static bool can_online_high_movable(struct zone *zone)
+static bool can_online_high_movable(int nid)
 {
-	return node_state(zone_to_nid(zone), N_NORMAL_MEMORY);
+	return node_state(nid, N_NORMAL_MEMORY);
 }
 #endif /* CONFIG_MOVABLE_NODE */
 
@@ -1087,7 +1087,7 @@ int __ref online_pages(unsigned long pfn, unsigned long nr_pages, int online_typ
 
 	if ((zone_idx(zone) > ZONE_NORMAL ||
 	    online_type == MMOP_ONLINE_MOVABLE) &&
-	    !can_online_high_movable(zone))
+	    !can_online_high_movable(pfn_to_nid(pfn)))
 		return -EINVAL;
 
 	if (online_type == MMOP_ONLINE_KERNEL) {
-- 
2.11.0

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

^ permalink raw reply	[flat|nested] 74+ messages in thread

* [PATCH 03/14] mm: drop page_initialized check from get_nid_for_pfn
  2017-05-15  8:58 ` Michal Hocko
@ 2017-05-15  8:58   ` Michal Hocko
  -1 siblings, 0 replies; 74+ messages in thread
From: Michal Hocko @ 2017-05-15  8:58 UTC (permalink / raw)
  To: Andrew Morton
  Cc: linux-mm, Mel Gorman, Vlastimil Babka, Andrea Arcangeli,
	Jerome Glisse, Reza Arbab, Yasuaki Ishimatsu, qiuxishi,
	Kani Toshimitsu, slaoub, Joonsoo Kim, Andi Kleen, David Rientjes,
	Daniel Kiper, Igor Mammedov, Vitaly Kuznetsov, LKML,
	Michal Hocko

From: Michal Hocko <mhocko@suse.com>

c04fc586c1a4 ("mm: show node to memory section relationship with
symlinks in sysfs") has added means to export memblock<->node
association into the sysfs. It has also introduced get_nid_for_pfn
which is a rather confusing counterpart of pfn_to_nid which checks also
whether the pfn page is already initialized (page_initialized).  This
is done by checking page::lru != NULL which doesn't make any sense at
all. Nothing in this path really relies on the lru list being used or
initialized. Just remove it because this will become a problem with
later patches.

Thanks to Reza Arbab for testing which revealed this to be a problem
(http://lkml.kernel.org/r/20170403202337.GA12482@dhcp22.suse.cz)

Acked-by: Vlastimil Babka <vbabka@suse.cz>
Signed-off-by: Michal Hocko <mhocko@suse.com>
---
 drivers/base/node.c | 7 -------
 1 file changed, 7 deletions(-)

diff --git a/drivers/base/node.c b/drivers/base/node.c
index 5548f9686016..06294d69779b 100644
--- a/drivers/base/node.c
+++ b/drivers/base/node.c
@@ -368,21 +368,14 @@ int unregister_cpu_under_node(unsigned int cpu, unsigned int nid)
 }
 
 #ifdef CONFIG_MEMORY_HOTPLUG_SPARSE
-#define page_initialized(page)  (page->lru.next)
-
 static int __ref get_nid_for_pfn(unsigned long pfn)
 {
-	struct page *page;
-
 	if (!pfn_valid_within(pfn))
 		return -1;
 #ifdef CONFIG_DEFERRED_STRUCT_PAGE_INIT
 	if (system_state == SYSTEM_BOOTING)
 		return early_pfn_to_nid(pfn);
 #endif
-	page = pfn_to_page(pfn);
-	if (!page_initialized(page))
-		return -1;
 	return pfn_to_nid(pfn);
 }
 
-- 
2.11.0

^ permalink raw reply	[flat|nested] 74+ messages in thread

* [PATCH 03/14] mm: drop page_initialized check from get_nid_for_pfn
@ 2017-05-15  8:58   ` Michal Hocko
  0 siblings, 0 replies; 74+ messages in thread
From: Michal Hocko @ 2017-05-15  8:58 UTC (permalink / raw)
  To: Andrew Morton
  Cc: linux-mm, Mel Gorman, Vlastimil Babka, Andrea Arcangeli,
	Jerome Glisse, Reza Arbab, Yasuaki Ishimatsu, qiuxishi,
	Kani Toshimitsu, slaoub, Joonsoo Kim, Andi Kleen, David Rientjes,
	Daniel Kiper, Igor Mammedov, Vitaly Kuznetsov, LKML,
	Michal Hocko

From: Michal Hocko <mhocko@suse.com>

c04fc586c1a4 ("mm: show node to memory section relationship with
symlinks in sysfs") has added means to export memblock<->node
association into the sysfs. It has also introduced get_nid_for_pfn
which is a rather confusing counterpart of pfn_to_nid which checks also
whether the pfn page is already initialized (page_initialized).  This
is done by checking page::lru != NULL which doesn't make any sense at
all. Nothing in this path really relies on the lru list being used or
initialized. Just remove it because this will become a problem with
later patches.

Thanks to Reza Arbab for testing which revealed this to be a problem
(http://lkml.kernel.org/r/20170403202337.GA12482@dhcp22.suse.cz)

Acked-by: Vlastimil Babka <vbabka@suse.cz>
Signed-off-by: Michal Hocko <mhocko@suse.com>
---
 drivers/base/node.c | 7 -------
 1 file changed, 7 deletions(-)

diff --git a/drivers/base/node.c b/drivers/base/node.c
index 5548f9686016..06294d69779b 100644
--- a/drivers/base/node.c
+++ b/drivers/base/node.c
@@ -368,21 +368,14 @@ int unregister_cpu_under_node(unsigned int cpu, unsigned int nid)
 }
 
 #ifdef CONFIG_MEMORY_HOTPLUG_SPARSE
-#define page_initialized(page)  (page->lru.next)
-
 static int __ref get_nid_for_pfn(unsigned long pfn)
 {
-	struct page *page;
-
 	if (!pfn_valid_within(pfn))
 		return -1;
 #ifdef CONFIG_DEFERRED_STRUCT_PAGE_INIT
 	if (system_state == SYSTEM_BOOTING)
 		return early_pfn_to_nid(pfn);
 #endif
-	page = pfn_to_page(pfn);
-	if (!page_initialized(page))
-		return -1;
 	return pfn_to_nid(pfn);
 }
 
-- 
2.11.0

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

^ permalink raw reply	[flat|nested] 74+ messages in thread

* [PATCH 04/14] mm, memory_hotplug: get rid of is_zone_device_section
  2017-05-15  8:58 ` Michal Hocko
@ 2017-05-15  8:58   ` Michal Hocko
  -1 siblings, 0 replies; 74+ messages in thread
From: Michal Hocko @ 2017-05-15  8:58 UTC (permalink / raw)
  To: Andrew Morton
  Cc: linux-mm, Mel Gorman, Vlastimil Babka, Andrea Arcangeli,
	Jerome Glisse, Reza Arbab, Yasuaki Ishimatsu, qiuxishi,
	Kani Toshimitsu, slaoub, Joonsoo Kim, Andi Kleen, David Rientjes,
	Daniel Kiper, Igor Mammedov, Vitaly Kuznetsov, LKML,
	Michal Hocko, Dan Williams

From: Michal Hocko <mhocko@suse.com>

device memory hotplug hooks into regular memory hotplug only half way.
It needs memory sections to track struct pages but there is no
need/desire to associate those sections with memory blocks and export
them to the userspace via sysfs because they cannot be onlined anyway.

This is currently expressed by for_device argument to arch_add_memory
which then makes sure to associate the given memory range with
ZONE_DEVICE. register_new_memory then relies on is_zone_device_section
to distinguish special memory hotplug from the regular one. While this
works now, later patches in this series want to move __add_zone outside
of arch_add_memory path so we have to come up with something else.

Add want_memblock down the __add_pages path and use it to control
whether the section->memblock association should be done. arch_add_memory
then just trivially want memblock for everything but for_device hotplug.

remove_memory_section doesn't need is_zone_device_section either. We can
simply skip all the memblock specific cleanup if there is no memblock
for the given section.

This shouldn't introduce any functional change.

Changes since v1
- return 0 if want_memblock == 0 from __add_section as per Jerome Glisse

Changes since v2
- fix remove_memory_section unlock on find_memory_block failure
  as per Jerome - spotted by Evgeny Baskakov

Tested-by: Dan Williams <dan.j.williams@intel.com>
Acked-by: Vlastimil Babka <vbabka@suse.cz>
Signed-off-by: Michal Hocko <mhocko@suse.com>
---
 arch/ia64/mm/init.c            |  2 +-
 arch/powerpc/mm/mem.c          |  2 +-
 arch/s390/mm/init.c            |  2 +-
 arch/sh/mm/init.c              |  2 +-
 arch/x86/mm/init_32.c          |  2 +-
 arch/x86/mm/init_64.c          |  2 +-
 drivers/base/memory.c          | 23 +++++++++--------------
 include/linux/memory_hotplug.h |  2 +-
 mm/memory_hotplug.c            |  9 ++++++---
 9 files changed, 22 insertions(+), 24 deletions(-)

diff --git a/arch/ia64/mm/init.c b/arch/ia64/mm/init.c
index 8f3efa682ee8..39e2aeb4669d 100644
--- a/arch/ia64/mm/init.c
+++ b/arch/ia64/mm/init.c
@@ -658,7 +658,7 @@ int arch_add_memory(int nid, u64 start, u64 size, bool for_device)
 
 	zone = pgdat->node_zones +
 		zone_for_memory(nid, start, size, ZONE_NORMAL, for_device);
-	ret = __add_pages(nid, zone, start_pfn, nr_pages);
+	ret = __add_pages(nid, zone, start_pfn, nr_pages, !for_device);
 
 	if (ret)
 		printk("%s: Problem encountered in __add_pages() as ret=%d\n",
diff --git a/arch/powerpc/mm/mem.c b/arch/powerpc/mm/mem.c
index 9ee536ec0739..e6b2e6618b6c 100644
--- a/arch/powerpc/mm/mem.c
+++ b/arch/powerpc/mm/mem.c
@@ -151,7 +151,7 @@ int arch_add_memory(int nid, u64 start, u64 size, bool for_device)
 	zone = pgdata->node_zones +
 		zone_for_memory(nid, start, size, 0, for_device);
 
-	return __add_pages(nid, zone, start_pfn, nr_pages);
+	return __add_pages(nid, zone, start_pfn, nr_pages, !for_device);
 }
 
 #ifdef CONFIG_MEMORY_HOTREMOVE
diff --git a/arch/s390/mm/init.c b/arch/s390/mm/init.c
index ee6a1d3d4983..893cf88cf02d 100644
--- a/arch/s390/mm/init.c
+++ b/arch/s390/mm/init.c
@@ -191,7 +191,7 @@ int arch_add_memory(int nid, u64 start, u64 size, bool for_device)
 			continue;
 		nr_pages = (start_pfn + size_pages > zone_end_pfn) ?
 			   zone_end_pfn - start_pfn : size_pages;
-		rc = __add_pages(nid, zone, start_pfn, nr_pages);
+		rc = __add_pages(nid, zone, start_pfn, nr_pages, !for_device);
 		if (rc)
 			break;
 		start_pfn += nr_pages;
diff --git a/arch/sh/mm/init.c b/arch/sh/mm/init.c
index 75491862d900..a9d57f75ae8c 100644
--- a/arch/sh/mm/init.c
+++ b/arch/sh/mm/init.c
@@ -498,7 +498,7 @@ int arch_add_memory(int nid, u64 start, u64 size, bool for_device)
 	ret = __add_pages(nid, pgdat->node_zones +
 			zone_for_memory(nid, start, size, ZONE_NORMAL,
 			for_device),
-			start_pfn, nr_pages);
+			start_pfn, nr_pages, !for_device);
 	if (unlikely(ret))
 		printk("%s: Failed, __add_pages() == %d\n", __func__, ret);
 
diff --git a/arch/x86/mm/init_32.c b/arch/x86/mm/init_32.c
index 99fb83819a5f..94594b889144 100644
--- a/arch/x86/mm/init_32.c
+++ b/arch/x86/mm/init_32.c
@@ -831,7 +831,7 @@ int arch_add_memory(int nid, u64 start, u64 size, bool for_device)
 	unsigned long start_pfn = start >> PAGE_SHIFT;
 	unsigned long nr_pages = size >> PAGE_SHIFT;
 
-	return __add_pages(nid, zone, start_pfn, nr_pages);
+	return __add_pages(nid, zone, start_pfn, nr_pages, !for_device);
 }
 
 #ifdef CONFIG_MEMORY_HOTREMOVE
diff --git a/arch/x86/mm/init_64.c b/arch/x86/mm/init_64.c
index 41270b96403d..2e004364a373 100644
--- a/arch/x86/mm/init_64.c
+++ b/arch/x86/mm/init_64.c
@@ -697,7 +697,7 @@ int arch_add_memory(int nid, u64 start, u64 size, bool for_device)
 
 	init_memory_mapping(start, start + size);
 
-	ret = __add_pages(nid, zone, start_pfn, nr_pages);
+	ret = __add_pages(nid, zone, start_pfn, nr_pages, !for_device);
 	WARN_ON_ONCE(ret);
 
 	/* update max_pfn, max_low_pfn and high_memory */
diff --git a/drivers/base/memory.c b/drivers/base/memory.c
index 90225ffee501..f8fd562c3f18 100644
--- a/drivers/base/memory.c
+++ b/drivers/base/memory.c
@@ -685,14 +685,6 @@ static int add_memory_block(int base_section_nr)
 	return 0;
 }
 
-static bool is_zone_device_section(struct mem_section *ms)
-{
-	struct page *page;
-
-	page = sparse_decode_mem_map(ms->section_mem_map, __section_nr(ms));
-	return is_zone_device_page(page);
-}
-
 /*
  * need an interface for the VM to add new memory regions,
  * but without onlining it.
@@ -702,9 +694,6 @@ int register_new_memory(int nid, struct mem_section *section)
 	int ret = 0;
 	struct memory_block *mem;
 
-	if (is_zone_device_section(section))
-		return 0;
-
 	mutex_lock(&mem_sysfs_mutex);
 
 	mem = find_memory_block(section);
@@ -741,11 +730,16 @@ static int remove_memory_section(unsigned long node_id,
 {
 	struct memory_block *mem;
 
-	if (is_zone_device_section(section))
-		return 0;
-
 	mutex_lock(&mem_sysfs_mutex);
+
+	/*
+	 * Some users of the memory hotplug do not want/need memblock to
+	 * track all sections. Skip over those.
+	 */
 	mem = find_memory_block(section);
+	if (!mem)
+		goto out_unlock;
+
 	unregister_mem_sect_under_nodes(mem, __section_nr(section));
 
 	mem->section_count--;
@@ -754,6 +748,7 @@ static int remove_memory_section(unsigned long node_id,
 	else
 		put_device(&mem->dev);
 
+out_unlock:
 	mutex_unlock(&mem_sysfs_mutex);
 	return 0;
 }
diff --git a/include/linux/memory_hotplug.h b/include/linux/memory_hotplug.h
index 134a2f69c21a..3c8cf86201c3 100644
--- a/include/linux/memory_hotplug.h
+++ b/include/linux/memory_hotplug.h
@@ -111,7 +111,7 @@ extern int __remove_pages(struct zone *zone, unsigned long start_pfn,
 
 /* reasonably generic interface to expand the physical pages in a zone  */
 extern int __add_pages(int nid, struct zone *zone, unsigned long start_pfn,
-	unsigned long nr_pages);
+	unsigned long nr_pages, bool want_memblock);
 
 #ifdef CONFIG_NUMA
 extern int memory_add_physaddr_to_nid(u64 start);
diff --git a/mm/memory_hotplug.c b/mm/memory_hotplug.c
index 6290d34b6331..a95120c56a9a 100644
--- a/mm/memory_hotplug.c
+++ b/mm/memory_hotplug.c
@@ -494,7 +494,7 @@ static int __meminit __add_zone(struct zone *zone, unsigned long phys_start_pfn)
 }
 
 static int __meminit __add_section(int nid, struct zone *zone,
-					unsigned long phys_start_pfn)
+		unsigned long phys_start_pfn, bool want_memblock)
 {
 	int ret;
 
@@ -511,6 +511,9 @@ static int __meminit __add_section(int nid, struct zone *zone,
 	if (ret < 0)
 		return ret;
 
+	if (!want_memblock)
+		return 0;
+
 	return register_new_memory(nid, __pfn_to_section(phys_start_pfn));
 }
 
@@ -521,7 +524,7 @@ static int __meminit __add_section(int nid, struct zone *zone,
  * add the new pages.
  */
 int __ref __add_pages(int nid, struct zone *zone, unsigned long phys_start_pfn,
-			unsigned long nr_pages)
+			unsigned long nr_pages, bool want_memblock)
 {
 	unsigned long i;
 	int err = 0;
@@ -549,7 +552,7 @@ int __ref __add_pages(int nid, struct zone *zone, unsigned long phys_start_pfn,
 	}
 
 	for (i = start_sec; i <= end_sec; i++) {
-		err = __add_section(nid, zone, section_nr_to_pfn(i));
+		err = __add_section(nid, zone, section_nr_to_pfn(i), want_memblock);
 
 		/*
 		 * EEXIST is finally dealt with by ioresource collision
-- 
2.11.0

^ permalink raw reply	[flat|nested] 74+ messages in thread

* [PATCH 04/14] mm, memory_hotplug: get rid of is_zone_device_section
@ 2017-05-15  8:58   ` Michal Hocko
  0 siblings, 0 replies; 74+ messages in thread
From: Michal Hocko @ 2017-05-15  8:58 UTC (permalink / raw)
  To: Andrew Morton
  Cc: linux-mm, Mel Gorman, Vlastimil Babka, Andrea Arcangeli,
	Jerome Glisse, Reza Arbab, Yasuaki Ishimatsu, qiuxishi,
	Kani Toshimitsu, slaoub, Joonsoo Kim, Andi Kleen, David Rientjes,
	Daniel Kiper, Igor Mammedov, Vitaly Kuznetsov, LKML,
	Michal Hocko, Dan Williams

From: Michal Hocko <mhocko@suse.com>

device memory hotplug hooks into regular memory hotplug only half way.
It needs memory sections to track struct pages but there is no
need/desire to associate those sections with memory blocks and export
them to the userspace via sysfs because they cannot be onlined anyway.

This is currently expressed by for_device argument to arch_add_memory
which then makes sure to associate the given memory range with
ZONE_DEVICE. register_new_memory then relies on is_zone_device_section
to distinguish special memory hotplug from the regular one. While this
works now, later patches in this series want to move __add_zone outside
of arch_add_memory path so we have to come up with something else.

Add want_memblock down the __add_pages path and use it to control
whether the section->memblock association should be done. arch_add_memory
then just trivially want memblock for everything but for_device hotplug.

remove_memory_section doesn't need is_zone_device_section either. We can
simply skip all the memblock specific cleanup if there is no memblock
for the given section.

This shouldn't introduce any functional change.

Changes since v1
- return 0 if want_memblock == 0 from __add_section as per Jerome Glisse

Changes since v2
- fix remove_memory_section unlock on find_memory_block failure
  as per Jerome - spotted by Evgeny Baskakov

Tested-by: Dan Williams <dan.j.williams@intel.com>
Acked-by: Vlastimil Babka <vbabka@suse.cz>
Signed-off-by: Michal Hocko <mhocko@suse.com>
---
 arch/ia64/mm/init.c            |  2 +-
 arch/powerpc/mm/mem.c          |  2 +-
 arch/s390/mm/init.c            |  2 +-
 arch/sh/mm/init.c              |  2 +-
 arch/x86/mm/init_32.c          |  2 +-
 arch/x86/mm/init_64.c          |  2 +-
 drivers/base/memory.c          | 23 +++++++++--------------
 include/linux/memory_hotplug.h |  2 +-
 mm/memory_hotplug.c            |  9 ++++++---
 9 files changed, 22 insertions(+), 24 deletions(-)

diff --git a/arch/ia64/mm/init.c b/arch/ia64/mm/init.c
index 8f3efa682ee8..39e2aeb4669d 100644
--- a/arch/ia64/mm/init.c
+++ b/arch/ia64/mm/init.c
@@ -658,7 +658,7 @@ int arch_add_memory(int nid, u64 start, u64 size, bool for_device)
 
 	zone = pgdat->node_zones +
 		zone_for_memory(nid, start, size, ZONE_NORMAL, for_device);
-	ret = __add_pages(nid, zone, start_pfn, nr_pages);
+	ret = __add_pages(nid, zone, start_pfn, nr_pages, !for_device);
 
 	if (ret)
 		printk("%s: Problem encountered in __add_pages() as ret=%d\n",
diff --git a/arch/powerpc/mm/mem.c b/arch/powerpc/mm/mem.c
index 9ee536ec0739..e6b2e6618b6c 100644
--- a/arch/powerpc/mm/mem.c
+++ b/arch/powerpc/mm/mem.c
@@ -151,7 +151,7 @@ int arch_add_memory(int nid, u64 start, u64 size, bool for_device)
 	zone = pgdata->node_zones +
 		zone_for_memory(nid, start, size, 0, for_device);
 
-	return __add_pages(nid, zone, start_pfn, nr_pages);
+	return __add_pages(nid, zone, start_pfn, nr_pages, !for_device);
 }
 
 #ifdef CONFIG_MEMORY_HOTREMOVE
diff --git a/arch/s390/mm/init.c b/arch/s390/mm/init.c
index ee6a1d3d4983..893cf88cf02d 100644
--- a/arch/s390/mm/init.c
+++ b/arch/s390/mm/init.c
@@ -191,7 +191,7 @@ int arch_add_memory(int nid, u64 start, u64 size, bool for_device)
 			continue;
 		nr_pages = (start_pfn + size_pages > zone_end_pfn) ?
 			   zone_end_pfn - start_pfn : size_pages;
-		rc = __add_pages(nid, zone, start_pfn, nr_pages);
+		rc = __add_pages(nid, zone, start_pfn, nr_pages, !for_device);
 		if (rc)
 			break;
 		start_pfn += nr_pages;
diff --git a/arch/sh/mm/init.c b/arch/sh/mm/init.c
index 75491862d900..a9d57f75ae8c 100644
--- a/arch/sh/mm/init.c
+++ b/arch/sh/mm/init.c
@@ -498,7 +498,7 @@ int arch_add_memory(int nid, u64 start, u64 size, bool for_device)
 	ret = __add_pages(nid, pgdat->node_zones +
 			zone_for_memory(nid, start, size, ZONE_NORMAL,
 			for_device),
-			start_pfn, nr_pages);
+			start_pfn, nr_pages, !for_device);
 	if (unlikely(ret))
 		printk("%s: Failed, __add_pages() == %d\n", __func__, ret);
 
diff --git a/arch/x86/mm/init_32.c b/arch/x86/mm/init_32.c
index 99fb83819a5f..94594b889144 100644
--- a/arch/x86/mm/init_32.c
+++ b/arch/x86/mm/init_32.c
@@ -831,7 +831,7 @@ int arch_add_memory(int nid, u64 start, u64 size, bool for_device)
 	unsigned long start_pfn = start >> PAGE_SHIFT;
 	unsigned long nr_pages = size >> PAGE_SHIFT;
 
-	return __add_pages(nid, zone, start_pfn, nr_pages);
+	return __add_pages(nid, zone, start_pfn, nr_pages, !for_device);
 }
 
 #ifdef CONFIG_MEMORY_HOTREMOVE
diff --git a/arch/x86/mm/init_64.c b/arch/x86/mm/init_64.c
index 41270b96403d..2e004364a373 100644
--- a/arch/x86/mm/init_64.c
+++ b/arch/x86/mm/init_64.c
@@ -697,7 +697,7 @@ int arch_add_memory(int nid, u64 start, u64 size, bool for_device)
 
 	init_memory_mapping(start, start + size);
 
-	ret = __add_pages(nid, zone, start_pfn, nr_pages);
+	ret = __add_pages(nid, zone, start_pfn, nr_pages, !for_device);
 	WARN_ON_ONCE(ret);
 
 	/* update max_pfn, max_low_pfn and high_memory */
diff --git a/drivers/base/memory.c b/drivers/base/memory.c
index 90225ffee501..f8fd562c3f18 100644
--- a/drivers/base/memory.c
+++ b/drivers/base/memory.c
@@ -685,14 +685,6 @@ static int add_memory_block(int base_section_nr)
 	return 0;
 }
 
-static bool is_zone_device_section(struct mem_section *ms)
-{
-	struct page *page;
-
-	page = sparse_decode_mem_map(ms->section_mem_map, __section_nr(ms));
-	return is_zone_device_page(page);
-}
-
 /*
  * need an interface for the VM to add new memory regions,
  * but without onlining it.
@@ -702,9 +694,6 @@ int register_new_memory(int nid, struct mem_section *section)
 	int ret = 0;
 	struct memory_block *mem;
 
-	if (is_zone_device_section(section))
-		return 0;
-
 	mutex_lock(&mem_sysfs_mutex);
 
 	mem = find_memory_block(section);
@@ -741,11 +730,16 @@ static int remove_memory_section(unsigned long node_id,
 {
 	struct memory_block *mem;
 
-	if (is_zone_device_section(section))
-		return 0;
-
 	mutex_lock(&mem_sysfs_mutex);
+
+	/*
+	 * Some users of the memory hotplug do not want/need memblock to
+	 * track all sections. Skip over those.
+	 */
 	mem = find_memory_block(section);
+	if (!mem)
+		goto out_unlock;
+
 	unregister_mem_sect_under_nodes(mem, __section_nr(section));
 
 	mem->section_count--;
@@ -754,6 +748,7 @@ static int remove_memory_section(unsigned long node_id,
 	else
 		put_device(&mem->dev);
 
+out_unlock:
 	mutex_unlock(&mem_sysfs_mutex);
 	return 0;
 }
diff --git a/include/linux/memory_hotplug.h b/include/linux/memory_hotplug.h
index 134a2f69c21a..3c8cf86201c3 100644
--- a/include/linux/memory_hotplug.h
+++ b/include/linux/memory_hotplug.h
@@ -111,7 +111,7 @@ extern int __remove_pages(struct zone *zone, unsigned long start_pfn,
 
 /* reasonably generic interface to expand the physical pages in a zone  */
 extern int __add_pages(int nid, struct zone *zone, unsigned long start_pfn,
-	unsigned long nr_pages);
+	unsigned long nr_pages, bool want_memblock);
 
 #ifdef CONFIG_NUMA
 extern int memory_add_physaddr_to_nid(u64 start);
diff --git a/mm/memory_hotplug.c b/mm/memory_hotplug.c
index 6290d34b6331..a95120c56a9a 100644
--- a/mm/memory_hotplug.c
+++ b/mm/memory_hotplug.c
@@ -494,7 +494,7 @@ static int __meminit __add_zone(struct zone *zone, unsigned long phys_start_pfn)
 }
 
 static int __meminit __add_section(int nid, struct zone *zone,
-					unsigned long phys_start_pfn)
+		unsigned long phys_start_pfn, bool want_memblock)
 {
 	int ret;
 
@@ -511,6 +511,9 @@ static int __meminit __add_section(int nid, struct zone *zone,
 	if (ret < 0)
 		return ret;
 
+	if (!want_memblock)
+		return 0;
+
 	return register_new_memory(nid, __pfn_to_section(phys_start_pfn));
 }
 
@@ -521,7 +524,7 @@ static int __meminit __add_section(int nid, struct zone *zone,
  * add the new pages.
  */
 int __ref __add_pages(int nid, struct zone *zone, unsigned long phys_start_pfn,
-			unsigned long nr_pages)
+			unsigned long nr_pages, bool want_memblock)
 {
 	unsigned long i;
 	int err = 0;
@@ -549,7 +552,7 @@ int __ref __add_pages(int nid, struct zone *zone, unsigned long phys_start_pfn,
 	}
 
 	for (i = start_sec; i <= end_sec; i++) {
-		err = __add_section(nid, zone, section_nr_to_pfn(i));
+		err = __add_section(nid, zone, section_nr_to_pfn(i), want_memblock);
 
 		/*
 		 * EEXIST is finally dealt with by ioresource collision
-- 
2.11.0

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

^ permalink raw reply	[flat|nested] 74+ messages in thread

* [PATCH 05/14] mm, memory_hotplug: split up register_one_node
  2017-05-15  8:58 ` Michal Hocko
@ 2017-05-15  8:58   ` Michal Hocko
  -1 siblings, 0 replies; 74+ messages in thread
From: Michal Hocko @ 2017-05-15  8:58 UTC (permalink / raw)
  To: Andrew Morton
  Cc: linux-mm, Mel Gorman, Vlastimil Babka, Andrea Arcangeli,
	Jerome Glisse, Reza Arbab, Yasuaki Ishimatsu, qiuxishi,
	Kani Toshimitsu, slaoub, Joonsoo Kim, Andi Kleen, David Rientjes,
	Daniel Kiper, Igor Mammedov, Vitaly Kuznetsov, LKML,
	Michal Hocko

From: Michal Hocko <mhocko@suse.com>

Memory hotplug (add_memory_resource) has to reinitialize node
infrastructure if the node is offline (one which went through the
complete add_memory(); remove_memory() cycle). That involves node
registration to the kobj infrastructure (register_node), the proper
association with cpus (register_cpu_under_node) and finally creation of
node<->memblock symlinks (link_mem_sections).

The last part requires to know node_start_pfn and node_spanned_pages
which we currently have but a leter patch will postpone this
initialization to the onlining phase which happens later. In fact we do
not need to rely on the early pgdat initialization even now because the
currently hot added pfn range is currently known.

Split register_one_node into core which does all the common work for
the boot time NUMA initialization and the hotplug (__register_one_node).
register_one_node keeps the full initialization while hotplug calls
__register_one_node and manually calls link_mem_sections for the proper
range.

This shouldn't introduce any functional change.

Acked-by: Vlastimil Babka <vbabka@suse.cz>
Signed-off-by: Michal Hocko <mhocko@suse.com>
---
 drivers/base/node.c  | 51 ++++++++++++++++++++-------------------------------
 include/linux/node.h | 35 ++++++++++++++++++++++++++++++++++-
 mm/memory_hotplug.c  | 17 ++++++++++++++++-
 3 files changed, 70 insertions(+), 33 deletions(-)

diff --git a/drivers/base/node.c b/drivers/base/node.c
index 06294d69779b..dff5b53f7905 100644
--- a/drivers/base/node.c
+++ b/drivers/base/node.c
@@ -461,10 +461,9 @@ int unregister_mem_sect_under_nodes(struct memory_block *mem_blk,
 	return 0;
 }
 
-static int link_mem_sections(int nid)
+int link_mem_sections(int nid, unsigned long start_pfn, unsigned long nr_pages)
 {
-	unsigned long start_pfn = NODE_DATA(nid)->node_start_pfn;
-	unsigned long end_pfn = start_pfn + NODE_DATA(nid)->node_spanned_pages;
+	unsigned long end_pfn = start_pfn + nr_pages;
 	unsigned long pfn;
 	struct memory_block *mem_blk = NULL;
 	int err = 0;
@@ -552,10 +551,7 @@ static int node_memory_callback(struct notifier_block *self,
 	return NOTIFY_OK;
 }
 #endif	/* CONFIG_HUGETLBFS */
-#else	/* !CONFIG_MEMORY_HOTPLUG_SPARSE */
-
-static int link_mem_sections(int nid) { return 0; }
-#endif	/* CONFIG_MEMORY_HOTPLUG_SPARSE */
+#endif /* CONFIG_MEMORY_HOTPLUG_SPARSE */
 
 #if !defined(CONFIG_MEMORY_HOTPLUG_SPARSE) || \
     !defined(CONFIG_HUGETLBFS)
@@ -569,39 +565,32 @@ static void init_node_hugetlb_work(int nid) { }
 
 #endif
 
-int register_one_node(int nid)
+int __register_one_node(int nid)
 {
-	int error = 0;
+	int p_node = parent_node(nid);
+	struct node *parent = NULL;
+	int error;
 	int cpu;
 
-	if (node_online(nid)) {
-		int p_node = parent_node(nid);
-		struct node *parent = NULL;
-
-		if (p_node != nid)
-			parent = node_devices[p_node];
-
-		node_devices[nid] = kzalloc(sizeof(struct node), GFP_KERNEL);
-		if (!node_devices[nid])
-			return -ENOMEM;
-
-		error = register_node(node_devices[nid], nid, parent);
+	if (p_node != nid)
+		parent = node_devices[p_node];
 
-		/* link cpu under this node */
-		for_each_present_cpu(cpu) {
-			if (cpu_to_node(cpu) == nid)
-				register_cpu_under_node(cpu, nid);
-		}
+	node_devices[nid] = kzalloc(sizeof(struct node), GFP_KERNEL);
+	if (!node_devices[nid])
+		return -ENOMEM;
 
-		/* link memory sections under this node */
-		error = link_mem_sections(nid);
+	error = register_node(node_devices[nid], nid, parent);
 
-		/* initialize work queue for memory hot plug */
-		init_node_hugetlb_work(nid);
+	/* link cpu under this node */
+	for_each_present_cpu(cpu) {
+		if (cpu_to_node(cpu) == nid)
+			register_cpu_under_node(cpu, nid);
 	}
 
-	return error;
+	/* initialize work queue for memory hot plug */
+	init_node_hugetlb_work(nid);
 
+	return error;
 }
 
 void unregister_one_node(int nid)
diff --git a/include/linux/node.h b/include/linux/node.h
index 2115ad5d6f19..d1751beb462c 100644
--- a/include/linux/node.h
+++ b/include/linux/node.h
@@ -30,9 +30,38 @@ struct memory_block;
 extern struct node *node_devices[];
 typedef  void (*node_registration_func_t)(struct node *);
 
+#if defined(CONFIG_MEMORY_HOTPLUG_SPARSE) && defined(CONFIG_NUMA)
+extern int link_mem_sections(int nid, unsigned long start_pfn, unsigned long nr_pages);
+#else
+static inline int link_mem_sections(int nid, unsigned long start_pfn, unsigned long nr_pages)
+{
+	return 0;
+}
+#endif
+
 extern void unregister_node(struct node *node);
 #ifdef CONFIG_NUMA
-extern int register_one_node(int nid);
+/* Core of the node registration - only memory hotplug should use this */
+extern int __register_one_node(int nid);
+
+/* Registers an online node */
+static inline int register_one_node(int nid)
+{
+	int error = 0;
+
+	if (node_online(nid)) {
+		struct pglist_data *pgdat = NODE_DATA(nid);
+
+		error = __register_one_node(nid);
+		if (error)
+			return error;
+		/* link memory sections under this node */
+		error = link_mem_sections(nid, pgdat->node_start_pfn, pgdat->node_spanned_pages);
+	}
+
+	return error;
+}
+
 extern void unregister_one_node(int nid);
 extern int register_cpu_under_node(unsigned int cpu, unsigned int nid);
 extern int unregister_cpu_under_node(unsigned int cpu, unsigned int nid);
@@ -46,6 +75,10 @@ extern void register_hugetlbfs_with_node(node_registration_func_t doregister,
 					 node_registration_func_t unregister);
 #endif
 #else
+static inline int __register_one_node(int nid)
+{
+	return 0;
+}
 static inline int register_one_node(int nid)
 {
 	return 0;
diff --git a/mm/memory_hotplug.c b/mm/memory_hotplug.c
index a95120c56a9a..05796ee974f7 100644
--- a/mm/memory_hotplug.c
+++ b/mm/memory_hotplug.c
@@ -1392,7 +1392,22 @@ int __ref add_memory_resource(int nid, struct resource *res, bool online)
 	node_set_online(nid);
 
 	if (new_node) {
-		ret = register_one_node(nid);
+		unsigned long start_pfn = start >> PAGE_SHIFT;
+		unsigned long nr_pages = size >> PAGE_SHIFT;
+
+		ret = __register_one_node(nid);
+		if (ret)
+			goto register_fail;
+
+		/*
+		 * link memory sections under this node. This is already
+		 * done when creatig memory section in register_new_memory
+		 * but that depends to have the node registered so offline
+		 * nodes have to go through register_node.
+		 * TODO clean up this mess.
+		 */
+		ret = link_mem_sections(nid, start_pfn, nr_pages);
+register_fail:
 		/*
 		 * If sysfs file of new node can't create, cpu on the node
 		 * can't be hot-added. There is no rollback way now.
-- 
2.11.0

^ permalink raw reply	[flat|nested] 74+ messages in thread

* [PATCH 05/14] mm, memory_hotplug: split up register_one_node
@ 2017-05-15  8:58   ` Michal Hocko
  0 siblings, 0 replies; 74+ messages in thread
From: Michal Hocko @ 2017-05-15  8:58 UTC (permalink / raw)
  To: Andrew Morton
  Cc: linux-mm, Mel Gorman, Vlastimil Babka, Andrea Arcangeli,
	Jerome Glisse, Reza Arbab, Yasuaki Ishimatsu, qiuxishi,
	Kani Toshimitsu, slaoub, Joonsoo Kim, Andi Kleen, David Rientjes,
	Daniel Kiper, Igor Mammedov, Vitaly Kuznetsov, LKML,
	Michal Hocko

From: Michal Hocko <mhocko@suse.com>

Memory hotplug (add_memory_resource) has to reinitialize node
infrastructure if the node is offline (one which went through the
complete add_memory(); remove_memory() cycle). That involves node
registration to the kobj infrastructure (register_node), the proper
association with cpus (register_cpu_under_node) and finally creation of
node<->memblock symlinks (link_mem_sections).

The last part requires to know node_start_pfn and node_spanned_pages
which we currently have but a leter patch will postpone this
initialization to the onlining phase which happens later. In fact we do
not need to rely on the early pgdat initialization even now because the
currently hot added pfn range is currently known.

Split register_one_node into core which does all the common work for
the boot time NUMA initialization and the hotplug (__register_one_node).
register_one_node keeps the full initialization while hotplug calls
__register_one_node and manually calls link_mem_sections for the proper
range.

This shouldn't introduce any functional change.

Acked-by: Vlastimil Babka <vbabka@suse.cz>
Signed-off-by: Michal Hocko <mhocko@suse.com>
---
 drivers/base/node.c  | 51 ++++++++++++++++++++-------------------------------
 include/linux/node.h | 35 ++++++++++++++++++++++++++++++++++-
 mm/memory_hotplug.c  | 17 ++++++++++++++++-
 3 files changed, 70 insertions(+), 33 deletions(-)

diff --git a/drivers/base/node.c b/drivers/base/node.c
index 06294d69779b..dff5b53f7905 100644
--- a/drivers/base/node.c
+++ b/drivers/base/node.c
@@ -461,10 +461,9 @@ int unregister_mem_sect_under_nodes(struct memory_block *mem_blk,
 	return 0;
 }
 
-static int link_mem_sections(int nid)
+int link_mem_sections(int nid, unsigned long start_pfn, unsigned long nr_pages)
 {
-	unsigned long start_pfn = NODE_DATA(nid)->node_start_pfn;
-	unsigned long end_pfn = start_pfn + NODE_DATA(nid)->node_spanned_pages;
+	unsigned long end_pfn = start_pfn + nr_pages;
 	unsigned long pfn;
 	struct memory_block *mem_blk = NULL;
 	int err = 0;
@@ -552,10 +551,7 @@ static int node_memory_callback(struct notifier_block *self,
 	return NOTIFY_OK;
 }
 #endif	/* CONFIG_HUGETLBFS */
-#else	/* !CONFIG_MEMORY_HOTPLUG_SPARSE */
-
-static int link_mem_sections(int nid) { return 0; }
-#endif	/* CONFIG_MEMORY_HOTPLUG_SPARSE */
+#endif /* CONFIG_MEMORY_HOTPLUG_SPARSE */
 
 #if !defined(CONFIG_MEMORY_HOTPLUG_SPARSE) || \
     !defined(CONFIG_HUGETLBFS)
@@ -569,39 +565,32 @@ static void init_node_hugetlb_work(int nid) { }
 
 #endif
 
-int register_one_node(int nid)
+int __register_one_node(int nid)
 {
-	int error = 0;
+	int p_node = parent_node(nid);
+	struct node *parent = NULL;
+	int error;
 	int cpu;
 
-	if (node_online(nid)) {
-		int p_node = parent_node(nid);
-		struct node *parent = NULL;
-
-		if (p_node != nid)
-			parent = node_devices[p_node];
-
-		node_devices[nid] = kzalloc(sizeof(struct node), GFP_KERNEL);
-		if (!node_devices[nid])
-			return -ENOMEM;
-
-		error = register_node(node_devices[nid], nid, parent);
+	if (p_node != nid)
+		parent = node_devices[p_node];
 
-		/* link cpu under this node */
-		for_each_present_cpu(cpu) {
-			if (cpu_to_node(cpu) == nid)
-				register_cpu_under_node(cpu, nid);
-		}
+	node_devices[nid] = kzalloc(sizeof(struct node), GFP_KERNEL);
+	if (!node_devices[nid])
+		return -ENOMEM;
 
-		/* link memory sections under this node */
-		error = link_mem_sections(nid);
+	error = register_node(node_devices[nid], nid, parent);
 
-		/* initialize work queue for memory hot plug */
-		init_node_hugetlb_work(nid);
+	/* link cpu under this node */
+	for_each_present_cpu(cpu) {
+		if (cpu_to_node(cpu) == nid)
+			register_cpu_under_node(cpu, nid);
 	}
 
-	return error;
+	/* initialize work queue for memory hot plug */
+	init_node_hugetlb_work(nid);
 
+	return error;
 }
 
 void unregister_one_node(int nid)
diff --git a/include/linux/node.h b/include/linux/node.h
index 2115ad5d6f19..d1751beb462c 100644
--- a/include/linux/node.h
+++ b/include/linux/node.h
@@ -30,9 +30,38 @@ struct memory_block;
 extern struct node *node_devices[];
 typedef  void (*node_registration_func_t)(struct node *);
 
+#if defined(CONFIG_MEMORY_HOTPLUG_SPARSE) && defined(CONFIG_NUMA)
+extern int link_mem_sections(int nid, unsigned long start_pfn, unsigned long nr_pages);
+#else
+static inline int link_mem_sections(int nid, unsigned long start_pfn, unsigned long nr_pages)
+{
+	return 0;
+}
+#endif
+
 extern void unregister_node(struct node *node);
 #ifdef CONFIG_NUMA
-extern int register_one_node(int nid);
+/* Core of the node registration - only memory hotplug should use this */
+extern int __register_one_node(int nid);
+
+/* Registers an online node */
+static inline int register_one_node(int nid)
+{
+	int error = 0;
+
+	if (node_online(nid)) {
+		struct pglist_data *pgdat = NODE_DATA(nid);
+
+		error = __register_one_node(nid);
+		if (error)
+			return error;
+		/* link memory sections under this node */
+		error = link_mem_sections(nid, pgdat->node_start_pfn, pgdat->node_spanned_pages);
+	}
+
+	return error;
+}
+
 extern void unregister_one_node(int nid);
 extern int register_cpu_under_node(unsigned int cpu, unsigned int nid);
 extern int unregister_cpu_under_node(unsigned int cpu, unsigned int nid);
@@ -46,6 +75,10 @@ extern void register_hugetlbfs_with_node(node_registration_func_t doregister,
 					 node_registration_func_t unregister);
 #endif
 #else
+static inline int __register_one_node(int nid)
+{
+	return 0;
+}
 static inline int register_one_node(int nid)
 {
 	return 0;
diff --git a/mm/memory_hotplug.c b/mm/memory_hotplug.c
index a95120c56a9a..05796ee974f7 100644
--- a/mm/memory_hotplug.c
+++ b/mm/memory_hotplug.c
@@ -1392,7 +1392,22 @@ int __ref add_memory_resource(int nid, struct resource *res, bool online)
 	node_set_online(nid);
 
 	if (new_node) {
-		ret = register_one_node(nid);
+		unsigned long start_pfn = start >> PAGE_SHIFT;
+		unsigned long nr_pages = size >> PAGE_SHIFT;
+
+		ret = __register_one_node(nid);
+		if (ret)
+			goto register_fail;
+
+		/*
+		 * link memory sections under this node. This is already
+		 * done when creatig memory section in register_new_memory
+		 * but that depends to have the node registered so offline
+		 * nodes have to go through register_node.
+		 * TODO clean up this mess.
+		 */
+		ret = link_mem_sections(nid, start_pfn, nr_pages);
+register_fail:
 		/*
 		 * If sysfs file of new node can't create, cpu on the node
 		 * can't be hot-added. There is no rollback way now.
-- 
2.11.0

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

^ permalink raw reply	[flat|nested] 74+ messages in thread

* [PATCH 06/14] mm, memory_hotplug: consider offline memblocks removable
  2017-05-15  8:58 ` Michal Hocko
@ 2017-05-15  8:58   ` Michal Hocko
  -1 siblings, 0 replies; 74+ messages in thread
From: Michal Hocko @ 2017-05-15  8:58 UTC (permalink / raw)
  To: Andrew Morton
  Cc: linux-mm, Mel Gorman, Vlastimil Babka, Andrea Arcangeli,
	Jerome Glisse, Reza Arbab, Yasuaki Ishimatsu, qiuxishi,
	Kani Toshimitsu, slaoub, Joonsoo Kim, Andi Kleen, David Rientjes,
	Daniel Kiper, Igor Mammedov, Vitaly Kuznetsov, LKML,
	Michal Hocko

From: Michal Hocko <mhocko@suse.com>

is_pageblock_removable_nolock relies on having zone association to
examine all the page blocks to check whether they are movable or free.
This is just wasting of cycles when the memblock is offline. Later patch
in the series will also change the time when the page is associated with
a zone so we let's bail out early if the memblock is offline.

Reported-by: Igor Mammedov <imammedo@redhat.com>
Acked-by: Vlastimil Babka <vbabka@suse.cz>
Signed-off-by: Michal Hocko <mhocko@suse.com>
---
 drivers/base/memory.c | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/drivers/base/memory.c b/drivers/base/memory.c
index f8fd562c3f18..1e884d82af6f 100644
--- a/drivers/base/memory.c
+++ b/drivers/base/memory.c
@@ -128,6 +128,9 @@ static ssize_t show_mem_removable(struct device *dev,
 	int ret = 1;
 	struct memory_block *mem = to_memory_block(dev);
 
+	if (mem->state != MEM_ONLINE)
+		goto out;
+
 	for (i = 0; i < sections_per_block; i++) {
 		if (!present_section_nr(mem->start_section_nr + i))
 			continue;
@@ -135,6 +138,7 @@ static ssize_t show_mem_removable(struct device *dev,
 		ret &= is_mem_section_removable(pfn, PAGES_PER_SECTION);
 	}
 
+out:
 	return sprintf(buf, "%d\n", ret);
 }
 
-- 
2.11.0

^ permalink raw reply	[flat|nested] 74+ messages in thread

* [PATCH 06/14] mm, memory_hotplug: consider offline memblocks removable
@ 2017-05-15  8:58   ` Michal Hocko
  0 siblings, 0 replies; 74+ messages in thread
From: Michal Hocko @ 2017-05-15  8:58 UTC (permalink / raw)
  To: Andrew Morton
  Cc: linux-mm, Mel Gorman, Vlastimil Babka, Andrea Arcangeli,
	Jerome Glisse, Reza Arbab, Yasuaki Ishimatsu, qiuxishi,
	Kani Toshimitsu, slaoub, Joonsoo Kim, Andi Kleen, David Rientjes,
	Daniel Kiper, Igor Mammedov, Vitaly Kuznetsov, LKML,
	Michal Hocko

From: Michal Hocko <mhocko@suse.com>

is_pageblock_removable_nolock relies on having zone association to
examine all the page blocks to check whether they are movable or free.
This is just wasting of cycles when the memblock is offline. Later patch
in the series will also change the time when the page is associated with
a zone so we let's bail out early if the memblock is offline.

Reported-by: Igor Mammedov <imammedo@redhat.com>
Acked-by: Vlastimil Babka <vbabka@suse.cz>
Signed-off-by: Michal Hocko <mhocko@suse.com>
---
 drivers/base/memory.c | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/drivers/base/memory.c b/drivers/base/memory.c
index f8fd562c3f18..1e884d82af6f 100644
--- a/drivers/base/memory.c
+++ b/drivers/base/memory.c
@@ -128,6 +128,9 @@ static ssize_t show_mem_removable(struct device *dev,
 	int ret = 1;
 	struct memory_block *mem = to_memory_block(dev);
 
+	if (mem->state != MEM_ONLINE)
+		goto out;
+
 	for (i = 0; i < sections_per_block; i++) {
 		if (!present_section_nr(mem->start_section_nr + i))
 			continue;
@@ -135,6 +138,7 @@ static ssize_t show_mem_removable(struct device *dev,
 		ret &= is_mem_section_removable(pfn, PAGES_PER_SECTION);
 	}
 
+out:
 	return sprintf(buf, "%d\n", ret);
 }
 
-- 
2.11.0

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

^ permalink raw reply	[flat|nested] 74+ messages in thread

* [PATCH 07/14] mm: consider zone which is not fully populated to have holes
  2017-05-15  8:58 ` Michal Hocko
@ 2017-05-15  8:58   ` Michal Hocko
  -1 siblings, 0 replies; 74+ messages in thread
From: Michal Hocko @ 2017-05-15  8:58 UTC (permalink / raw)
  To: Andrew Morton
  Cc: linux-mm, Mel Gorman, Vlastimil Babka, Andrea Arcangeli,
	Jerome Glisse, Reza Arbab, Yasuaki Ishimatsu, qiuxishi,
	Kani Toshimitsu, slaoub, Joonsoo Kim, Andi Kleen, David Rientjes,
	Daniel Kiper, Igor Mammedov, Vitaly Kuznetsov, LKML,
	Michal Hocko

From: Michal Hocko <mhocko@suse.com>

__pageblock_pfn_to_page has two users currently, set_zone_contiguous
which checks whether the given zone contains holes and
pageblock_pfn_to_page which then carefully returns a first valid
page from the given pfn range for the given zone. This doesn't handle
zones which are not fully populated though. Memory pageblocks can be
offlined or might not have been onlined yet. In such a case the zone
should be considered to have holes otherwise pfn walkers can touch
and play with offline pages.

Current callers of pageblock_pfn_to_page in compaction seem to work
properly right now because they only isolate PageBuddy
(isolate_freepages_block) or PageLRU resp. __PageMovable
(isolate_migratepages_block) which will be always false for these pages.
It would be safer to skip these pages altogether, though.

In order to do this patch adds a new memory section state
(SECTION_IS_ONLINE) which is set in memory_present (during boot
time) or in online_pages_range during the memory hotplug. Similarly
offline_mem_sections clears the bit and it is called when the memory
range is offlined.

pfn_to_online_page helper is then added which check the mem section and
only returns a page if it is onlined already.

Use the new helper in __pageblock_pfn_to_page and skip the whole page
block in such a case.

Changes since v3
- clarify pfn_valid semantic - requested by Joonsoo

Signed-off-by: Michal Hocko <mhocko@suse.com>
---
 include/linux/memory_hotplug.h | 21 ++++++++++++++++++++
 include/linux/mmzone.h         | 35 ++++++++++++++++++++++++++------
 mm/memory_hotplug.c            |  3 +++
 mm/page_alloc.c                |  5 ++++-
 mm/sparse.c                    | 45 +++++++++++++++++++++++++++++++++++++++++-
 5 files changed, 101 insertions(+), 8 deletions(-)

diff --git a/include/linux/memory_hotplug.h b/include/linux/memory_hotplug.h
index 3c8cf86201c3..fc1c873504eb 100644
--- a/include/linux/memory_hotplug.h
+++ b/include/linux/memory_hotplug.h
@@ -14,6 +14,19 @@ struct memory_block;
 struct resource;
 
 #ifdef CONFIG_MEMORY_HOTPLUG
+/*
+ * Return page for the valid pfn only if the page is online. All pfn
+ * walkers which rely on the fully initialized page->flags and others
+ * should use this rather than pfn_valid && pfn_to_page
+ */
+#define pfn_to_online_page(pfn)				\
+({							\
+	struct page *___page = NULL;			\
+							\
+	if (online_section_nr(pfn_to_section_nr(pfn)))	\
+		___page = pfn_to_page(pfn);		\
+	___page;					\
+})
 
 /*
  * Types for free bootmem stored in page->lru.next. These have to be in
@@ -203,6 +216,14 @@ extern void set_zone_contiguous(struct zone *zone);
 extern void clear_zone_contiguous(struct zone *zone);
 
 #else /* ! CONFIG_MEMORY_HOTPLUG */
+#define pfn_to_online_page(pfn)			\
+({						\
+	struct page *___page = NULL;		\
+	if (pfn_valid(pfn))			\
+		___page = pfn_to_page(pfn);	\
+	___page;				\
+ })
+
 /*
  * Stub functions for when hotplug is off
  */
diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h
index 8363dd27b8af..927ad95a4552 100644
--- a/include/linux/mmzone.h
+++ b/include/linux/mmzone.h
@@ -1143,9 +1143,10 @@ extern unsigned long usemap_size(void);
  */
 #define	SECTION_MARKED_PRESENT	(1UL<<0)
 #define SECTION_HAS_MEM_MAP	(1UL<<1)
-#define SECTION_MAP_LAST_BIT	(1UL<<2)
+#define SECTION_IS_ONLINE	(1UL<<2)
+#define SECTION_MAP_LAST_BIT	(1UL<<3)
 #define SECTION_MAP_MASK	(~(SECTION_MAP_LAST_BIT-1))
-#define SECTION_NID_SHIFT	2
+#define SECTION_NID_SHIFT	3
 
 static inline struct page *__section_mem_map_addr(struct mem_section *section)
 {
@@ -1174,6 +1175,23 @@ static inline int valid_section_nr(unsigned long nr)
 	return valid_section(__nr_to_section(nr));
 }
 
+static inline int online_section(struct mem_section *section)
+{
+	return (section && (section->section_mem_map & SECTION_IS_ONLINE));
+}
+
+static inline int online_section_nr(unsigned long nr)
+{
+	return online_section(__nr_to_section(nr));
+}
+
+#ifdef CONFIG_MEMORY_HOTPLUG
+void online_mem_sections(unsigned long start_pfn, unsigned long end_pfn);
+#ifdef CONFIG_MEMORY_HOTREMOVE
+void offline_mem_sections(unsigned long start_pfn, unsigned long end_pfn);
+#endif
+#endif
+
 static inline struct mem_section *__pfn_to_section(unsigned long pfn)
 {
 	return __nr_to_section(pfn_to_section_nr(pfn));
@@ -1252,10 +1270,15 @@ unsigned long __init node_memmap_size_bytes(int, unsigned long, unsigned long);
 #ifdef CONFIG_ARCH_HAS_HOLES_MEMORYMODEL
 /*
  * pfn_valid() is meant to be able to tell if a given PFN has valid memmap
- * associated with it or not. In FLATMEM, it is expected that holes always
- * have valid memmap as long as there is valid PFNs either side of the hole.
- * In SPARSEMEM, it is assumed that a valid section has a memmap for the
- * entire section.
+ * associated with it or not. This means that a struct page exists for this
+ * pfn. The caller cannot assume the page is fully initialized in general.
+ * Hotplugable pages might not have been onlined yet. pfn_to_online_page()
+ * will ensure the struct page is fully online and initialized. Special pages
+ * (e.g. ZONE_DEVICE) are never onlined and should be treated accordingly.
+ *
+ * In FLATMEM, it is expected that holes always have valid memmap as long as
+ * there is valid PFNs either side of the hole. In SPARSEMEM, it is assumed
+ * that a valid section has a memmap for the entire section.
  *
  * However, an ARM, and maybe other embedded architectures in the future
  * free memmap backing holes to save memory on the assumption the memmap is
diff --git a/mm/memory_hotplug.c b/mm/memory_hotplug.c
index 05796ee974f7..c3a146028ba6 100644
--- a/mm/memory_hotplug.c
+++ b/mm/memory_hotplug.c
@@ -929,6 +929,9 @@ static int online_pages_range(unsigned long start_pfn, unsigned long nr_pages,
 	unsigned long i;
 	unsigned long onlined_pages = *(unsigned long *)arg;
 	struct page *page;
+
+	online_mem_sections(start_pfn, start_pfn + nr_pages);
+
 	if (PageReserved(pfn_to_page(start_pfn)))
 		for (i = 0; i < nr_pages; i++) {
 			page = pfn_to_page(start_pfn + i);
diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index c1670f090107..7e5151a7dd7b 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -1353,7 +1353,9 @@ struct page *__pageblock_pfn_to_page(unsigned long start_pfn,
 	if (!pfn_valid(start_pfn) || !pfn_valid(end_pfn))
 		return NULL;
 
-	start_page = pfn_to_page(start_pfn);
+	start_page = pfn_to_online_page(start_pfn);
+	if (!start_page)
+		return NULL;
 
 	if (page_zone(start_page) != zone)
 		return NULL;
@@ -7671,6 +7673,7 @@ __offline_isolated_pages(unsigned long start_pfn, unsigned long end_pfn)
 			break;
 	if (pfn == end_pfn)
 		return;
+	offline_mem_sections(pfn, end_pfn);
 	zone = page_zone(pfn_to_page(pfn));
 	spin_lock_irqsave(&zone->lock, flags);
 	pfn = start_pfn;
diff --git a/mm/sparse.c b/mm/sparse.c
index 5032c9a619de..9d7fd666015e 100644
--- a/mm/sparse.c
+++ b/mm/sparse.c
@@ -222,7 +222,8 @@ void __init memory_present(int nid, unsigned long start, unsigned long end)
 
 		ms = __nr_to_section(section);
 		if (!ms->section_mem_map) {
-			ms->section_mem_map = sparse_encode_early_nid(nid);
+			ms->section_mem_map = sparse_encode_early_nid(nid) |
+							SECTION_IS_ONLINE;
 			section_mark_present(ms);
 		}
 	}
@@ -622,6 +623,48 @@ void __init sparse_init(void)
 }
 
 #ifdef CONFIG_MEMORY_HOTPLUG
+
+/* Mark all memory sections within the pfn range as online */
+void online_mem_sections(unsigned long start_pfn, unsigned long end_pfn)
+{
+	unsigned long pfn;
+
+	for (pfn = start_pfn; pfn < end_pfn; pfn += PAGES_PER_SECTION) {
+		unsigned long section_nr = pfn_to_section_nr(start_pfn);
+		struct mem_section *ms;
+
+		/* onlining code should never touch invalid ranges */
+		if (WARN_ON(!valid_section_nr(section_nr)))
+			continue;
+
+		ms = __nr_to_section(section_nr);
+		ms->section_mem_map |= SECTION_IS_ONLINE;
+	}
+}
+
+#ifdef CONFIG_MEMORY_HOTREMOVE
+/* Mark all memory sections within the pfn range as online */
+void offline_mem_sections(unsigned long start_pfn, unsigned long end_pfn)
+{
+	unsigned long pfn;
+
+	for (pfn = start_pfn; pfn < end_pfn; pfn += PAGES_PER_SECTION) {
+		unsigned long section_nr = pfn_to_section_nr(start_pfn);
+		struct mem_section *ms;
+
+		/*
+		 * TODO this needs some double checking. Offlining code makes
+		 * sure to check pfn_valid but those checks might be just bogus
+		 */
+		if (WARN_ON(!valid_section_nr(section_nr)))
+			continue;
+
+		ms = __nr_to_section(section_nr);
+		ms->section_mem_map &= ~SECTION_IS_ONLINE;
+	}
+}
+#endif
+
 #ifdef CONFIG_SPARSEMEM_VMEMMAP
 static inline struct page *kmalloc_section_memmap(unsigned long pnum, int nid)
 {
-- 
2.11.0

^ permalink raw reply	[flat|nested] 74+ messages in thread

* [PATCH 07/14] mm: consider zone which is not fully populated to have holes
@ 2017-05-15  8:58   ` Michal Hocko
  0 siblings, 0 replies; 74+ messages in thread
From: Michal Hocko @ 2017-05-15  8:58 UTC (permalink / raw)
  To: Andrew Morton
  Cc: linux-mm, Mel Gorman, Vlastimil Babka, Andrea Arcangeli,
	Jerome Glisse, Reza Arbab, Yasuaki Ishimatsu, qiuxishi,
	Kani Toshimitsu, slaoub, Joonsoo Kim, Andi Kleen, David Rientjes,
	Daniel Kiper, Igor Mammedov, Vitaly Kuznetsov, LKML,
	Michal Hocko

From: Michal Hocko <mhocko@suse.com>

__pageblock_pfn_to_page has two users currently, set_zone_contiguous
which checks whether the given zone contains holes and
pageblock_pfn_to_page which then carefully returns a first valid
page from the given pfn range for the given zone. This doesn't handle
zones which are not fully populated though. Memory pageblocks can be
offlined or might not have been onlined yet. In such a case the zone
should be considered to have holes otherwise pfn walkers can touch
and play with offline pages.

Current callers of pageblock_pfn_to_page in compaction seem to work
properly right now because they only isolate PageBuddy
(isolate_freepages_block) or PageLRU resp. __PageMovable
(isolate_migratepages_block) which will be always false for these pages.
It would be safer to skip these pages altogether, though.

In order to do this patch adds a new memory section state
(SECTION_IS_ONLINE) which is set in memory_present (during boot
time) or in online_pages_range during the memory hotplug. Similarly
offline_mem_sections clears the bit and it is called when the memory
range is offlined.

pfn_to_online_page helper is then added which check the mem section and
only returns a page if it is onlined already.

Use the new helper in __pageblock_pfn_to_page and skip the whole page
block in such a case.

Changes since v3
- clarify pfn_valid semantic - requested by Joonsoo

Signed-off-by: Michal Hocko <mhocko@suse.com>
---
 include/linux/memory_hotplug.h | 21 ++++++++++++++++++++
 include/linux/mmzone.h         | 35 ++++++++++++++++++++++++++------
 mm/memory_hotplug.c            |  3 +++
 mm/page_alloc.c                |  5 ++++-
 mm/sparse.c                    | 45 +++++++++++++++++++++++++++++++++++++++++-
 5 files changed, 101 insertions(+), 8 deletions(-)

diff --git a/include/linux/memory_hotplug.h b/include/linux/memory_hotplug.h
index 3c8cf86201c3..fc1c873504eb 100644
--- a/include/linux/memory_hotplug.h
+++ b/include/linux/memory_hotplug.h
@@ -14,6 +14,19 @@ struct memory_block;
 struct resource;
 
 #ifdef CONFIG_MEMORY_HOTPLUG
+/*
+ * Return page for the valid pfn only if the page is online. All pfn
+ * walkers which rely on the fully initialized page->flags and others
+ * should use this rather than pfn_valid && pfn_to_page
+ */
+#define pfn_to_online_page(pfn)				\
+({							\
+	struct page *___page = NULL;			\
+							\
+	if (online_section_nr(pfn_to_section_nr(pfn)))	\
+		___page = pfn_to_page(pfn);		\
+	___page;					\
+})
 
 /*
  * Types for free bootmem stored in page->lru.next. These have to be in
@@ -203,6 +216,14 @@ extern void set_zone_contiguous(struct zone *zone);
 extern void clear_zone_contiguous(struct zone *zone);
 
 #else /* ! CONFIG_MEMORY_HOTPLUG */
+#define pfn_to_online_page(pfn)			\
+({						\
+	struct page *___page = NULL;		\
+	if (pfn_valid(pfn))			\
+		___page = pfn_to_page(pfn);	\
+	___page;				\
+ })
+
 /*
  * Stub functions for when hotplug is off
  */
diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h
index 8363dd27b8af..927ad95a4552 100644
--- a/include/linux/mmzone.h
+++ b/include/linux/mmzone.h
@@ -1143,9 +1143,10 @@ extern unsigned long usemap_size(void);
  */
 #define	SECTION_MARKED_PRESENT	(1UL<<0)
 #define SECTION_HAS_MEM_MAP	(1UL<<1)
-#define SECTION_MAP_LAST_BIT	(1UL<<2)
+#define SECTION_IS_ONLINE	(1UL<<2)
+#define SECTION_MAP_LAST_BIT	(1UL<<3)
 #define SECTION_MAP_MASK	(~(SECTION_MAP_LAST_BIT-1))
-#define SECTION_NID_SHIFT	2
+#define SECTION_NID_SHIFT	3
 
 static inline struct page *__section_mem_map_addr(struct mem_section *section)
 {
@@ -1174,6 +1175,23 @@ static inline int valid_section_nr(unsigned long nr)
 	return valid_section(__nr_to_section(nr));
 }
 
+static inline int online_section(struct mem_section *section)
+{
+	return (section && (section->section_mem_map & SECTION_IS_ONLINE));
+}
+
+static inline int online_section_nr(unsigned long nr)
+{
+	return online_section(__nr_to_section(nr));
+}
+
+#ifdef CONFIG_MEMORY_HOTPLUG
+void online_mem_sections(unsigned long start_pfn, unsigned long end_pfn);
+#ifdef CONFIG_MEMORY_HOTREMOVE
+void offline_mem_sections(unsigned long start_pfn, unsigned long end_pfn);
+#endif
+#endif
+
 static inline struct mem_section *__pfn_to_section(unsigned long pfn)
 {
 	return __nr_to_section(pfn_to_section_nr(pfn));
@@ -1252,10 +1270,15 @@ unsigned long __init node_memmap_size_bytes(int, unsigned long, unsigned long);
 #ifdef CONFIG_ARCH_HAS_HOLES_MEMORYMODEL
 /*
  * pfn_valid() is meant to be able to tell if a given PFN has valid memmap
- * associated with it or not. In FLATMEM, it is expected that holes always
- * have valid memmap as long as there is valid PFNs either side of the hole.
- * In SPARSEMEM, it is assumed that a valid section has a memmap for the
- * entire section.
+ * associated with it or not. This means that a struct page exists for this
+ * pfn. The caller cannot assume the page is fully initialized in general.
+ * Hotplugable pages might not have been onlined yet. pfn_to_online_page()
+ * will ensure the struct page is fully online and initialized. Special pages
+ * (e.g. ZONE_DEVICE) are never onlined and should be treated accordingly.
+ *
+ * In FLATMEM, it is expected that holes always have valid memmap as long as
+ * there is valid PFNs either side of the hole. In SPARSEMEM, it is assumed
+ * that a valid section has a memmap for the entire section.
  *
  * However, an ARM, and maybe other embedded architectures in the future
  * free memmap backing holes to save memory on the assumption the memmap is
diff --git a/mm/memory_hotplug.c b/mm/memory_hotplug.c
index 05796ee974f7..c3a146028ba6 100644
--- a/mm/memory_hotplug.c
+++ b/mm/memory_hotplug.c
@@ -929,6 +929,9 @@ static int online_pages_range(unsigned long start_pfn, unsigned long nr_pages,
 	unsigned long i;
 	unsigned long onlined_pages = *(unsigned long *)arg;
 	struct page *page;
+
+	online_mem_sections(start_pfn, start_pfn + nr_pages);
+
 	if (PageReserved(pfn_to_page(start_pfn)))
 		for (i = 0; i < nr_pages; i++) {
 			page = pfn_to_page(start_pfn + i);
diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index c1670f090107..7e5151a7dd7b 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -1353,7 +1353,9 @@ struct page *__pageblock_pfn_to_page(unsigned long start_pfn,
 	if (!pfn_valid(start_pfn) || !pfn_valid(end_pfn))
 		return NULL;
 
-	start_page = pfn_to_page(start_pfn);
+	start_page = pfn_to_online_page(start_pfn);
+	if (!start_page)
+		return NULL;
 
 	if (page_zone(start_page) != zone)
 		return NULL;
@@ -7671,6 +7673,7 @@ __offline_isolated_pages(unsigned long start_pfn, unsigned long end_pfn)
 			break;
 	if (pfn == end_pfn)
 		return;
+	offline_mem_sections(pfn, end_pfn);
 	zone = page_zone(pfn_to_page(pfn));
 	spin_lock_irqsave(&zone->lock, flags);
 	pfn = start_pfn;
diff --git a/mm/sparse.c b/mm/sparse.c
index 5032c9a619de..9d7fd666015e 100644
--- a/mm/sparse.c
+++ b/mm/sparse.c
@@ -222,7 +222,8 @@ void __init memory_present(int nid, unsigned long start, unsigned long end)
 
 		ms = __nr_to_section(section);
 		if (!ms->section_mem_map) {
-			ms->section_mem_map = sparse_encode_early_nid(nid);
+			ms->section_mem_map = sparse_encode_early_nid(nid) |
+							SECTION_IS_ONLINE;
 			section_mark_present(ms);
 		}
 	}
@@ -622,6 +623,48 @@ void __init sparse_init(void)
 }
 
 #ifdef CONFIG_MEMORY_HOTPLUG
+
+/* Mark all memory sections within the pfn range as online */
+void online_mem_sections(unsigned long start_pfn, unsigned long end_pfn)
+{
+	unsigned long pfn;
+
+	for (pfn = start_pfn; pfn < end_pfn; pfn += PAGES_PER_SECTION) {
+		unsigned long section_nr = pfn_to_section_nr(start_pfn);
+		struct mem_section *ms;
+
+		/* onlining code should never touch invalid ranges */
+		if (WARN_ON(!valid_section_nr(section_nr)))
+			continue;
+
+		ms = __nr_to_section(section_nr);
+		ms->section_mem_map |= SECTION_IS_ONLINE;
+	}
+}
+
+#ifdef CONFIG_MEMORY_HOTREMOVE
+/* Mark all memory sections within the pfn range as online */
+void offline_mem_sections(unsigned long start_pfn, unsigned long end_pfn)
+{
+	unsigned long pfn;
+
+	for (pfn = start_pfn; pfn < end_pfn; pfn += PAGES_PER_SECTION) {
+		unsigned long section_nr = pfn_to_section_nr(start_pfn);
+		struct mem_section *ms;
+
+		/*
+		 * TODO this needs some double checking. Offlining code makes
+		 * sure to check pfn_valid but those checks might be just bogus
+		 */
+		if (WARN_ON(!valid_section_nr(section_nr)))
+			continue;
+
+		ms = __nr_to_section(section_nr);
+		ms->section_mem_map &= ~SECTION_IS_ONLINE;
+	}
+}
+#endif
+
 #ifdef CONFIG_SPARSEMEM_VMEMMAP
 static inline struct page *kmalloc_section_memmap(unsigned long pnum, int nid)
 {
-- 
2.11.0

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

^ permalink raw reply	[flat|nested] 74+ messages in thread

* [PATCH 08/14] mm, compaction: skip over holes in __reset_isolation_suitable
  2017-05-15  8:58 ` Michal Hocko
@ 2017-05-15  8:58   ` Michal Hocko
  -1 siblings, 0 replies; 74+ messages in thread
From: Michal Hocko @ 2017-05-15  8:58 UTC (permalink / raw)
  To: Andrew Morton
  Cc: linux-mm, Mel Gorman, Vlastimil Babka, Andrea Arcangeli,
	Jerome Glisse, Reza Arbab, Yasuaki Ishimatsu, qiuxishi,
	Kani Toshimitsu, slaoub, Joonsoo Kim, Andi Kleen, David Rientjes,
	Daniel Kiper, Igor Mammedov, Vitaly Kuznetsov, LKML,
	Michal Hocko

From: Michal Hocko <mhocko@suse.com>

__reset_isolation_suitable walks the whole zone pfn range and it tries
to jump over holes by checking the zone for each page. It might still
stumble over offline pages, though. Skip those by checking
pfn_to_online_page()

Signed-off-by: Michal Hocko <mhocko@suse.com>
---
 mm/compaction.c | 5 ++---
 1 file changed, 2 insertions(+), 3 deletions(-)

diff --git a/mm/compaction.c b/mm/compaction.c
index 613c59e928cb..fb548e4c7bd4 100644
--- a/mm/compaction.c
+++ b/mm/compaction.c
@@ -236,10 +236,9 @@ static void __reset_isolation_suitable(struct zone *zone)
 
 		cond_resched();
 
-		if (!pfn_valid(pfn))
+		page = pfn_to_online_page(pfn);
+		if (!page)
 			continue;
-
-		page = pfn_to_page(pfn);
 		if (zone != page_zone(page))
 			continue;
 
-- 
2.11.0

^ permalink raw reply	[flat|nested] 74+ messages in thread

* [PATCH 08/14] mm, compaction: skip over holes in __reset_isolation_suitable
@ 2017-05-15  8:58   ` Michal Hocko
  0 siblings, 0 replies; 74+ messages in thread
From: Michal Hocko @ 2017-05-15  8:58 UTC (permalink / raw)
  To: Andrew Morton
  Cc: linux-mm, Mel Gorman, Vlastimil Babka, Andrea Arcangeli,
	Jerome Glisse, Reza Arbab, Yasuaki Ishimatsu, qiuxishi,
	Kani Toshimitsu, slaoub, Joonsoo Kim, Andi Kleen, David Rientjes,
	Daniel Kiper, Igor Mammedov, Vitaly Kuznetsov, LKML,
	Michal Hocko

From: Michal Hocko <mhocko@suse.com>

__reset_isolation_suitable walks the whole zone pfn range and it tries
to jump over holes by checking the zone for each page. It might still
stumble over offline pages, though. Skip those by checking
pfn_to_online_page()

Signed-off-by: Michal Hocko <mhocko@suse.com>
---
 mm/compaction.c | 5 ++---
 1 file changed, 2 insertions(+), 3 deletions(-)

diff --git a/mm/compaction.c b/mm/compaction.c
index 613c59e928cb..fb548e4c7bd4 100644
--- a/mm/compaction.c
+++ b/mm/compaction.c
@@ -236,10 +236,9 @@ static void __reset_isolation_suitable(struct zone *zone)
 
 		cond_resched();
 
-		if (!pfn_valid(pfn))
+		page = pfn_to_online_page(pfn);
+		if (!page)
 			continue;
-
-		page = pfn_to_page(pfn);
 		if (zone != page_zone(page))
 			continue;
 
-- 
2.11.0

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

^ permalink raw reply	[flat|nested] 74+ messages in thread

* [PATCH 09/14] mm: __first_valid_page skip over offline pages
  2017-05-15  8:58 ` Michal Hocko
@ 2017-05-15  8:58   ` Michal Hocko
  -1 siblings, 0 replies; 74+ messages in thread
From: Michal Hocko @ 2017-05-15  8:58 UTC (permalink / raw)
  To: Andrew Morton
  Cc: linux-mm, Mel Gorman, Vlastimil Babka, Andrea Arcangeli,
	Jerome Glisse, Reza Arbab, Yasuaki Ishimatsu, qiuxishi,
	Kani Toshimitsu, slaoub, Joonsoo Kim, Andi Kleen, David Rientjes,
	Daniel Kiper, Igor Mammedov, Vitaly Kuznetsov, LKML,
	Michal Hocko

From: Michal Hocko <mhocko@suse.com>

__first_valid_page skips over invalid pfns in the range but it might
still stumble over offline pages. At least start_isolate_page_range
will mark those set_migratetype_isolate. This doesn't represent
any immediate AFAICS because alloc_contig_range will fail to isolate
those pages but it relies on not fully initialized page which will
become a problem later when we stop associating offline pages to zones.
Use pfn_to_online_page to handle this.

This is more a preparatory patch than a fix.

Signed-off-by: Michal Hocko <mhocko@suse.com>
---
 mm/page_isolation.c | 26 ++++++++++++++++++--------
 1 file changed, 18 insertions(+), 8 deletions(-)

diff --git a/mm/page_isolation.c b/mm/page_isolation.c
index 5092e4ef00c8..3606104893e0 100644
--- a/mm/page_isolation.c
+++ b/mm/page_isolation.c
@@ -138,12 +138,18 @@ static inline struct page *
 __first_valid_page(unsigned long pfn, unsigned long nr_pages)
 {
 	int i;
-	for (i = 0; i < nr_pages; i++)
-		if (pfn_valid_within(pfn + i))
-			break;
-	if (unlikely(i == nr_pages))
-		return NULL;
-	return pfn_to_page(pfn + i);
+
+	for (i = 0; i < nr_pages; i++) {
+		struct page *page;
+
+		if (!pfn_valid_within(pfn + i))
+			continue;
+		page = pfn_to_online_page(pfn + i);
+		if (!page)
+			continue;
+		return page;
+	}
+	return NULL;
 }
 
 /*
@@ -184,8 +190,12 @@ int start_isolate_page_range(unsigned long start_pfn, unsigned long end_pfn,
 undo:
 	for (pfn = start_pfn;
 	     pfn < undo_pfn;
-	     pfn += pageblock_nr_pages)
-		unset_migratetype_isolate(pfn_to_page(pfn), migratetype);
+	     pfn += pageblock_nr_pages) {
+		struct page *page = pfn_to_online_page(pfn);
+		if (!page)
+			continue;
+		unset_migratetype_isolate(page, migratetype);
+	}
 
 	return -EBUSY;
 }
-- 
2.11.0

^ permalink raw reply	[flat|nested] 74+ messages in thread

* [PATCH 09/14] mm: __first_valid_page skip over offline pages
@ 2017-05-15  8:58   ` Michal Hocko
  0 siblings, 0 replies; 74+ messages in thread
From: Michal Hocko @ 2017-05-15  8:58 UTC (permalink / raw)
  To: Andrew Morton
  Cc: linux-mm, Mel Gorman, Vlastimil Babka, Andrea Arcangeli,
	Jerome Glisse, Reza Arbab, Yasuaki Ishimatsu, qiuxishi,
	Kani Toshimitsu, slaoub, Joonsoo Kim, Andi Kleen, David Rientjes,
	Daniel Kiper, Igor Mammedov, Vitaly Kuznetsov, LKML,
	Michal Hocko

From: Michal Hocko <mhocko@suse.com>

__first_valid_page skips over invalid pfns in the range but it might
still stumble over offline pages. At least start_isolate_page_range
will mark those set_migratetype_isolate. This doesn't represent
any immediate AFAICS because alloc_contig_range will fail to isolate
those pages but it relies on not fully initialized page which will
become a problem later when we stop associating offline pages to zones.
Use pfn_to_online_page to handle this.

This is more a preparatory patch than a fix.

Signed-off-by: Michal Hocko <mhocko@suse.com>
---
 mm/page_isolation.c | 26 ++++++++++++++++++--------
 1 file changed, 18 insertions(+), 8 deletions(-)

diff --git a/mm/page_isolation.c b/mm/page_isolation.c
index 5092e4ef00c8..3606104893e0 100644
--- a/mm/page_isolation.c
+++ b/mm/page_isolation.c
@@ -138,12 +138,18 @@ static inline struct page *
 __first_valid_page(unsigned long pfn, unsigned long nr_pages)
 {
 	int i;
-	for (i = 0; i < nr_pages; i++)
-		if (pfn_valid_within(pfn + i))
-			break;
-	if (unlikely(i == nr_pages))
-		return NULL;
-	return pfn_to_page(pfn + i);
+
+	for (i = 0; i < nr_pages; i++) {
+		struct page *page;
+
+		if (!pfn_valid_within(pfn + i))
+			continue;
+		page = pfn_to_online_page(pfn + i);
+		if (!page)
+			continue;
+		return page;
+	}
+	return NULL;
 }
 
 /*
@@ -184,8 +190,12 @@ int start_isolate_page_range(unsigned long start_pfn, unsigned long end_pfn,
 undo:
 	for (pfn = start_pfn;
 	     pfn < undo_pfn;
-	     pfn += pageblock_nr_pages)
-		unset_migratetype_isolate(pfn_to_page(pfn), migratetype);
+	     pfn += pageblock_nr_pages) {
+		struct page *page = pfn_to_online_page(pfn);
+		if (!page)
+			continue;
+		unset_migratetype_isolate(page, migratetype);
+	}
 
 	return -EBUSY;
 }
-- 
2.11.0

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

^ permalink raw reply	[flat|nested] 74+ messages in thread

* [PATCH 10/14] mm, vmstat: skip reporting offline pages in pagetypeinfo
  2017-05-15  8:58 ` Michal Hocko
@ 2017-05-15  8:58   ` Michal Hocko
  -1 siblings, 0 replies; 74+ messages in thread
From: Michal Hocko @ 2017-05-15  8:58 UTC (permalink / raw)
  To: Andrew Morton
  Cc: linux-mm, Mel Gorman, Vlastimil Babka, Andrea Arcangeli,
	Jerome Glisse, Reza Arbab, Yasuaki Ishimatsu, qiuxishi,
	Kani Toshimitsu, slaoub, Joonsoo Kim, Andi Kleen, David Rientjes,
	Daniel Kiper, Igor Mammedov, Vitaly Kuznetsov, LKML,
	Michal Hocko

From: Michal Hocko <mhocko@suse.com>

pagetypeinfo_showblockcount_print skips over invalid pfns but it would
report pages which are offline because those have a valid pfn. Their
migrate type is misleading at best. Now that we have pfn_to_online_page()
we can use it instead of pfn_valid() and fix this.

Noticed-by: Joonsoo Kim <js1304@gmail.com>
Signed-off-by: Michal Hocko <mhocko@suse.com>
---
 mm/vmstat.c | 4 +---
 1 file changed, 1 insertion(+), 3 deletions(-)

diff --git a/mm/vmstat.c b/mm/vmstat.c
index 571d3ec05566..c432e581f9a9 100644
--- a/mm/vmstat.c
+++ b/mm/vmstat.c
@@ -1223,11 +1223,9 @@ static void pagetypeinfo_showblockcount_print(struct seq_file *m,
 	for (pfn = start_pfn; pfn < end_pfn; pfn += pageblock_nr_pages) {
 		struct page *page;
 
-		if (!pfn_valid(pfn))
+		if (!pfn_to_online_page(pfn))
 			continue;
 
-		page = pfn_to_page(pfn);
-
 		/* Watch for unexpected holes punched in the memmap */
 		if (!memmap_valid_within(pfn, page, zone))
 			continue;
-- 
2.11.0

^ permalink raw reply	[flat|nested] 74+ messages in thread

* [PATCH 10/14] mm, vmstat: skip reporting offline pages in pagetypeinfo
@ 2017-05-15  8:58   ` Michal Hocko
  0 siblings, 0 replies; 74+ messages in thread
From: Michal Hocko @ 2017-05-15  8:58 UTC (permalink / raw)
  To: Andrew Morton
  Cc: linux-mm, Mel Gorman, Vlastimil Babka, Andrea Arcangeli,
	Jerome Glisse, Reza Arbab, Yasuaki Ishimatsu, qiuxishi,
	Kani Toshimitsu, slaoub, Joonsoo Kim, Andi Kleen, David Rientjes,
	Daniel Kiper, Igor Mammedov, Vitaly Kuznetsov, LKML,
	Michal Hocko

From: Michal Hocko <mhocko@suse.com>

pagetypeinfo_showblockcount_print skips over invalid pfns but it would
report pages which are offline because those have a valid pfn. Their
migrate type is misleading at best. Now that we have pfn_to_online_page()
we can use it instead of pfn_valid() and fix this.

Noticed-by: Joonsoo Kim <js1304@gmail.com>
Signed-off-by: Michal Hocko <mhocko@suse.com>
---
 mm/vmstat.c | 4 +---
 1 file changed, 1 insertion(+), 3 deletions(-)

diff --git a/mm/vmstat.c b/mm/vmstat.c
index 571d3ec05566..c432e581f9a9 100644
--- a/mm/vmstat.c
+++ b/mm/vmstat.c
@@ -1223,11 +1223,9 @@ static void pagetypeinfo_showblockcount_print(struct seq_file *m,
 	for (pfn = start_pfn; pfn < end_pfn; pfn += pageblock_nr_pages) {
 		struct page *page;
 
-		if (!pfn_valid(pfn))
+		if (!pfn_to_online_page(pfn))
 			continue;
 
-		page = pfn_to_page(pfn);
-
 		/* Watch for unexpected holes punched in the memmap */
 		if (!memmap_valid_within(pfn, page, zone))
 			continue;
-- 
2.11.0

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

^ permalink raw reply	[flat|nested] 74+ messages in thread

* [PATCH 11/14] mm, memory_hotplug: do not associate hotadded memory to zones until online
  2017-05-15  8:58 ` Michal Hocko
@ 2017-05-15  8:58   ` Michal Hocko
  -1 siblings, 0 replies; 74+ messages in thread
From: Michal Hocko @ 2017-05-15  8:58 UTC (permalink / raw)
  To: Andrew Morton
  Cc: linux-mm, Mel Gorman, Vlastimil Babka, Andrea Arcangeli,
	Jerome Glisse, Reza Arbab, Yasuaki Ishimatsu, qiuxishi,
	Kani Toshimitsu, slaoub, Joonsoo Kim, Andi Kleen, David Rientjes,
	Daniel Kiper, Igor Mammedov, Vitaly Kuznetsov, LKML,
	Michal Hocko, Dan Williams, Heiko Carstens, Martin Schwidefsky

From: Michal Hocko <mhocko@suse.com>

The current memory hotplug implementation relies on having all the
struct pages associate with a zone/node during the physical hotplug phase
(arch_add_memory->__add_pages->__add_section->__add_zone). In the vast
majority of cases this means that they are added to ZONE_NORMAL. This
has been so since 9d99aaa31f59 ("[PATCH] x86_64: Support memory hotadd
without sparsemem") and it wasn't a big deal back then because movable
onlining didn't exist yet.

Much later memory hotplug wanted to (ab)use ZONE_MOVABLE for movable
onlining 511c2aba8f07 ("mm, memory-hotplug: dynamic configure movable
memory and portion memory") and then things got more complicated. Rather
than reconsidering the zone association which was no longer needed
(because the memory hotplug already depended on SPARSEMEM) a convoluted
semantic of zone shifting has been developed. Only the currently last
memblock or the one adjacent to the zone_movable can be onlined movable.
This essentially means that the online type changes as the new memblocks
are added.

Let's simulate memory hot online manually
$ echo 0x100000000 > /sys/devices/system/memory/probe
$ grep . /sys/devices/system/memory/memory32/valid_zones
Normal Movable

$ echo $((0x100000000+(128<<20))) > /sys/devices/system/memory/probe
$ grep . /sys/devices/system/memory/memory3?/valid_zones
/sys/devices/system/memory/memory32/valid_zones:Normal
/sys/devices/system/memory/memory33/valid_zones:Normal Movable

$ echo $((0x100000000+2*(128<<20))) > /sys/devices/system/memory/probe
$ grep . /sys/devices/system/memory/memory3?/valid_zones
/sys/devices/system/memory/memory32/valid_zones:Normal
/sys/devices/system/memory/memory33/valid_zones:Normal
/sys/devices/system/memory/memory34/valid_zones:Normal Movable

$ echo online_movable > /sys/devices/system/memory/memory34/state
$ grep . /sys/devices/system/memory/memory3?/valid_zones
/sys/devices/system/memory/memory32/valid_zones:Normal
/sys/devices/system/memory/memory33/valid_zones:Normal Movable
/sys/devices/system/memory/memory34/valid_zones:Movable Normal

This is an awkward semantic because an udev event is sent as soon as the
block is onlined and an udev handler might want to online it based on
some policy (e.g. association with a node) but it will inherently race
with new blocks showing up.

This patch changes the physical online phase to not associate pages
with any zone at all. All the pages are just marked reserved and wait
for the onlining phase to be associated with the zone as per the online
request. There are only two requirements
	- existing ZONE_NORMAL and ZONE_MOVABLE cannot overlap
	- ZONE_NORMAL precedes ZONE_MOVABLE in physical addresses
the later on is not an inherent requirement and can be changed in the
future. It preserves the current behavior and made the code slightly
simpler. This is subject to change in future.

This means that the same physical online steps as above will lead to the
following state:
Normal Movable

/sys/devices/system/memory/memory32/valid_zones:Normal Movable
/sys/devices/system/memory/memory33/valid_zones:Normal Movable

/sys/devices/system/memory/memory32/valid_zones:Normal Movable
/sys/devices/system/memory/memory33/valid_zones:Normal Movable
/sys/devices/system/memory/memory34/valid_zones:Normal Movable

/sys/devices/system/memory/memory32/valid_zones:Normal Movable
/sys/devices/system/memory/memory33/valid_zones:Normal Movable
/sys/devices/system/memory/memory34/valid_zones:Movable

Implementation:
The current move_pfn_range is reimplemented to check the above
requirements (allow_online_pfn_range) and then updates the respective
zone (move_pfn_range_to_zone), the pgdat and links all the pages in the
pfn range with the zone/node. __add_pages is updated to not require the
zone and only initializes sections in the range. This allowed to
simplify the arch_add_memory code (s390 could get rid of quite some
of code).

devm_memremap_pages is the only user of arch_add_memory which relies
on the zone association because it only hooks into the memory hotplug
only half way. It uses it to associate the new memory with ZONE_DEVICE
but doesn't allow it to be {on,off}lined via sysfs. This means that this
particular code path has to call move_pfn_range_to_zone explicitly.

The original zone shifting code is kept in place and will be removed in
the follow up patch for an easier review.

Please note that this patch also changes the original behavior when
offlining a memory block adjacent to another zone (Normal vs. Movable)
used to allow to change its movable type. This will be handled later.

Changes since v1
- we have to associate the page with the node early (in __add_section),
  because pfn_to_node depends on struct page containing this
  information - based on testing by Reza Arbab
- resize_{zone,pgdat}_range has to check whether they are popoulated -
  Reza Arbab
- fix devm_memremap_pages to use pfn rather than physical address -
  Jérôme Glisse
- move_pfn_range has to check for intersection with zone_movable rather
  than to rely on allow_online_pfn_range(MMOP_ONLINE_MOVABLE) for
  MMOP_ONLINE_KEEP

Changes since v2
- fix show_valid_zones nr_pages calculation
- allow_online_pfn_range has to check managed pages rather than present
- zone_intersects fix bogus check
- fix zone_intersects + few nits as per Vlastimil

Cc: Martin Schwidefsky <schwidefsky@de.ibm.com>
Cc: linux-arch@vger.kernel.org
Tested-by: Dan Williams <dan.j.williams@intel.com>
Acked-by: Heiko Carstens <heiko.carstens@de.ibm.com> # For s390 bits
Tested-by: Reza Arbab <arbab@linux.vnet.ibm.com>
Signed-off-by: Michal Hocko <mhocko@suse.com>
---
 arch/ia64/mm/init.c            |   9 +-
 arch/powerpc/mm/mem.c          |  10 +-
 arch/s390/mm/init.c            |  30 +-----
 arch/sh/mm/init.c              |   8 +-
 arch/x86/mm/init_32.c          |   5 +-
 arch/x86/mm/init_64.c          |   9 +-
 drivers/base/memory.c          |  52 ++++++-----
 include/linux/memory_hotplug.h |  13 +--
 include/linux/mmzone.h         |  20 ++++
 kernel/memremap.c              |   4 +
 mm/memory_hotplug.c            | 204 +++++++++++++++++++++++++----------------
 mm/sparse.c                    |   3 +-
 12 files changed, 193 insertions(+), 174 deletions(-)

diff --git a/arch/ia64/mm/init.c b/arch/ia64/mm/init.c
index 39e2aeb4669d..80db57d063d0 100644
--- a/arch/ia64/mm/init.c
+++ b/arch/ia64/mm/init.c
@@ -648,18 +648,11 @@ mem_init (void)
 #ifdef CONFIG_MEMORY_HOTPLUG
 int arch_add_memory(int nid, u64 start, u64 size, bool for_device)
 {
-	pg_data_t *pgdat;
-	struct zone *zone;
 	unsigned long start_pfn = start >> PAGE_SHIFT;
 	unsigned long nr_pages = size >> PAGE_SHIFT;
 	int ret;
 
-	pgdat = NODE_DATA(nid);
-
-	zone = pgdat->node_zones +
-		zone_for_memory(nid, start, size, ZONE_NORMAL, for_device);
-	ret = __add_pages(nid, zone, start_pfn, nr_pages, !for_device);
-
+	ret = __add_pages(nid, start_pfn, nr_pages, !for_device);
 	if (ret)
 		printk("%s: Problem encountered in __add_pages() as ret=%d\n",
 		       __func__,  ret);
diff --git a/arch/powerpc/mm/mem.c b/arch/powerpc/mm/mem.c
index e6b2e6618b6c..72c46eb53215 100644
--- a/arch/powerpc/mm/mem.c
+++ b/arch/powerpc/mm/mem.c
@@ -128,16 +128,12 @@ int __weak remove_section_mapping(unsigned long start, unsigned long end)
 
 int arch_add_memory(int nid, u64 start, u64 size, bool for_device)
 {
-	struct pglist_data *pgdata;
-	struct zone *zone;
 	unsigned long start_pfn = start >> PAGE_SHIFT;
 	unsigned long nr_pages = size >> PAGE_SHIFT;
 	int rc;
 
 	resize_hpt_for_hotplug(memblock_phys_mem_size());
 
-	pgdata = NODE_DATA(nid);
-
 	start = (unsigned long)__va(start);
 	rc = create_section_mapping(start, start + size);
 	if (rc) {
@@ -147,11 +143,7 @@ int arch_add_memory(int nid, u64 start, u64 size, bool for_device)
 		return -EFAULT;
 	}
 
-	/* this should work for most non-highmem platforms */
-	zone = pgdata->node_zones +
-		zone_for_memory(nid, start, size, 0, for_device);
-
-	return __add_pages(nid, zone, start_pfn, nr_pages, !for_device);
+	return __add_pages(nid, start_pfn, nr_pages, !for_device);
 }
 
 #ifdef CONFIG_MEMORY_HOTREMOVE
diff --git a/arch/s390/mm/init.c b/arch/s390/mm/init.c
index 893cf88cf02d..862824924ba6 100644
--- a/arch/s390/mm/init.c
+++ b/arch/s390/mm/init.c
@@ -164,41 +164,15 @@ unsigned long memory_block_size_bytes(void)
 #ifdef CONFIG_MEMORY_HOTPLUG
 int arch_add_memory(int nid, u64 start, u64 size, bool for_device)
 {
-	unsigned long zone_start_pfn, zone_end_pfn, nr_pages;
 	unsigned long start_pfn = PFN_DOWN(start);
 	unsigned long size_pages = PFN_DOWN(size);
-	pg_data_t *pgdat = NODE_DATA(nid);
-	struct zone *zone;
-	int rc, i;
+	int rc;
 
 	rc = vmem_add_mapping(start, size);
 	if (rc)
 		return rc;
 
-	for (i = 0; i < MAX_NR_ZONES; i++) {
-		zone = pgdat->node_zones + i;
-		if (zone_idx(zone) != ZONE_MOVABLE) {
-			/* Add range within existing zone limits, if possible */
-			zone_start_pfn = zone->zone_start_pfn;
-			zone_end_pfn = zone->zone_start_pfn +
-				       zone->spanned_pages;
-		} else {
-			/* Add remaining range to ZONE_MOVABLE */
-			zone_start_pfn = start_pfn;
-			zone_end_pfn = start_pfn + size_pages;
-		}
-		if (start_pfn < zone_start_pfn || start_pfn >= zone_end_pfn)
-			continue;
-		nr_pages = (start_pfn + size_pages > zone_end_pfn) ?
-			   zone_end_pfn - start_pfn : size_pages;
-		rc = __add_pages(nid, zone, start_pfn, nr_pages, !for_device);
-		if (rc)
-			break;
-		start_pfn += nr_pages;
-		size_pages -= nr_pages;
-		if (!size_pages)
-			break;
-	}
+	rc = __add_pages(nid, start_pfn, size_pages, !for_device);
 	if (rc)
 		vmem_remove_mapping(start, size);
 	return rc;
diff --git a/arch/sh/mm/init.c b/arch/sh/mm/init.c
index a9d57f75ae8c..3813a610a2bb 100644
--- a/arch/sh/mm/init.c
+++ b/arch/sh/mm/init.c
@@ -487,18 +487,12 @@ void free_initrd_mem(unsigned long start, unsigned long end)
 #ifdef CONFIG_MEMORY_HOTPLUG
 int arch_add_memory(int nid, u64 start, u64 size, bool for_device)
 {
-	pg_data_t *pgdat;
 	unsigned long start_pfn = PFN_DOWN(start);
 	unsigned long nr_pages = size >> PAGE_SHIFT;
 	int ret;
 
-	pgdat = NODE_DATA(nid);
-
 	/* We only have ZONE_NORMAL, so this is easy.. */
-	ret = __add_pages(nid, pgdat->node_zones +
-			zone_for_memory(nid, start, size, ZONE_NORMAL,
-			for_device),
-			start_pfn, nr_pages, !for_device);
+	ret = __add_pages(nid, start_pfn, nr_pages, !for_device);
 	if (unlikely(ret))
 		printk("%s: Failed, __add_pages() == %d\n", __func__, ret);
 
diff --git a/arch/x86/mm/init_32.c b/arch/x86/mm/init_32.c
index 94594b889144..a424066d0552 100644
--- a/arch/x86/mm/init_32.c
+++ b/arch/x86/mm/init_32.c
@@ -825,13 +825,10 @@ void __init mem_init(void)
 #ifdef CONFIG_MEMORY_HOTPLUG
 int arch_add_memory(int nid, u64 start, u64 size, bool for_device)
 {
-	struct pglist_data *pgdata = NODE_DATA(nid);
-	struct zone *zone = pgdata->node_zones +
-		zone_for_memory(nid, start, size, ZONE_HIGHMEM, for_device);
 	unsigned long start_pfn = start >> PAGE_SHIFT;
 	unsigned long nr_pages = size >> PAGE_SHIFT;
 
-	return __add_pages(nid, zone, start_pfn, nr_pages, !for_device);
+	return __add_pages(nid, start_pfn, nr_pages, !for_device);
 }
 
 #ifdef CONFIG_MEMORY_HOTREMOVE
diff --git a/arch/x86/mm/init_64.c b/arch/x86/mm/init_64.c
index 2e004364a373..884c1d0a57b3 100644
--- a/arch/x86/mm/init_64.c
+++ b/arch/x86/mm/init_64.c
@@ -682,22 +682,15 @@ static void  update_end_of_memory_vars(u64 start, u64 size)
 	}
 }
 
-/*
- * Memory is added always to NORMAL zone. This means you will never get
- * additional DMA/DMA32 memory.
- */
 int arch_add_memory(int nid, u64 start, u64 size, bool for_device)
 {
-	struct pglist_data *pgdat = NODE_DATA(nid);
-	struct zone *zone = pgdat->node_zones +
-		zone_for_memory(nid, start, size, ZONE_NORMAL, for_device);
 	unsigned long start_pfn = start >> PAGE_SHIFT;
 	unsigned long nr_pages = size >> PAGE_SHIFT;
 	int ret;
 
 	init_memory_mapping(start, start + size);
 
-	ret = __add_pages(nid, zone, start_pfn, nr_pages, !for_device);
+	ret = __add_pages(nid, start_pfn, nr_pages, !for_device);
 	WARN_ON_ONCE(ret);
 
 	/* update max_pfn, max_low_pfn and high_memory */
diff --git a/drivers/base/memory.c b/drivers/base/memory.c
index 1e884d82af6f..b86fda30ce62 100644
--- a/drivers/base/memory.c
+++ b/drivers/base/memory.c
@@ -392,39 +392,43 @@ static ssize_t show_valid_zones(struct device *dev,
 				struct device_attribute *attr, char *buf)
 {
 	struct memory_block *mem = to_memory_block(dev);
-	unsigned long start_pfn, end_pfn;
-	unsigned long valid_start, valid_end, valid_pages;
+	unsigned long start_pfn = section_nr_to_pfn(mem->start_section_nr);
 	unsigned long nr_pages = PAGES_PER_SECTION * sections_per_block;
-	struct zone *zone;
-	int zone_shift = 0;
+	unsigned long valid_start_pfn, valid_end_pfn;
+	bool append = false;
+	int nid;
 
-	start_pfn = section_nr_to_pfn(mem->start_section_nr);
-	end_pfn = start_pfn + nr_pages;
-
-	/* The block contains more than one zone can not be offlined. */
-	if (!test_pages_in_a_zone(start_pfn, end_pfn, &valid_start, &valid_end))
+	/*
+	 * The block contains more than one zone can not be offlined.
+	 * This can happen e.g. for ZONE_DMA and ZONE_DMA32
+	 */
+	if (!test_pages_in_a_zone(start_pfn, start_pfn + nr_pages, &valid_start_pfn, &valid_end_pfn))
 		return sprintf(buf, "none\n");
 
-	zone = page_zone(pfn_to_page(valid_start));
-	valid_pages = valid_end - valid_start;
-
-	/* MMOP_ONLINE_KEEP */
-	sprintf(buf, "%s", zone->name);
+	start_pfn = valid_start_pfn;
+	nr_pages = valid_end_pfn - start_pfn;
 
-	/* MMOP_ONLINE_KERNEL */
-	zone_can_shift(valid_start, valid_pages, ZONE_NORMAL, &zone_shift);
-	if (zone_shift) {
-		strcat(buf, " ");
-		strcat(buf, (zone + zone_shift)->name);
+	/*
+	 * Check the existing zone. Make sure that we do that only on the
+	 * online nodes otherwise the page_zone is not reliable
+	 */
+	if (mem->state == MEM_ONLINE) {
+		strcat(buf, page_zone(pfn_to_page(start_pfn))->name);
+		goto out;
 	}
 
-	/* MMOP_ONLINE_MOVABLE */
-	zone_can_shift(valid_start, valid_pages, ZONE_MOVABLE, &zone_shift);
-	if (zone_shift) {
-		strcat(buf, " ");
-		strcat(buf, (zone + zone_shift)->name);
+	nid = pfn_to_nid(start_pfn);
+	if (allow_online_pfn_range(nid, start_pfn, nr_pages, MMOP_ONLINE_KERNEL)) {
+		strcat(buf, NODE_DATA(nid)->node_zones[ZONE_NORMAL].name);
+		append = true;
 	}
 
+	if (allow_online_pfn_range(nid, start_pfn, nr_pages, MMOP_ONLINE_MOVABLE)) {
+		if (append)
+			strcat(buf, " ");
+		strcat(buf, NODE_DATA(nid)->node_zones[ZONE_MOVABLE].name);
+	}
+out:
 	strcat(buf, "\n");
 
 	return strlen(buf);
diff --git a/include/linux/memory_hotplug.h b/include/linux/memory_hotplug.h
index fc1c873504eb..9cd76ff7b0c5 100644
--- a/include/linux/memory_hotplug.h
+++ b/include/linux/memory_hotplug.h
@@ -122,8 +122,8 @@ extern int __remove_pages(struct zone *zone, unsigned long start_pfn,
 	unsigned long nr_pages);
 #endif /* CONFIG_MEMORY_HOTREMOVE */
 
-/* reasonably generic interface to expand the physical pages in a zone  */
-extern int __add_pages(int nid, struct zone *zone, unsigned long start_pfn,
+/* reasonably generic interface to expand the physical pages */
+extern int __add_pages(int nid, unsigned long start_pfn,
 	unsigned long nr_pages, bool want_memblock);
 
 #ifdef CONFIG_NUMA
@@ -298,15 +298,16 @@ extern int add_memory_resource(int nid, struct resource *resource, bool online);
 extern int zone_for_memory(int nid, u64 start, u64 size, int zone_default,
 		bool for_device);
 extern int arch_add_memory(int nid, u64 start, u64 size, bool for_device);
+extern void move_pfn_range_to_zone(struct zone *zone, unsigned long start_pfn,
+		unsigned long nr_pages);
 extern int offline_pages(unsigned long start_pfn, unsigned long nr_pages);
 extern bool is_memblock_offlined(struct memory_block *mem);
 extern void remove_memory(int nid, u64 start, u64 size);
-extern int sparse_add_one_section(struct zone *zone, unsigned long start_pfn);
+extern int sparse_add_one_section(struct pglist_data *pgdat, unsigned long start_pfn);
 extern void sparse_remove_one_section(struct zone *zone, struct mem_section *ms,
 		unsigned long map_offset);
 extern struct page *sparse_decode_mem_map(unsigned long coded_mem_map,
 					  unsigned long pnum);
-extern bool zone_can_shift(unsigned long pfn, unsigned long nr_pages,
-			  enum zone_type target, int *zone_shift);
-
+extern bool allow_online_pfn_range(int nid, unsigned long pfn, unsigned long nr_pages,
+		int online_type);
 #endif /* __LINUX_MEMORY_HOTPLUG_H */
diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h
index 927ad95a4552..f887fccb6ef0 100644
--- a/include/linux/mmzone.h
+++ b/include/linux/mmzone.h
@@ -533,6 +533,26 @@ static inline bool zone_is_empty(struct zone *zone)
 }
 
 /*
+ * Return true if [start_pfn, start_pfn + nr_pages) range has a non-empty
+ * intersection with the given zone
+ */
+static inline bool zone_intersects(struct zone *zone,
+		unsigned long start_pfn, unsigned long nr_pages)
+{
+	if (zone_is_empty(zone))
+		return false;
+	if (start_pfn >= zone_end_pfn(zone))
+		return false;
+
+	if (zone->zone_start_pfn <= start_pfn)
+		return true;
+	if (start_pfn + nr_pages > zone->zone_start_pfn)
+		return true;
+
+	return false;
+}
+
+/*
  * The "priority" of VM scanning is how much of the queues we will scan in one
  * go. A value of 12 for DEF_PRIORITY implies that we will scan 1/4096th of the
  * queues ("queue_length >> 12") during an aging round.
diff --git a/kernel/memremap.c b/kernel/memremap.c
index 23a6483c3666..281eb478856a 100644
--- a/kernel/memremap.c
+++ b/kernel/memremap.c
@@ -359,6 +359,10 @@ void *devm_memremap_pages(struct device *dev, struct resource *res,
 
 	mem_hotplug_begin();
 	error = arch_add_memory(nid, align_start, align_size, true);
+	if (!error)
+		move_pfn_range_to_zone(&NODE_DATA(nid)->node_zones[ZONE_DEVICE],
+					align_start >> PAGE_SHIFT,
+					align_size >> PAGE_SHIFT);
 	mem_hotplug_done();
 	if (error)
 		goto err_add_memory;
diff --git a/mm/memory_hotplug.c b/mm/memory_hotplug.c
index c3a146028ba6..dc363370e9d8 100644
--- a/mm/memory_hotplug.c
+++ b/mm/memory_hotplug.c
@@ -433,25 +433,6 @@ static int __meminit move_pfn_range_right(struct zone *z1, struct zone *z2,
 	return -1;
 }
 
-static struct zone * __meminit move_pfn_range(int zone_shift,
-		unsigned long start_pfn, unsigned long end_pfn)
-{
-	struct zone *zone = page_zone(pfn_to_page(start_pfn));
-	int ret = 0;
-
-	if (zone_shift < 0)
-		ret = move_pfn_range_left(zone + zone_shift, zone,
-					  start_pfn, end_pfn);
-	else if (zone_shift)
-		ret = move_pfn_range_right(zone, zone + zone_shift,
-					   start_pfn, end_pfn);
-
-	if (ret)
-		return NULL;
-
-	return zone + zone_shift;
-}
-
 static void __meminit grow_pgdat_span(struct pglist_data *pgdat, unsigned long start_pfn,
 				      unsigned long end_pfn)
 {
@@ -493,23 +474,35 @@ static int __meminit __add_zone(struct zone *zone, unsigned long phys_start_pfn)
 	return 0;
 }
 
-static int __meminit __add_section(int nid, struct zone *zone,
-		unsigned long phys_start_pfn, bool want_memblock)
+static int __meminit __add_section(int nid, unsigned long phys_start_pfn,
+		bool want_memblock)
 {
 	int ret;
+	int i;
 
 	if (pfn_valid(phys_start_pfn))
 		return -EEXIST;
 
-	ret = sparse_add_one_section(zone, phys_start_pfn);
-
+	ret = sparse_add_one_section(NODE_DATA(nid), phys_start_pfn);
 	if (ret < 0)
 		return ret;
 
-	ret = __add_zone(zone, phys_start_pfn);
+	/*
+	 * Make all the pages reserved so that nobody will stumble over half
+	 * initialized state.
+	 * FIXME: We also have to associate it with a node because pfn_to_node
+	 * relies on having page with the proper node.
+	 */
+	for (i = 0; i < PAGES_PER_SECTION; i++) {
+		unsigned long pfn = phys_start_pfn + i;
+		struct page *page;
+		if (!pfn_valid(pfn))
+			continue;
 
-	if (ret < 0)
-		return ret;
+		page = pfn_to_page(pfn);
+		set_page_node(page, nid);
+		SetPageReserved(page);
+	}
 
 	if (!want_memblock)
 		return 0;
@@ -523,7 +516,7 @@ static int __meminit __add_section(int nid, struct zone *zone,
  * call this function after deciding the zone to which to
  * add the new pages.
  */
-int __ref __add_pages(int nid, struct zone *zone, unsigned long phys_start_pfn,
+int __ref __add_pages(int nid, unsigned long phys_start_pfn,
 			unsigned long nr_pages, bool want_memblock)
 {
 	unsigned long i;
@@ -531,8 +524,6 @@ int __ref __add_pages(int nid, struct zone *zone, unsigned long phys_start_pfn,
 	int start_sec, end_sec;
 	struct vmem_altmap *altmap;
 
-	clear_zone_contiguous(zone);
-
 	/* during initialize mem_map, align hot-added range to section */
 	start_sec = pfn_to_section_nr(phys_start_pfn);
 	end_sec = pfn_to_section_nr(phys_start_pfn + nr_pages - 1);
@@ -552,7 +543,7 @@ int __ref __add_pages(int nid, struct zone *zone, unsigned long phys_start_pfn,
 	}
 
 	for (i = start_sec; i <= end_sec; i++) {
-		err = __add_section(nid, zone, section_nr_to_pfn(i), want_memblock);
+		err = __add_section(nid, section_nr_to_pfn(i), want_memblock);
 
 		/*
 		 * EEXIST is finally dealt with by ioresource collision
@@ -565,7 +556,6 @@ int __ref __add_pages(int nid, struct zone *zone, unsigned long phys_start_pfn,
 	}
 	vmemmap_populate_print_last();
 out:
-	set_zone_contiguous(zone);
 	return err;
 }
 EXPORT_SYMBOL_GPL(__add_pages);
@@ -1037,39 +1027,114 @@ static void node_states_set_node(int node, struct memory_notify *arg)
 	node_set_state(node, N_MEMORY);
 }
 
-bool zone_can_shift(unsigned long pfn, unsigned long nr_pages,
-		   enum zone_type target, int *zone_shift)
+bool allow_online_pfn_range(int nid, unsigned long pfn, unsigned long nr_pages, int online_type)
 {
-	struct zone *zone = page_zone(pfn_to_page(pfn));
-	enum zone_type idx = zone_idx(zone);
-	int i;
+	struct pglist_data *pgdat = NODE_DATA(nid);
+	struct zone *movable_zone = &pgdat->node_zones[ZONE_MOVABLE];
+	struct zone *normal_zone =  &pgdat->node_zones[ZONE_NORMAL];
 
-	*zone_shift = 0;
+	/*
+	 * TODO there shouldn't be any inherent reason to have ZONE_NORMAL
+	 * physically before ZONE_MOVABLE. All we need is they do not
+	 * overlap. Historically we didn't allow ZONE_NORMAL after ZONE_MOVABLE
+	 * though so let's stick with it for simplicity for now.
+	 * TODO make sure we do not overlap with ZONE_DEVICE
+	 */
+	if (online_type == MMOP_ONLINE_KERNEL) {
+		if (zone_is_empty(movable_zone))
+			return true;
+		return movable_zone->zone_start_pfn >= pfn + nr_pages;
+	} else if (online_type == MMOP_ONLINE_MOVABLE) {
+		return zone_end_pfn(normal_zone) <= pfn;
+	}
 
-	if (idx < target) {
-		/* pages must be at end of current zone */
-		if (pfn + nr_pages != zone_end_pfn(zone))
-			return false;
+	/* MMOP_ONLINE_KEEP will always succeed and inherits the current zone */
+	return online_type == MMOP_ONLINE_KEEP;
+}
+
+static void __meminit resize_zone_range(struct zone *zone, unsigned long start_pfn,
+		unsigned long nr_pages)
+{
+	unsigned long old_end_pfn = zone_end_pfn(zone);
+
+	if (zone_is_empty(zone) || start_pfn < zone->zone_start_pfn)
+		zone->zone_start_pfn = start_pfn;
+
+	zone->spanned_pages = max(start_pfn + nr_pages, old_end_pfn) - zone->zone_start_pfn;
+}
+
+static void __meminit resize_pgdat_range(struct pglist_data *pgdat, unsigned long start_pfn,
+                                     unsigned long nr_pages)
+{
+	unsigned long old_end_pfn = pgdat_end_pfn(pgdat);
 
-		/* no zones in use between current zone and target */
-		for (i = idx + 1; i < target; i++)
-			if (zone_is_initialized(zone - idx + i))
-				return false;
+	if (!pgdat->node_spanned_pages || start_pfn < pgdat->node_start_pfn)
+		pgdat->node_start_pfn = start_pfn;
+
+	pgdat->node_spanned_pages = max(start_pfn + nr_pages, old_end_pfn) - pgdat->node_start_pfn;
+}
+
+void move_pfn_range_to_zone(struct zone *zone,
+		unsigned long start_pfn, unsigned long nr_pages)
+{
+	struct pglist_data *pgdat = zone->zone_pgdat;
+	int nid = pgdat->node_id;
+	unsigned long flags;
+	unsigned long i;
+
+	if (zone_is_empty(zone))
+		init_currently_empty_zone(zone, start_pfn, nr_pages);
+
+	clear_zone_contiguous(zone);
+
+	/* TODO Huh pgdat is irqsave while zone is not. It used to be like that before */
+	pgdat_resize_lock(pgdat, &flags);
+	zone_span_writelock(zone);
+	resize_zone_range(zone, start_pfn, nr_pages);
+	zone_span_writeunlock(zone);
+	resize_pgdat_range(pgdat, start_pfn, nr_pages);
+	pgdat_resize_unlock(pgdat, &flags);
+
+	/*
+	 * TODO now we have a visible range of pages which are not associated
+	 * with their zone properly. Not nice but set_pfnblock_flags_mask
+	 * expects the zone spans the pfn range. All the pages in the range
+	 * are reserved so nobody should be touching them so we should be safe
+	 */
+	memmap_init_zone(nr_pages, nid, zone_idx(zone), start_pfn, MEMMAP_HOTPLUG);
+	for (i = 0; i < nr_pages; i++) {
+		unsigned long pfn = start_pfn + i;
+		set_page_links(pfn_to_page(pfn), zone_idx(zone), nid, pfn);
 	}
 
-	if (target < idx) {
-		/* pages must be at beginning of current zone */
-		if (pfn != zone->zone_start_pfn)
-			return false;
+	set_zone_contiguous(zone);
+}
+
+/*
+ * Associates the given pfn range with the given node and the zone appropriate
+ * for the given online type.
+ */
+static struct zone * __meminit move_pfn_range(int online_type, int nid,
+		unsigned long start_pfn, unsigned long nr_pages)
+{
+	struct pglist_data *pgdat = NODE_DATA(nid);
+	struct zone *zone = &pgdat->node_zones[ZONE_NORMAL];
 
-		/* no zones in use between current zone and target */
-		for (i = target + 1; i < idx; i++)
-			if (zone_is_initialized(zone - idx + i))
-				return false;
+	if (online_type == MMOP_ONLINE_KEEP) {
+		struct zone *movable_zone = &pgdat->node_zones[ZONE_MOVABLE];
+		/*
+		 * MMOP_ONLINE_KEEP inherits the current zone which is
+		 * ZONE_NORMAL by default but we might be within ZONE_MOVABLE
+		 * already.
+		 */
+		if (zone_intersects(movable_zone, start_pfn, nr_pages))
+			zone = movable_zone;
+	} else if (online_type == MMOP_ONLINE_MOVABLE) {
+		zone = &pgdat->node_zones[ZONE_MOVABLE];
 	}
 
-	*zone_shift = target - idx;
-	return true;
+	move_pfn_range_to_zone(zone, start_pfn, nr_pages);
+	return zone;
 }
 
 /* Must be protected by mem_hotplug_begin() */
@@ -1082,38 +1147,21 @@ int __ref online_pages(unsigned long pfn, unsigned long nr_pages, int online_typ
 	int nid;
 	int ret;
 	struct memory_notify arg;
-	int zone_shift = 0;
 
-	/*
-	 * This doesn't need a lock to do pfn_to_page().
-	 * The section can't be removed here because of the
-	 * memory_block->state_mutex.
-	 */
-	zone = page_zone(pfn_to_page(pfn));
-
-	if ((zone_idx(zone) > ZONE_NORMAL ||
-	    online_type == MMOP_ONLINE_MOVABLE) &&
-	    !can_online_high_movable(pfn_to_nid(pfn)))
+	nid = pfn_to_nid(pfn);
+	if (!allow_online_pfn_range(nid, pfn, nr_pages, online_type))
 		return -EINVAL;
 
-	if (online_type == MMOP_ONLINE_KERNEL) {
-		if (!zone_can_shift(pfn, nr_pages, ZONE_NORMAL, &zone_shift))
-			return -EINVAL;
-	} else if (online_type == MMOP_ONLINE_MOVABLE) {
-		if (!zone_can_shift(pfn, nr_pages, ZONE_MOVABLE, &zone_shift))
-			return -EINVAL;
-	}
-
-	zone = move_pfn_range(zone_shift, pfn, pfn + nr_pages);
-	if (!zone)
+	if (online_type == MMOP_ONLINE_MOVABLE && !can_online_high_movable(nid))
 		return -EINVAL;
 
+	/* associate pfn range with the zone */
+	zone = move_pfn_range(online_type, nid, pfn, nr_pages);
+
 	arg.start_pfn = pfn;
 	arg.nr_pages = nr_pages;
 	node_states_check_changes_online(nr_pages, zone, &arg);
 
-	nid = zone_to_nid(zone);
-
 	ret = memory_notify(MEM_GOING_ONLINE, &arg);
 	ret = notifier_to_errno(ret);
 	if (ret)
diff --git a/mm/sparse.c b/mm/sparse.c
index 9d7fd666015e..7b4be3fd5cac 100644
--- a/mm/sparse.c
+++ b/mm/sparse.c
@@ -761,10 +761,9 @@ static void free_map_bootmem(struct page *memmap)
  * set.  If this is <=0, then that means that the passed-in
  * map was not consumed and must be freed.
  */
-int __meminit sparse_add_one_section(struct zone *zone, unsigned long start_pfn)
+int __meminit sparse_add_one_section(struct pglist_data *pgdat, unsigned long start_pfn)
 {
 	unsigned long section_nr = pfn_to_section_nr(start_pfn);
-	struct pglist_data *pgdat = zone->zone_pgdat;
 	struct mem_section *ms;
 	struct page *memmap;
 	unsigned long *usemap;
-- 
2.11.0

^ permalink raw reply	[flat|nested] 74+ messages in thread

* [PATCH 11/14] mm, memory_hotplug: do not associate hotadded memory to zones until online
@ 2017-05-15  8:58   ` Michal Hocko
  0 siblings, 0 replies; 74+ messages in thread
From: Michal Hocko @ 2017-05-15  8:58 UTC (permalink / raw)
  To: Andrew Morton
  Cc: linux-mm, Mel Gorman, Vlastimil Babka, Andrea Arcangeli,
	Jerome Glisse, Reza Arbab, Yasuaki Ishimatsu, qiuxishi,
	Kani Toshimitsu, slaoub, Joonsoo Kim, Andi Kleen, David Rientjes,
	Daniel Kiper, Igor Mammedov, Vitaly Kuznetsov, LKML,
	Michal Hocko, Dan Williams, Heiko Carstens, Martin Schwidefsky

From: Michal Hocko <mhocko@suse.com>

The current memory hotplug implementation relies on having all the
struct pages associate with a zone/node during the physical hotplug phase
(arch_add_memory->__add_pages->__add_section->__add_zone). In the vast
majority of cases this means that they are added to ZONE_NORMAL. This
has been so since 9d99aaa31f59 ("[PATCH] x86_64: Support memory hotadd
without sparsemem") and it wasn't a big deal back then because movable
onlining didn't exist yet.

Much later memory hotplug wanted to (ab)use ZONE_MOVABLE for movable
onlining 511c2aba8f07 ("mm, memory-hotplug: dynamic configure movable
memory and portion memory") and then things got more complicated. Rather
than reconsidering the zone association which was no longer needed
(because the memory hotplug already depended on SPARSEMEM) a convoluted
semantic of zone shifting has been developed. Only the currently last
memblock or the one adjacent to the zone_movable can be onlined movable.
This essentially means that the online type changes as the new memblocks
are added.

Let's simulate memory hot online manually
$ echo 0x100000000 > /sys/devices/system/memory/probe
$ grep . /sys/devices/system/memory/memory32/valid_zones
Normal Movable

$ echo $((0x100000000+(128<<20))) > /sys/devices/system/memory/probe
$ grep . /sys/devices/system/memory/memory3?/valid_zones
/sys/devices/system/memory/memory32/valid_zones:Normal
/sys/devices/system/memory/memory33/valid_zones:Normal Movable

$ echo $((0x100000000+2*(128<<20))) > /sys/devices/system/memory/probe
$ grep . /sys/devices/system/memory/memory3?/valid_zones
/sys/devices/system/memory/memory32/valid_zones:Normal
/sys/devices/system/memory/memory33/valid_zones:Normal
/sys/devices/system/memory/memory34/valid_zones:Normal Movable

$ echo online_movable > /sys/devices/system/memory/memory34/state
$ grep . /sys/devices/system/memory/memory3?/valid_zones
/sys/devices/system/memory/memory32/valid_zones:Normal
/sys/devices/system/memory/memory33/valid_zones:Normal Movable
/sys/devices/system/memory/memory34/valid_zones:Movable Normal

This is an awkward semantic because an udev event is sent as soon as the
block is onlined and an udev handler might want to online it based on
some policy (e.g. association with a node) but it will inherently race
with new blocks showing up.

This patch changes the physical online phase to not associate pages
with any zone at all. All the pages are just marked reserved and wait
for the onlining phase to be associated with the zone as per the online
request. There are only two requirements
	- existing ZONE_NORMAL and ZONE_MOVABLE cannot overlap
	- ZONE_NORMAL precedes ZONE_MOVABLE in physical addresses
the later on is not an inherent requirement and can be changed in the
future. It preserves the current behavior and made the code slightly
simpler. This is subject to change in future.

This means that the same physical online steps as above will lead to the
following state:
Normal Movable

/sys/devices/system/memory/memory32/valid_zones:Normal Movable
/sys/devices/system/memory/memory33/valid_zones:Normal Movable

/sys/devices/system/memory/memory32/valid_zones:Normal Movable
/sys/devices/system/memory/memory33/valid_zones:Normal Movable
/sys/devices/system/memory/memory34/valid_zones:Normal Movable

/sys/devices/system/memory/memory32/valid_zones:Normal Movable
/sys/devices/system/memory/memory33/valid_zones:Normal Movable
/sys/devices/system/memory/memory34/valid_zones:Movable

Implementation:
The current move_pfn_range is reimplemented to check the above
requirements (allow_online_pfn_range) and then updates the respective
zone (move_pfn_range_to_zone), the pgdat and links all the pages in the
pfn range with the zone/node. __add_pages is updated to not require the
zone and only initializes sections in the range. This allowed to
simplify the arch_add_memory code (s390 could get rid of quite some
of code).

devm_memremap_pages is the only user of arch_add_memory which relies
on the zone association because it only hooks into the memory hotplug
only half way. It uses it to associate the new memory with ZONE_DEVICE
but doesn't allow it to be {on,off}lined via sysfs. This means that this
particular code path has to call move_pfn_range_to_zone explicitly.

The original zone shifting code is kept in place and will be removed in
the follow up patch for an easier review.

Please note that this patch also changes the original behavior when
offlining a memory block adjacent to another zone (Normal vs. Movable)
used to allow to change its movable type. This will be handled later.

Changes since v1
- we have to associate the page with the node early (in __add_section),
  because pfn_to_node depends on struct page containing this
  information - based on testing by Reza Arbab
- resize_{zone,pgdat}_range has to check whether they are popoulated -
  Reza Arbab
- fix devm_memremap_pages to use pfn rather than physical address -
  JA(C)rA'me Glisse
- move_pfn_range has to check for intersection with zone_movable rather
  than to rely on allow_online_pfn_range(MMOP_ONLINE_MOVABLE) for
  MMOP_ONLINE_KEEP

Changes since v2
- fix show_valid_zones nr_pages calculation
- allow_online_pfn_range has to check managed pages rather than present
- zone_intersects fix bogus check
- fix zone_intersects + few nits as per Vlastimil

Cc: Martin Schwidefsky <schwidefsky@de.ibm.com>
Cc: linux-arch@vger.kernel.org
Tested-by: Dan Williams <dan.j.williams@intel.com>
Acked-by: Heiko Carstens <heiko.carstens@de.ibm.com> # For s390 bits
Tested-by: Reza Arbab <arbab@linux.vnet.ibm.com>
Signed-off-by: Michal Hocko <mhocko@suse.com>
---
 arch/ia64/mm/init.c            |   9 +-
 arch/powerpc/mm/mem.c          |  10 +-
 arch/s390/mm/init.c            |  30 +-----
 arch/sh/mm/init.c              |   8 +-
 arch/x86/mm/init_32.c          |   5 +-
 arch/x86/mm/init_64.c          |   9 +-
 drivers/base/memory.c          |  52 ++++++-----
 include/linux/memory_hotplug.h |  13 +--
 include/linux/mmzone.h         |  20 ++++
 kernel/memremap.c              |   4 +
 mm/memory_hotplug.c            | 204 +++++++++++++++++++++++++----------------
 mm/sparse.c                    |   3 +-
 12 files changed, 193 insertions(+), 174 deletions(-)

diff --git a/arch/ia64/mm/init.c b/arch/ia64/mm/init.c
index 39e2aeb4669d..80db57d063d0 100644
--- a/arch/ia64/mm/init.c
+++ b/arch/ia64/mm/init.c
@@ -648,18 +648,11 @@ mem_init (void)
 #ifdef CONFIG_MEMORY_HOTPLUG
 int arch_add_memory(int nid, u64 start, u64 size, bool for_device)
 {
-	pg_data_t *pgdat;
-	struct zone *zone;
 	unsigned long start_pfn = start >> PAGE_SHIFT;
 	unsigned long nr_pages = size >> PAGE_SHIFT;
 	int ret;
 
-	pgdat = NODE_DATA(nid);
-
-	zone = pgdat->node_zones +
-		zone_for_memory(nid, start, size, ZONE_NORMAL, for_device);
-	ret = __add_pages(nid, zone, start_pfn, nr_pages, !for_device);
-
+	ret = __add_pages(nid, start_pfn, nr_pages, !for_device);
 	if (ret)
 		printk("%s: Problem encountered in __add_pages() as ret=%d\n",
 		       __func__,  ret);
diff --git a/arch/powerpc/mm/mem.c b/arch/powerpc/mm/mem.c
index e6b2e6618b6c..72c46eb53215 100644
--- a/arch/powerpc/mm/mem.c
+++ b/arch/powerpc/mm/mem.c
@@ -128,16 +128,12 @@ int __weak remove_section_mapping(unsigned long start, unsigned long end)
 
 int arch_add_memory(int nid, u64 start, u64 size, bool for_device)
 {
-	struct pglist_data *pgdata;
-	struct zone *zone;
 	unsigned long start_pfn = start >> PAGE_SHIFT;
 	unsigned long nr_pages = size >> PAGE_SHIFT;
 	int rc;
 
 	resize_hpt_for_hotplug(memblock_phys_mem_size());
 
-	pgdata = NODE_DATA(nid);
-
 	start = (unsigned long)__va(start);
 	rc = create_section_mapping(start, start + size);
 	if (rc) {
@@ -147,11 +143,7 @@ int arch_add_memory(int nid, u64 start, u64 size, bool for_device)
 		return -EFAULT;
 	}
 
-	/* this should work for most non-highmem platforms */
-	zone = pgdata->node_zones +
-		zone_for_memory(nid, start, size, 0, for_device);
-
-	return __add_pages(nid, zone, start_pfn, nr_pages, !for_device);
+	return __add_pages(nid, start_pfn, nr_pages, !for_device);
 }
 
 #ifdef CONFIG_MEMORY_HOTREMOVE
diff --git a/arch/s390/mm/init.c b/arch/s390/mm/init.c
index 893cf88cf02d..862824924ba6 100644
--- a/arch/s390/mm/init.c
+++ b/arch/s390/mm/init.c
@@ -164,41 +164,15 @@ unsigned long memory_block_size_bytes(void)
 #ifdef CONFIG_MEMORY_HOTPLUG
 int arch_add_memory(int nid, u64 start, u64 size, bool for_device)
 {
-	unsigned long zone_start_pfn, zone_end_pfn, nr_pages;
 	unsigned long start_pfn = PFN_DOWN(start);
 	unsigned long size_pages = PFN_DOWN(size);
-	pg_data_t *pgdat = NODE_DATA(nid);
-	struct zone *zone;
-	int rc, i;
+	int rc;
 
 	rc = vmem_add_mapping(start, size);
 	if (rc)
 		return rc;
 
-	for (i = 0; i < MAX_NR_ZONES; i++) {
-		zone = pgdat->node_zones + i;
-		if (zone_idx(zone) != ZONE_MOVABLE) {
-			/* Add range within existing zone limits, if possible */
-			zone_start_pfn = zone->zone_start_pfn;
-			zone_end_pfn = zone->zone_start_pfn +
-				       zone->spanned_pages;
-		} else {
-			/* Add remaining range to ZONE_MOVABLE */
-			zone_start_pfn = start_pfn;
-			zone_end_pfn = start_pfn + size_pages;
-		}
-		if (start_pfn < zone_start_pfn || start_pfn >= zone_end_pfn)
-			continue;
-		nr_pages = (start_pfn + size_pages > zone_end_pfn) ?
-			   zone_end_pfn - start_pfn : size_pages;
-		rc = __add_pages(nid, zone, start_pfn, nr_pages, !for_device);
-		if (rc)
-			break;
-		start_pfn += nr_pages;
-		size_pages -= nr_pages;
-		if (!size_pages)
-			break;
-	}
+	rc = __add_pages(nid, start_pfn, size_pages, !for_device);
 	if (rc)
 		vmem_remove_mapping(start, size);
 	return rc;
diff --git a/arch/sh/mm/init.c b/arch/sh/mm/init.c
index a9d57f75ae8c..3813a610a2bb 100644
--- a/arch/sh/mm/init.c
+++ b/arch/sh/mm/init.c
@@ -487,18 +487,12 @@ void free_initrd_mem(unsigned long start, unsigned long end)
 #ifdef CONFIG_MEMORY_HOTPLUG
 int arch_add_memory(int nid, u64 start, u64 size, bool for_device)
 {
-	pg_data_t *pgdat;
 	unsigned long start_pfn = PFN_DOWN(start);
 	unsigned long nr_pages = size >> PAGE_SHIFT;
 	int ret;
 
-	pgdat = NODE_DATA(nid);
-
 	/* We only have ZONE_NORMAL, so this is easy.. */
-	ret = __add_pages(nid, pgdat->node_zones +
-			zone_for_memory(nid, start, size, ZONE_NORMAL,
-			for_device),
-			start_pfn, nr_pages, !for_device);
+	ret = __add_pages(nid, start_pfn, nr_pages, !for_device);
 	if (unlikely(ret))
 		printk("%s: Failed, __add_pages() == %d\n", __func__, ret);
 
diff --git a/arch/x86/mm/init_32.c b/arch/x86/mm/init_32.c
index 94594b889144..a424066d0552 100644
--- a/arch/x86/mm/init_32.c
+++ b/arch/x86/mm/init_32.c
@@ -825,13 +825,10 @@ void __init mem_init(void)
 #ifdef CONFIG_MEMORY_HOTPLUG
 int arch_add_memory(int nid, u64 start, u64 size, bool for_device)
 {
-	struct pglist_data *pgdata = NODE_DATA(nid);
-	struct zone *zone = pgdata->node_zones +
-		zone_for_memory(nid, start, size, ZONE_HIGHMEM, for_device);
 	unsigned long start_pfn = start >> PAGE_SHIFT;
 	unsigned long nr_pages = size >> PAGE_SHIFT;
 
-	return __add_pages(nid, zone, start_pfn, nr_pages, !for_device);
+	return __add_pages(nid, start_pfn, nr_pages, !for_device);
 }
 
 #ifdef CONFIG_MEMORY_HOTREMOVE
diff --git a/arch/x86/mm/init_64.c b/arch/x86/mm/init_64.c
index 2e004364a373..884c1d0a57b3 100644
--- a/arch/x86/mm/init_64.c
+++ b/arch/x86/mm/init_64.c
@@ -682,22 +682,15 @@ static void  update_end_of_memory_vars(u64 start, u64 size)
 	}
 }
 
-/*
- * Memory is added always to NORMAL zone. This means you will never get
- * additional DMA/DMA32 memory.
- */
 int arch_add_memory(int nid, u64 start, u64 size, bool for_device)
 {
-	struct pglist_data *pgdat = NODE_DATA(nid);
-	struct zone *zone = pgdat->node_zones +
-		zone_for_memory(nid, start, size, ZONE_NORMAL, for_device);
 	unsigned long start_pfn = start >> PAGE_SHIFT;
 	unsigned long nr_pages = size >> PAGE_SHIFT;
 	int ret;
 
 	init_memory_mapping(start, start + size);
 
-	ret = __add_pages(nid, zone, start_pfn, nr_pages, !for_device);
+	ret = __add_pages(nid, start_pfn, nr_pages, !for_device);
 	WARN_ON_ONCE(ret);
 
 	/* update max_pfn, max_low_pfn and high_memory */
diff --git a/drivers/base/memory.c b/drivers/base/memory.c
index 1e884d82af6f..b86fda30ce62 100644
--- a/drivers/base/memory.c
+++ b/drivers/base/memory.c
@@ -392,39 +392,43 @@ static ssize_t show_valid_zones(struct device *dev,
 				struct device_attribute *attr, char *buf)
 {
 	struct memory_block *mem = to_memory_block(dev);
-	unsigned long start_pfn, end_pfn;
-	unsigned long valid_start, valid_end, valid_pages;
+	unsigned long start_pfn = section_nr_to_pfn(mem->start_section_nr);
 	unsigned long nr_pages = PAGES_PER_SECTION * sections_per_block;
-	struct zone *zone;
-	int zone_shift = 0;
+	unsigned long valid_start_pfn, valid_end_pfn;
+	bool append = false;
+	int nid;
 
-	start_pfn = section_nr_to_pfn(mem->start_section_nr);
-	end_pfn = start_pfn + nr_pages;
-
-	/* The block contains more than one zone can not be offlined. */
-	if (!test_pages_in_a_zone(start_pfn, end_pfn, &valid_start, &valid_end))
+	/*
+	 * The block contains more than one zone can not be offlined.
+	 * This can happen e.g. for ZONE_DMA and ZONE_DMA32
+	 */
+	if (!test_pages_in_a_zone(start_pfn, start_pfn + nr_pages, &valid_start_pfn, &valid_end_pfn))
 		return sprintf(buf, "none\n");
 
-	zone = page_zone(pfn_to_page(valid_start));
-	valid_pages = valid_end - valid_start;
-
-	/* MMOP_ONLINE_KEEP */
-	sprintf(buf, "%s", zone->name);
+	start_pfn = valid_start_pfn;
+	nr_pages = valid_end_pfn - start_pfn;
 
-	/* MMOP_ONLINE_KERNEL */
-	zone_can_shift(valid_start, valid_pages, ZONE_NORMAL, &zone_shift);
-	if (zone_shift) {
-		strcat(buf, " ");
-		strcat(buf, (zone + zone_shift)->name);
+	/*
+	 * Check the existing zone. Make sure that we do that only on the
+	 * online nodes otherwise the page_zone is not reliable
+	 */
+	if (mem->state == MEM_ONLINE) {
+		strcat(buf, page_zone(pfn_to_page(start_pfn))->name);
+		goto out;
 	}
 
-	/* MMOP_ONLINE_MOVABLE */
-	zone_can_shift(valid_start, valid_pages, ZONE_MOVABLE, &zone_shift);
-	if (zone_shift) {
-		strcat(buf, " ");
-		strcat(buf, (zone + zone_shift)->name);
+	nid = pfn_to_nid(start_pfn);
+	if (allow_online_pfn_range(nid, start_pfn, nr_pages, MMOP_ONLINE_KERNEL)) {
+		strcat(buf, NODE_DATA(nid)->node_zones[ZONE_NORMAL].name);
+		append = true;
 	}
 
+	if (allow_online_pfn_range(nid, start_pfn, nr_pages, MMOP_ONLINE_MOVABLE)) {
+		if (append)
+			strcat(buf, " ");
+		strcat(buf, NODE_DATA(nid)->node_zones[ZONE_MOVABLE].name);
+	}
+out:
 	strcat(buf, "\n");
 
 	return strlen(buf);
diff --git a/include/linux/memory_hotplug.h b/include/linux/memory_hotplug.h
index fc1c873504eb..9cd76ff7b0c5 100644
--- a/include/linux/memory_hotplug.h
+++ b/include/linux/memory_hotplug.h
@@ -122,8 +122,8 @@ extern int __remove_pages(struct zone *zone, unsigned long start_pfn,
 	unsigned long nr_pages);
 #endif /* CONFIG_MEMORY_HOTREMOVE */
 
-/* reasonably generic interface to expand the physical pages in a zone  */
-extern int __add_pages(int nid, struct zone *zone, unsigned long start_pfn,
+/* reasonably generic interface to expand the physical pages */
+extern int __add_pages(int nid, unsigned long start_pfn,
 	unsigned long nr_pages, bool want_memblock);
 
 #ifdef CONFIG_NUMA
@@ -298,15 +298,16 @@ extern int add_memory_resource(int nid, struct resource *resource, bool online);
 extern int zone_for_memory(int nid, u64 start, u64 size, int zone_default,
 		bool for_device);
 extern int arch_add_memory(int nid, u64 start, u64 size, bool for_device);
+extern void move_pfn_range_to_zone(struct zone *zone, unsigned long start_pfn,
+		unsigned long nr_pages);
 extern int offline_pages(unsigned long start_pfn, unsigned long nr_pages);
 extern bool is_memblock_offlined(struct memory_block *mem);
 extern void remove_memory(int nid, u64 start, u64 size);
-extern int sparse_add_one_section(struct zone *zone, unsigned long start_pfn);
+extern int sparse_add_one_section(struct pglist_data *pgdat, unsigned long start_pfn);
 extern void sparse_remove_one_section(struct zone *zone, struct mem_section *ms,
 		unsigned long map_offset);
 extern struct page *sparse_decode_mem_map(unsigned long coded_mem_map,
 					  unsigned long pnum);
-extern bool zone_can_shift(unsigned long pfn, unsigned long nr_pages,
-			  enum zone_type target, int *zone_shift);
-
+extern bool allow_online_pfn_range(int nid, unsigned long pfn, unsigned long nr_pages,
+		int online_type);
 #endif /* __LINUX_MEMORY_HOTPLUG_H */
diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h
index 927ad95a4552..f887fccb6ef0 100644
--- a/include/linux/mmzone.h
+++ b/include/linux/mmzone.h
@@ -533,6 +533,26 @@ static inline bool zone_is_empty(struct zone *zone)
 }
 
 /*
+ * Return true if [start_pfn, start_pfn + nr_pages) range has a non-empty
+ * intersection with the given zone
+ */
+static inline bool zone_intersects(struct zone *zone,
+		unsigned long start_pfn, unsigned long nr_pages)
+{
+	if (zone_is_empty(zone))
+		return false;
+	if (start_pfn >= zone_end_pfn(zone))
+		return false;
+
+	if (zone->zone_start_pfn <= start_pfn)
+		return true;
+	if (start_pfn + nr_pages > zone->zone_start_pfn)
+		return true;
+
+	return false;
+}
+
+/*
  * The "priority" of VM scanning is how much of the queues we will scan in one
  * go. A value of 12 for DEF_PRIORITY implies that we will scan 1/4096th of the
  * queues ("queue_length >> 12") during an aging round.
diff --git a/kernel/memremap.c b/kernel/memremap.c
index 23a6483c3666..281eb478856a 100644
--- a/kernel/memremap.c
+++ b/kernel/memremap.c
@@ -359,6 +359,10 @@ void *devm_memremap_pages(struct device *dev, struct resource *res,
 
 	mem_hotplug_begin();
 	error = arch_add_memory(nid, align_start, align_size, true);
+	if (!error)
+		move_pfn_range_to_zone(&NODE_DATA(nid)->node_zones[ZONE_DEVICE],
+					align_start >> PAGE_SHIFT,
+					align_size >> PAGE_SHIFT);
 	mem_hotplug_done();
 	if (error)
 		goto err_add_memory;
diff --git a/mm/memory_hotplug.c b/mm/memory_hotplug.c
index c3a146028ba6..dc363370e9d8 100644
--- a/mm/memory_hotplug.c
+++ b/mm/memory_hotplug.c
@@ -433,25 +433,6 @@ static int __meminit move_pfn_range_right(struct zone *z1, struct zone *z2,
 	return -1;
 }
 
-static struct zone * __meminit move_pfn_range(int zone_shift,
-		unsigned long start_pfn, unsigned long end_pfn)
-{
-	struct zone *zone = page_zone(pfn_to_page(start_pfn));
-	int ret = 0;
-
-	if (zone_shift < 0)
-		ret = move_pfn_range_left(zone + zone_shift, zone,
-					  start_pfn, end_pfn);
-	else if (zone_shift)
-		ret = move_pfn_range_right(zone, zone + zone_shift,
-					   start_pfn, end_pfn);
-
-	if (ret)
-		return NULL;
-
-	return zone + zone_shift;
-}
-
 static void __meminit grow_pgdat_span(struct pglist_data *pgdat, unsigned long start_pfn,
 				      unsigned long end_pfn)
 {
@@ -493,23 +474,35 @@ static int __meminit __add_zone(struct zone *zone, unsigned long phys_start_pfn)
 	return 0;
 }
 
-static int __meminit __add_section(int nid, struct zone *zone,
-		unsigned long phys_start_pfn, bool want_memblock)
+static int __meminit __add_section(int nid, unsigned long phys_start_pfn,
+		bool want_memblock)
 {
 	int ret;
+	int i;
 
 	if (pfn_valid(phys_start_pfn))
 		return -EEXIST;
 
-	ret = sparse_add_one_section(zone, phys_start_pfn);
-
+	ret = sparse_add_one_section(NODE_DATA(nid), phys_start_pfn);
 	if (ret < 0)
 		return ret;
 
-	ret = __add_zone(zone, phys_start_pfn);
+	/*
+	 * Make all the pages reserved so that nobody will stumble over half
+	 * initialized state.
+	 * FIXME: We also have to associate it with a node because pfn_to_node
+	 * relies on having page with the proper node.
+	 */
+	for (i = 0; i < PAGES_PER_SECTION; i++) {
+		unsigned long pfn = phys_start_pfn + i;
+		struct page *page;
+		if (!pfn_valid(pfn))
+			continue;
 
-	if (ret < 0)
-		return ret;
+		page = pfn_to_page(pfn);
+		set_page_node(page, nid);
+		SetPageReserved(page);
+	}
 
 	if (!want_memblock)
 		return 0;
@@ -523,7 +516,7 @@ static int __meminit __add_section(int nid, struct zone *zone,
  * call this function after deciding the zone to which to
  * add the new pages.
  */
-int __ref __add_pages(int nid, struct zone *zone, unsigned long phys_start_pfn,
+int __ref __add_pages(int nid, unsigned long phys_start_pfn,
 			unsigned long nr_pages, bool want_memblock)
 {
 	unsigned long i;
@@ -531,8 +524,6 @@ int __ref __add_pages(int nid, struct zone *zone, unsigned long phys_start_pfn,
 	int start_sec, end_sec;
 	struct vmem_altmap *altmap;
 
-	clear_zone_contiguous(zone);
-
 	/* during initialize mem_map, align hot-added range to section */
 	start_sec = pfn_to_section_nr(phys_start_pfn);
 	end_sec = pfn_to_section_nr(phys_start_pfn + nr_pages - 1);
@@ -552,7 +543,7 @@ int __ref __add_pages(int nid, struct zone *zone, unsigned long phys_start_pfn,
 	}
 
 	for (i = start_sec; i <= end_sec; i++) {
-		err = __add_section(nid, zone, section_nr_to_pfn(i), want_memblock);
+		err = __add_section(nid, section_nr_to_pfn(i), want_memblock);
 
 		/*
 		 * EEXIST is finally dealt with by ioresource collision
@@ -565,7 +556,6 @@ int __ref __add_pages(int nid, struct zone *zone, unsigned long phys_start_pfn,
 	}
 	vmemmap_populate_print_last();
 out:
-	set_zone_contiguous(zone);
 	return err;
 }
 EXPORT_SYMBOL_GPL(__add_pages);
@@ -1037,39 +1027,114 @@ static void node_states_set_node(int node, struct memory_notify *arg)
 	node_set_state(node, N_MEMORY);
 }
 
-bool zone_can_shift(unsigned long pfn, unsigned long nr_pages,
-		   enum zone_type target, int *zone_shift)
+bool allow_online_pfn_range(int nid, unsigned long pfn, unsigned long nr_pages, int online_type)
 {
-	struct zone *zone = page_zone(pfn_to_page(pfn));
-	enum zone_type idx = zone_idx(zone);
-	int i;
+	struct pglist_data *pgdat = NODE_DATA(nid);
+	struct zone *movable_zone = &pgdat->node_zones[ZONE_MOVABLE];
+	struct zone *normal_zone =  &pgdat->node_zones[ZONE_NORMAL];
 
-	*zone_shift = 0;
+	/*
+	 * TODO there shouldn't be any inherent reason to have ZONE_NORMAL
+	 * physically before ZONE_MOVABLE. All we need is they do not
+	 * overlap. Historically we didn't allow ZONE_NORMAL after ZONE_MOVABLE
+	 * though so let's stick with it for simplicity for now.
+	 * TODO make sure we do not overlap with ZONE_DEVICE
+	 */
+	if (online_type == MMOP_ONLINE_KERNEL) {
+		if (zone_is_empty(movable_zone))
+			return true;
+		return movable_zone->zone_start_pfn >= pfn + nr_pages;
+	} else if (online_type == MMOP_ONLINE_MOVABLE) {
+		return zone_end_pfn(normal_zone) <= pfn;
+	}
 
-	if (idx < target) {
-		/* pages must be at end of current zone */
-		if (pfn + nr_pages != zone_end_pfn(zone))
-			return false;
+	/* MMOP_ONLINE_KEEP will always succeed and inherits the current zone */
+	return online_type == MMOP_ONLINE_KEEP;
+}
+
+static void __meminit resize_zone_range(struct zone *zone, unsigned long start_pfn,
+		unsigned long nr_pages)
+{
+	unsigned long old_end_pfn = zone_end_pfn(zone);
+
+	if (zone_is_empty(zone) || start_pfn < zone->zone_start_pfn)
+		zone->zone_start_pfn = start_pfn;
+
+	zone->spanned_pages = max(start_pfn + nr_pages, old_end_pfn) - zone->zone_start_pfn;
+}
+
+static void __meminit resize_pgdat_range(struct pglist_data *pgdat, unsigned long start_pfn,
+                                     unsigned long nr_pages)
+{
+	unsigned long old_end_pfn = pgdat_end_pfn(pgdat);
 
-		/* no zones in use between current zone and target */
-		for (i = idx + 1; i < target; i++)
-			if (zone_is_initialized(zone - idx + i))
-				return false;
+	if (!pgdat->node_spanned_pages || start_pfn < pgdat->node_start_pfn)
+		pgdat->node_start_pfn = start_pfn;
+
+	pgdat->node_spanned_pages = max(start_pfn + nr_pages, old_end_pfn) - pgdat->node_start_pfn;
+}
+
+void move_pfn_range_to_zone(struct zone *zone,
+		unsigned long start_pfn, unsigned long nr_pages)
+{
+	struct pglist_data *pgdat = zone->zone_pgdat;
+	int nid = pgdat->node_id;
+	unsigned long flags;
+	unsigned long i;
+
+	if (zone_is_empty(zone))
+		init_currently_empty_zone(zone, start_pfn, nr_pages);
+
+	clear_zone_contiguous(zone);
+
+	/* TODO Huh pgdat is irqsave while zone is not. It used to be like that before */
+	pgdat_resize_lock(pgdat, &flags);
+	zone_span_writelock(zone);
+	resize_zone_range(zone, start_pfn, nr_pages);
+	zone_span_writeunlock(zone);
+	resize_pgdat_range(pgdat, start_pfn, nr_pages);
+	pgdat_resize_unlock(pgdat, &flags);
+
+	/*
+	 * TODO now we have a visible range of pages which are not associated
+	 * with their zone properly. Not nice but set_pfnblock_flags_mask
+	 * expects the zone spans the pfn range. All the pages in the range
+	 * are reserved so nobody should be touching them so we should be safe
+	 */
+	memmap_init_zone(nr_pages, nid, zone_idx(zone), start_pfn, MEMMAP_HOTPLUG);
+	for (i = 0; i < nr_pages; i++) {
+		unsigned long pfn = start_pfn + i;
+		set_page_links(pfn_to_page(pfn), zone_idx(zone), nid, pfn);
 	}
 
-	if (target < idx) {
-		/* pages must be at beginning of current zone */
-		if (pfn != zone->zone_start_pfn)
-			return false;
+	set_zone_contiguous(zone);
+}
+
+/*
+ * Associates the given pfn range with the given node and the zone appropriate
+ * for the given online type.
+ */
+static struct zone * __meminit move_pfn_range(int online_type, int nid,
+		unsigned long start_pfn, unsigned long nr_pages)
+{
+	struct pglist_data *pgdat = NODE_DATA(nid);
+	struct zone *zone = &pgdat->node_zones[ZONE_NORMAL];
 
-		/* no zones in use between current zone and target */
-		for (i = target + 1; i < idx; i++)
-			if (zone_is_initialized(zone - idx + i))
-				return false;
+	if (online_type == MMOP_ONLINE_KEEP) {
+		struct zone *movable_zone = &pgdat->node_zones[ZONE_MOVABLE];
+		/*
+		 * MMOP_ONLINE_KEEP inherits the current zone which is
+		 * ZONE_NORMAL by default but we might be within ZONE_MOVABLE
+		 * already.
+		 */
+		if (zone_intersects(movable_zone, start_pfn, nr_pages))
+			zone = movable_zone;
+	} else if (online_type == MMOP_ONLINE_MOVABLE) {
+		zone = &pgdat->node_zones[ZONE_MOVABLE];
 	}
 
-	*zone_shift = target - idx;
-	return true;
+	move_pfn_range_to_zone(zone, start_pfn, nr_pages);
+	return zone;
 }
 
 /* Must be protected by mem_hotplug_begin() */
@@ -1082,38 +1147,21 @@ int __ref online_pages(unsigned long pfn, unsigned long nr_pages, int online_typ
 	int nid;
 	int ret;
 	struct memory_notify arg;
-	int zone_shift = 0;
 
-	/*
-	 * This doesn't need a lock to do pfn_to_page().
-	 * The section can't be removed here because of the
-	 * memory_block->state_mutex.
-	 */
-	zone = page_zone(pfn_to_page(pfn));
-
-	if ((zone_idx(zone) > ZONE_NORMAL ||
-	    online_type == MMOP_ONLINE_MOVABLE) &&
-	    !can_online_high_movable(pfn_to_nid(pfn)))
+	nid = pfn_to_nid(pfn);
+	if (!allow_online_pfn_range(nid, pfn, nr_pages, online_type))
 		return -EINVAL;
 
-	if (online_type == MMOP_ONLINE_KERNEL) {
-		if (!zone_can_shift(pfn, nr_pages, ZONE_NORMAL, &zone_shift))
-			return -EINVAL;
-	} else if (online_type == MMOP_ONLINE_MOVABLE) {
-		if (!zone_can_shift(pfn, nr_pages, ZONE_MOVABLE, &zone_shift))
-			return -EINVAL;
-	}
-
-	zone = move_pfn_range(zone_shift, pfn, pfn + nr_pages);
-	if (!zone)
+	if (online_type == MMOP_ONLINE_MOVABLE && !can_online_high_movable(nid))
 		return -EINVAL;
 
+	/* associate pfn range with the zone */
+	zone = move_pfn_range(online_type, nid, pfn, nr_pages);
+
 	arg.start_pfn = pfn;
 	arg.nr_pages = nr_pages;
 	node_states_check_changes_online(nr_pages, zone, &arg);
 
-	nid = zone_to_nid(zone);
-
 	ret = memory_notify(MEM_GOING_ONLINE, &arg);
 	ret = notifier_to_errno(ret);
 	if (ret)
diff --git a/mm/sparse.c b/mm/sparse.c
index 9d7fd666015e..7b4be3fd5cac 100644
--- a/mm/sparse.c
+++ b/mm/sparse.c
@@ -761,10 +761,9 @@ static void free_map_bootmem(struct page *memmap)
  * set.  If this is <=0, then that means that the passed-in
  * map was not consumed and must be freed.
  */
-int __meminit sparse_add_one_section(struct zone *zone, unsigned long start_pfn)
+int __meminit sparse_add_one_section(struct pglist_data *pgdat, unsigned long start_pfn)
 {
 	unsigned long section_nr = pfn_to_section_nr(start_pfn);
-	struct pglist_data *pgdat = zone->zone_pgdat;
 	struct mem_section *ms;
 	struct page *memmap;
 	unsigned long *usemap;
-- 
2.11.0

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

^ permalink raw reply	[flat|nested] 74+ messages in thread

* [PATCH 12/14] mm, memory_hotplug: replace for_device by want_memblock in arch_add_memory
  2017-05-15  8:58 ` Michal Hocko
@ 2017-05-15  8:58   ` Michal Hocko
  -1 siblings, 0 replies; 74+ messages in thread
From: Michal Hocko @ 2017-05-15  8:58 UTC (permalink / raw)
  To: Andrew Morton
  Cc: linux-mm, Mel Gorman, Vlastimil Babka, Andrea Arcangeli,
	Jerome Glisse, Reza Arbab, Yasuaki Ishimatsu, qiuxishi,
	Kani Toshimitsu, slaoub, Joonsoo Kim, Andi Kleen, David Rientjes,
	Daniel Kiper, Igor Mammedov, Vitaly Kuznetsov, LKML,
	Michal Hocko, Dan Williams

From: Michal Hocko <mhocko@suse.com>

arch_add_memory gets for_device argument which then controls whether we
want to create memblocks for created memory sections. Simplify the logic
by telling whether we want memblocks directly rather than going through
pointless negation. This also makes the api easier to understand because
it is clear what we want rather than nothing telling for_device which
can mean anything.

This shouldn't introduce any functional change.

Tested-by: Dan Williams <dan.j.williams@intel.com>
Acked-by: Vlastimil Babka <vbabka@suse.cz>
Signed-off-by: Michal Hocko <mhocko@suse.com>
---
 arch/ia64/mm/init.c            | 4 ++--
 arch/powerpc/mm/mem.c          | 4 ++--
 arch/s390/mm/init.c            | 4 ++--
 arch/sh/mm/init.c              | 4 ++--
 arch/x86/mm/init_32.c          | 4 ++--
 arch/x86/mm/init_64.c          | 4 ++--
 include/linux/memory_hotplug.h | 2 +-
 kernel/memremap.c              | 2 +-
 mm/memory_hotplug.c            | 2 +-
 9 files changed, 15 insertions(+), 15 deletions(-)

diff --git a/arch/ia64/mm/init.c b/arch/ia64/mm/init.c
index 80db57d063d0..a4e8d6bd9cfa 100644
--- a/arch/ia64/mm/init.c
+++ b/arch/ia64/mm/init.c
@@ -646,13 +646,13 @@ mem_init (void)
 }
 
 #ifdef CONFIG_MEMORY_HOTPLUG
-int arch_add_memory(int nid, u64 start, u64 size, bool for_device)
+int arch_add_memory(int nid, u64 start, u64 size, bool want_memblock)
 {
 	unsigned long start_pfn = start >> PAGE_SHIFT;
 	unsigned long nr_pages = size >> PAGE_SHIFT;
 	int ret;
 
-	ret = __add_pages(nid, start_pfn, nr_pages, !for_device);
+	ret = __add_pages(nid, start_pfn, nr_pages, want_memblock);
 	if (ret)
 		printk("%s: Problem encountered in __add_pages() as ret=%d\n",
 		       __func__,  ret);
diff --git a/arch/powerpc/mm/mem.c b/arch/powerpc/mm/mem.c
index 72c46eb53215..de5a90e1ceaa 100644
--- a/arch/powerpc/mm/mem.c
+++ b/arch/powerpc/mm/mem.c
@@ -126,7 +126,7 @@ int __weak remove_section_mapping(unsigned long start, unsigned long end)
 	return -ENODEV;
 }
 
-int arch_add_memory(int nid, u64 start, u64 size, bool for_device)
+int arch_add_memory(int nid, u64 start, u64 size, bool want_memblock)
 {
 	unsigned long start_pfn = start >> PAGE_SHIFT;
 	unsigned long nr_pages = size >> PAGE_SHIFT;
@@ -143,7 +143,7 @@ int arch_add_memory(int nid, u64 start, u64 size, bool for_device)
 		return -EFAULT;
 	}
 
-	return __add_pages(nid, start_pfn, nr_pages, !for_device);
+	return __add_pages(nid, start_pfn, nr_pages, want_memblock);
 }
 
 #ifdef CONFIG_MEMORY_HOTREMOVE
diff --git a/arch/s390/mm/init.c b/arch/s390/mm/init.c
index 862824924ba6..f4fb5d191562 100644
--- a/arch/s390/mm/init.c
+++ b/arch/s390/mm/init.c
@@ -162,7 +162,7 @@ unsigned long memory_block_size_bytes(void)
 }
 
 #ifdef CONFIG_MEMORY_HOTPLUG
-int arch_add_memory(int nid, u64 start, u64 size, bool for_device)
+int arch_add_memory(int nid, u64 start, u64 size, bool want_memblock)
 {
 	unsigned long start_pfn = PFN_DOWN(start);
 	unsigned long size_pages = PFN_DOWN(size);
@@ -172,7 +172,7 @@ int arch_add_memory(int nid, u64 start, u64 size, bool for_device)
 	if (rc)
 		return rc;
 
-	rc = __add_pages(nid, start_pfn, size_pages, !for_device);
+	rc = __add_pages(nid, start_pfn, size_pages, want_memblock);
 	if (rc)
 		vmem_remove_mapping(start, size);
 	return rc;
diff --git a/arch/sh/mm/init.c b/arch/sh/mm/init.c
index 3813a610a2bb..bf726af5f1a5 100644
--- a/arch/sh/mm/init.c
+++ b/arch/sh/mm/init.c
@@ -485,14 +485,14 @@ void free_initrd_mem(unsigned long start, unsigned long end)
 #endif
 
 #ifdef CONFIG_MEMORY_HOTPLUG
-int arch_add_memory(int nid, u64 start, u64 size, bool for_device)
+int arch_add_memory(int nid, u64 start, u64 size, bool want_memblock)
 {
 	unsigned long start_pfn = PFN_DOWN(start);
 	unsigned long nr_pages = size >> PAGE_SHIFT;
 	int ret;
 
 	/* We only have ZONE_NORMAL, so this is easy.. */
-	ret = __add_pages(nid, start_pfn, nr_pages, !for_device);
+	ret = __add_pages(nid, start_pfn, nr_pages, want_memblock);
 	if (unlikely(ret))
 		printk("%s: Failed, __add_pages() == %d\n", __func__, ret);
 
diff --git a/arch/x86/mm/init_32.c b/arch/x86/mm/init_32.c
index a424066d0552..8a64a6f2848d 100644
--- a/arch/x86/mm/init_32.c
+++ b/arch/x86/mm/init_32.c
@@ -823,12 +823,12 @@ void __init mem_init(void)
 }
 
 #ifdef CONFIG_MEMORY_HOTPLUG
-int arch_add_memory(int nid, u64 start, u64 size, bool for_device)
+int arch_add_memory(int nid, u64 start, u64 size, bool want_memblock)
 {
 	unsigned long start_pfn = start >> PAGE_SHIFT;
 	unsigned long nr_pages = size >> PAGE_SHIFT;
 
-	return __add_pages(nid, start_pfn, nr_pages, !for_device);
+	return __add_pages(nid, start_pfn, nr_pages, want_memblock);
 }
 
 #ifdef CONFIG_MEMORY_HOTREMOVE
diff --git a/arch/x86/mm/init_64.c b/arch/x86/mm/init_64.c
index 884c1d0a57b3..88e04804e5a8 100644
--- a/arch/x86/mm/init_64.c
+++ b/arch/x86/mm/init_64.c
@@ -682,7 +682,7 @@ static void  update_end_of_memory_vars(u64 start, u64 size)
 	}
 }
 
-int arch_add_memory(int nid, u64 start, u64 size, bool for_device)
+int arch_add_memory(int nid, u64 start, u64 size, bool want_memblock)
 {
 	unsigned long start_pfn = start >> PAGE_SHIFT;
 	unsigned long nr_pages = size >> PAGE_SHIFT;
@@ -690,7 +690,7 @@ int arch_add_memory(int nid, u64 start, u64 size, bool for_device)
 
 	init_memory_mapping(start, start + size);
 
-	ret = __add_pages(nid, start_pfn, nr_pages, !for_device);
+	ret = __add_pages(nid, start_pfn, nr_pages, want_memblock);
 	WARN_ON_ONCE(ret);
 
 	/* update max_pfn, max_low_pfn and high_memory */
diff --git a/include/linux/memory_hotplug.h b/include/linux/memory_hotplug.h
index 9cd76ff7b0c5..e1193c44aed1 100644
--- a/include/linux/memory_hotplug.h
+++ b/include/linux/memory_hotplug.h
@@ -297,7 +297,7 @@ extern int add_memory(int nid, u64 start, u64 size);
 extern int add_memory_resource(int nid, struct resource *resource, bool online);
 extern int zone_for_memory(int nid, u64 start, u64 size, int zone_default,
 		bool for_device);
-extern int arch_add_memory(int nid, u64 start, u64 size, bool for_device);
+extern int arch_add_memory(int nid, u64 start, u64 size, bool want_memblock);
 extern void move_pfn_range_to_zone(struct zone *zone, unsigned long start_pfn,
 		unsigned long nr_pages);
 extern int offline_pages(unsigned long start_pfn, unsigned long nr_pages);
diff --git a/kernel/memremap.c b/kernel/memremap.c
index 281eb478856a..124bed776532 100644
--- a/kernel/memremap.c
+++ b/kernel/memremap.c
@@ -358,7 +358,7 @@ void *devm_memremap_pages(struct device *dev, struct resource *res,
 		goto err_pfn_remap;
 
 	mem_hotplug_begin();
-	error = arch_add_memory(nid, align_start, align_size, true);
+	error = arch_add_memory(nid, align_start, align_size, false);
 	if (!error)
 		move_pfn_range_to_zone(&NODE_DATA(nid)->node_zones[ZONE_DEVICE],
 					align_start >> PAGE_SHIFT,
diff --git a/mm/memory_hotplug.c b/mm/memory_hotplug.c
index dc363370e9d8..497898943212 100644
--- a/mm/memory_hotplug.c
+++ b/mm/memory_hotplug.c
@@ -1434,7 +1434,7 @@ int __ref add_memory_resource(int nid, struct resource *res, bool online)
 	}
 
 	/* call arch's memory hotadd */
-	ret = arch_add_memory(nid, start, size, false);
+	ret = arch_add_memory(nid, start, size, true);
 
 	if (ret < 0)
 		goto error;
-- 
2.11.0

^ permalink raw reply	[flat|nested] 74+ messages in thread

* [PATCH 12/14] mm, memory_hotplug: replace for_device by want_memblock in arch_add_memory
@ 2017-05-15  8:58   ` Michal Hocko
  0 siblings, 0 replies; 74+ messages in thread
From: Michal Hocko @ 2017-05-15  8:58 UTC (permalink / raw)
  To: Andrew Morton
  Cc: linux-mm, Mel Gorman, Vlastimil Babka, Andrea Arcangeli,
	Jerome Glisse, Reza Arbab, Yasuaki Ishimatsu, qiuxishi,
	Kani Toshimitsu, slaoub, Joonsoo Kim, Andi Kleen, David Rientjes,
	Daniel Kiper, Igor Mammedov, Vitaly Kuznetsov, LKML,
	Michal Hocko, Dan Williams

From: Michal Hocko <mhocko@suse.com>

arch_add_memory gets for_device argument which then controls whether we
want to create memblocks for created memory sections. Simplify the logic
by telling whether we want memblocks directly rather than going through
pointless negation. This also makes the api easier to understand because
it is clear what we want rather than nothing telling for_device which
can mean anything.

This shouldn't introduce any functional change.

Tested-by: Dan Williams <dan.j.williams@intel.com>
Acked-by: Vlastimil Babka <vbabka@suse.cz>
Signed-off-by: Michal Hocko <mhocko@suse.com>
---
 arch/ia64/mm/init.c            | 4 ++--
 arch/powerpc/mm/mem.c          | 4 ++--
 arch/s390/mm/init.c            | 4 ++--
 arch/sh/mm/init.c              | 4 ++--
 arch/x86/mm/init_32.c          | 4 ++--
 arch/x86/mm/init_64.c          | 4 ++--
 include/linux/memory_hotplug.h | 2 +-
 kernel/memremap.c              | 2 +-
 mm/memory_hotplug.c            | 2 +-
 9 files changed, 15 insertions(+), 15 deletions(-)

diff --git a/arch/ia64/mm/init.c b/arch/ia64/mm/init.c
index 80db57d063d0..a4e8d6bd9cfa 100644
--- a/arch/ia64/mm/init.c
+++ b/arch/ia64/mm/init.c
@@ -646,13 +646,13 @@ mem_init (void)
 }
 
 #ifdef CONFIG_MEMORY_HOTPLUG
-int arch_add_memory(int nid, u64 start, u64 size, bool for_device)
+int arch_add_memory(int nid, u64 start, u64 size, bool want_memblock)
 {
 	unsigned long start_pfn = start >> PAGE_SHIFT;
 	unsigned long nr_pages = size >> PAGE_SHIFT;
 	int ret;
 
-	ret = __add_pages(nid, start_pfn, nr_pages, !for_device);
+	ret = __add_pages(nid, start_pfn, nr_pages, want_memblock);
 	if (ret)
 		printk("%s: Problem encountered in __add_pages() as ret=%d\n",
 		       __func__,  ret);
diff --git a/arch/powerpc/mm/mem.c b/arch/powerpc/mm/mem.c
index 72c46eb53215..de5a90e1ceaa 100644
--- a/arch/powerpc/mm/mem.c
+++ b/arch/powerpc/mm/mem.c
@@ -126,7 +126,7 @@ int __weak remove_section_mapping(unsigned long start, unsigned long end)
 	return -ENODEV;
 }
 
-int arch_add_memory(int nid, u64 start, u64 size, bool for_device)
+int arch_add_memory(int nid, u64 start, u64 size, bool want_memblock)
 {
 	unsigned long start_pfn = start >> PAGE_SHIFT;
 	unsigned long nr_pages = size >> PAGE_SHIFT;
@@ -143,7 +143,7 @@ int arch_add_memory(int nid, u64 start, u64 size, bool for_device)
 		return -EFAULT;
 	}
 
-	return __add_pages(nid, start_pfn, nr_pages, !for_device);
+	return __add_pages(nid, start_pfn, nr_pages, want_memblock);
 }
 
 #ifdef CONFIG_MEMORY_HOTREMOVE
diff --git a/arch/s390/mm/init.c b/arch/s390/mm/init.c
index 862824924ba6..f4fb5d191562 100644
--- a/arch/s390/mm/init.c
+++ b/arch/s390/mm/init.c
@@ -162,7 +162,7 @@ unsigned long memory_block_size_bytes(void)
 }
 
 #ifdef CONFIG_MEMORY_HOTPLUG
-int arch_add_memory(int nid, u64 start, u64 size, bool for_device)
+int arch_add_memory(int nid, u64 start, u64 size, bool want_memblock)
 {
 	unsigned long start_pfn = PFN_DOWN(start);
 	unsigned long size_pages = PFN_DOWN(size);
@@ -172,7 +172,7 @@ int arch_add_memory(int nid, u64 start, u64 size, bool for_device)
 	if (rc)
 		return rc;
 
-	rc = __add_pages(nid, start_pfn, size_pages, !for_device);
+	rc = __add_pages(nid, start_pfn, size_pages, want_memblock);
 	if (rc)
 		vmem_remove_mapping(start, size);
 	return rc;
diff --git a/arch/sh/mm/init.c b/arch/sh/mm/init.c
index 3813a610a2bb..bf726af5f1a5 100644
--- a/arch/sh/mm/init.c
+++ b/arch/sh/mm/init.c
@@ -485,14 +485,14 @@ void free_initrd_mem(unsigned long start, unsigned long end)
 #endif
 
 #ifdef CONFIG_MEMORY_HOTPLUG
-int arch_add_memory(int nid, u64 start, u64 size, bool for_device)
+int arch_add_memory(int nid, u64 start, u64 size, bool want_memblock)
 {
 	unsigned long start_pfn = PFN_DOWN(start);
 	unsigned long nr_pages = size >> PAGE_SHIFT;
 	int ret;
 
 	/* We only have ZONE_NORMAL, so this is easy.. */
-	ret = __add_pages(nid, start_pfn, nr_pages, !for_device);
+	ret = __add_pages(nid, start_pfn, nr_pages, want_memblock);
 	if (unlikely(ret))
 		printk("%s: Failed, __add_pages() == %d\n", __func__, ret);
 
diff --git a/arch/x86/mm/init_32.c b/arch/x86/mm/init_32.c
index a424066d0552..8a64a6f2848d 100644
--- a/arch/x86/mm/init_32.c
+++ b/arch/x86/mm/init_32.c
@@ -823,12 +823,12 @@ void __init mem_init(void)
 }
 
 #ifdef CONFIG_MEMORY_HOTPLUG
-int arch_add_memory(int nid, u64 start, u64 size, bool for_device)
+int arch_add_memory(int nid, u64 start, u64 size, bool want_memblock)
 {
 	unsigned long start_pfn = start >> PAGE_SHIFT;
 	unsigned long nr_pages = size >> PAGE_SHIFT;
 
-	return __add_pages(nid, start_pfn, nr_pages, !for_device);
+	return __add_pages(nid, start_pfn, nr_pages, want_memblock);
 }
 
 #ifdef CONFIG_MEMORY_HOTREMOVE
diff --git a/arch/x86/mm/init_64.c b/arch/x86/mm/init_64.c
index 884c1d0a57b3..88e04804e5a8 100644
--- a/arch/x86/mm/init_64.c
+++ b/arch/x86/mm/init_64.c
@@ -682,7 +682,7 @@ static void  update_end_of_memory_vars(u64 start, u64 size)
 	}
 }
 
-int arch_add_memory(int nid, u64 start, u64 size, bool for_device)
+int arch_add_memory(int nid, u64 start, u64 size, bool want_memblock)
 {
 	unsigned long start_pfn = start >> PAGE_SHIFT;
 	unsigned long nr_pages = size >> PAGE_SHIFT;
@@ -690,7 +690,7 @@ int arch_add_memory(int nid, u64 start, u64 size, bool for_device)
 
 	init_memory_mapping(start, start + size);
 
-	ret = __add_pages(nid, start_pfn, nr_pages, !for_device);
+	ret = __add_pages(nid, start_pfn, nr_pages, want_memblock);
 	WARN_ON_ONCE(ret);
 
 	/* update max_pfn, max_low_pfn and high_memory */
diff --git a/include/linux/memory_hotplug.h b/include/linux/memory_hotplug.h
index 9cd76ff7b0c5..e1193c44aed1 100644
--- a/include/linux/memory_hotplug.h
+++ b/include/linux/memory_hotplug.h
@@ -297,7 +297,7 @@ extern int add_memory(int nid, u64 start, u64 size);
 extern int add_memory_resource(int nid, struct resource *resource, bool online);
 extern int zone_for_memory(int nid, u64 start, u64 size, int zone_default,
 		bool for_device);
-extern int arch_add_memory(int nid, u64 start, u64 size, bool for_device);
+extern int arch_add_memory(int nid, u64 start, u64 size, bool want_memblock);
 extern void move_pfn_range_to_zone(struct zone *zone, unsigned long start_pfn,
 		unsigned long nr_pages);
 extern int offline_pages(unsigned long start_pfn, unsigned long nr_pages);
diff --git a/kernel/memremap.c b/kernel/memremap.c
index 281eb478856a..124bed776532 100644
--- a/kernel/memremap.c
+++ b/kernel/memremap.c
@@ -358,7 +358,7 @@ void *devm_memremap_pages(struct device *dev, struct resource *res,
 		goto err_pfn_remap;
 
 	mem_hotplug_begin();
-	error = arch_add_memory(nid, align_start, align_size, true);
+	error = arch_add_memory(nid, align_start, align_size, false);
 	if (!error)
 		move_pfn_range_to_zone(&NODE_DATA(nid)->node_zones[ZONE_DEVICE],
 					align_start >> PAGE_SHIFT,
diff --git a/mm/memory_hotplug.c b/mm/memory_hotplug.c
index dc363370e9d8..497898943212 100644
--- a/mm/memory_hotplug.c
+++ b/mm/memory_hotplug.c
@@ -1434,7 +1434,7 @@ int __ref add_memory_resource(int nid, struct resource *res, bool online)
 	}
 
 	/* call arch's memory hotadd */
-	ret = arch_add_memory(nid, start, size, false);
+	ret = arch_add_memory(nid, start, size, true);
 
 	if (ret < 0)
 		goto error;
-- 
2.11.0

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

^ permalink raw reply	[flat|nested] 74+ messages in thread

* [PATCH 13/14] mm, memory_hotplug: fix the section mismatch warning
  2017-05-15  8:58 ` Michal Hocko
@ 2017-05-15  8:58   ` Michal Hocko
  -1 siblings, 0 replies; 74+ messages in thread
From: Michal Hocko @ 2017-05-15  8:58 UTC (permalink / raw)
  To: Andrew Morton
  Cc: linux-mm, Mel Gorman, Vlastimil Babka, Andrea Arcangeli,
	Jerome Glisse, Reza Arbab, Yasuaki Ishimatsu, qiuxishi,
	Kani Toshimitsu, slaoub, Joonsoo Kim, Andi Kleen, David Rientjes,
	Daniel Kiper, Igor Mammedov, Vitaly Kuznetsov, LKML,
	Michal Hocko, Tobias Regnery

From: Michal Hocko <mhocko@suse.com>

Tobias has reported following section mismatches introduced by "mm,
memory_hotplug: do not associate hotadded memory to zones until online".

WARNING: mm/built-in.o(.text+0x5a1c2): Section mismatch in reference from the function move_pfn_range_to_zone() to the function .meminit.text:memmap_init_zone()
The function move_pfn_range_to_zone() references
the function __meminit memmap_init_zone().
This is often because move_pfn_range_to_zone lacks a __meminit
annotation or the annotation of memmap_init_zone is wrong.

WARNING: mm/built-in.o(.text+0x5a25b): Section mismatch in reference from the function move_pfn_range_to_zone() to the function .meminit.text:init_currently_empty_zone()
The function move_pfn_range_to_zone() references
the function __meminit init_currently_empty_zone().
This is often because move_pfn_range_to_zone lacks a __meminit
annotation or the annotation of init_currently_empty_zone is wrong.

WARNING: vmlinux.o(.text+0x188aa2): Section mismatch in reference from the function move_pfn_range_to_zone() to the function .meminit.text:memmap_init_zone()
The function move_pfn_range_to_zone() references
the function __meminit memmap_init_zone().
This is often because move_pfn_range_to_zone lacks a __meminit
annotation or the annotation of memmap_init_zone is wrong.

WARNING: vmlinux.o(.text+0x188b3b): Section mismatch in reference from the function move_pfn_range_to_zone() to the function .meminit.text:init_currently_empty_zone()
The function move_pfn_range_to_zone() references
the function __meminit init_currently_empty_zone().
This is often because move_pfn_range_to_zone lacks a __meminit
annotation or the annotation of init_currently_empty_zone is wrong.

Both memmap_init_zone and init_currently_empty_zone are marked __meminit
but move_pfn_range_to_zone is used outside of __meminit sections (e.g.
devm_memremap_pages) so we have to hide it from the checker by __ref
annotation.

Reported-by: Tobias Regnery <tobias.regnery@gmail.com>
Signed-off-by: Michal Hocko <mhocko@suse.com>
---
 mm/memory_hotplug.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/mm/memory_hotplug.c b/mm/memory_hotplug.c
index 497898943212..2013e54178ad 100644
--- a/mm/memory_hotplug.c
+++ b/mm/memory_hotplug.c
@@ -1074,7 +1074,7 @@ static void __meminit resize_pgdat_range(struct pglist_data *pgdat, unsigned lon
 	pgdat->node_spanned_pages = max(start_pfn + nr_pages, old_end_pfn) - pgdat->node_start_pfn;
 }
 
-void move_pfn_range_to_zone(struct zone *zone,
+void __ref move_pfn_range_to_zone(struct zone *zone,
 		unsigned long start_pfn, unsigned long nr_pages)
 {
 	struct pglist_data *pgdat = zone->zone_pgdat;
-- 
2.11.0

^ permalink raw reply	[flat|nested] 74+ messages in thread

* [PATCH 13/14] mm, memory_hotplug: fix the section mismatch warning
@ 2017-05-15  8:58   ` Michal Hocko
  0 siblings, 0 replies; 74+ messages in thread
From: Michal Hocko @ 2017-05-15  8:58 UTC (permalink / raw)
  To: Andrew Morton
  Cc: linux-mm, Mel Gorman, Vlastimil Babka, Andrea Arcangeli,
	Jerome Glisse, Reza Arbab, Yasuaki Ishimatsu, qiuxishi,
	Kani Toshimitsu, slaoub, Joonsoo Kim, Andi Kleen, David Rientjes,
	Daniel Kiper, Igor Mammedov, Vitaly Kuznetsov, LKML,
	Michal Hocko, Tobias Regnery

From: Michal Hocko <mhocko@suse.com>

Tobias has reported following section mismatches introduced by "mm,
memory_hotplug: do not associate hotadded memory to zones until online".

WARNING: mm/built-in.o(.text+0x5a1c2): Section mismatch in reference from the function move_pfn_range_to_zone() to the function .meminit.text:memmap_init_zone()
The function move_pfn_range_to_zone() references
the function __meminit memmap_init_zone().
This is often because move_pfn_range_to_zone lacks a __meminit
annotation or the annotation of memmap_init_zone is wrong.

WARNING: mm/built-in.o(.text+0x5a25b): Section mismatch in reference from the function move_pfn_range_to_zone() to the function .meminit.text:init_currently_empty_zone()
The function move_pfn_range_to_zone() references
the function __meminit init_currently_empty_zone().
This is often because move_pfn_range_to_zone lacks a __meminit
annotation or the annotation of init_currently_empty_zone is wrong.

WARNING: vmlinux.o(.text+0x188aa2): Section mismatch in reference from the function move_pfn_range_to_zone() to the function .meminit.text:memmap_init_zone()
The function move_pfn_range_to_zone() references
the function __meminit memmap_init_zone().
This is often because move_pfn_range_to_zone lacks a __meminit
annotation or the annotation of memmap_init_zone is wrong.

WARNING: vmlinux.o(.text+0x188b3b): Section mismatch in reference from the function move_pfn_range_to_zone() to the function .meminit.text:init_currently_empty_zone()
The function move_pfn_range_to_zone() references
the function __meminit init_currently_empty_zone().
This is often because move_pfn_range_to_zone lacks a __meminit
annotation or the annotation of init_currently_empty_zone is wrong.

Both memmap_init_zone and init_currently_empty_zone are marked __meminit
but move_pfn_range_to_zone is used outside of __meminit sections (e.g.
devm_memremap_pages) so we have to hide it from the checker by __ref
annotation.

Reported-by: Tobias Regnery <tobias.regnery@gmail.com>
Signed-off-by: Michal Hocko <mhocko@suse.com>
---
 mm/memory_hotplug.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/mm/memory_hotplug.c b/mm/memory_hotplug.c
index 497898943212..2013e54178ad 100644
--- a/mm/memory_hotplug.c
+++ b/mm/memory_hotplug.c
@@ -1074,7 +1074,7 @@ static void __meminit resize_pgdat_range(struct pglist_data *pgdat, unsigned lon
 	pgdat->node_spanned_pages = max(start_pfn + nr_pages, old_end_pfn) - pgdat->node_start_pfn;
 }
 
-void move_pfn_range_to_zone(struct zone *zone,
+void __ref move_pfn_range_to_zone(struct zone *zone,
 		unsigned long start_pfn, unsigned long nr_pages)
 {
 	struct pglist_data *pgdat = zone->zone_pgdat;
-- 
2.11.0

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

^ permalink raw reply	[flat|nested] 74+ messages in thread

* [PATCH 14/14] mm, memory_hotplug: remove unused cruft after memory hotplug rework
  2017-05-15  8:58 ` Michal Hocko
@ 2017-05-15  8:58   ` Michal Hocko
  -1 siblings, 0 replies; 74+ messages in thread
From: Michal Hocko @ 2017-05-15  8:58 UTC (permalink / raw)
  To: Andrew Morton
  Cc: linux-mm, Mel Gorman, Vlastimil Babka, Andrea Arcangeli,
	Jerome Glisse, Reza Arbab, Yasuaki Ishimatsu, qiuxishi,
	Kani Toshimitsu, slaoub, Joonsoo Kim, Andi Kleen, David Rientjes,
	Daniel Kiper, Igor Mammedov, Vitaly Kuznetsov, LKML,
	Michal Hocko

From: Michal Hocko <mhocko@suse.com>

zone_for_memory doesn't have any user anymore as well as the whole zone
shifting infrastructure so drop them all.

This shouldn't introduce any functional changes.

Acked-by: Vlastimil Babka <vbabka@suse.cz>
Signed-off-by: Michal Hocko <mhocko@suse.com>
---
 include/linux/memory_hotplug.h |   2 -
 mm/memory_hotplug.c            | 207 -----------------------------------------
 2 files changed, 209 deletions(-)

diff --git a/include/linux/memory_hotplug.h b/include/linux/memory_hotplug.h
index e1193c44aed1..aec88657ec49 100644
--- a/include/linux/memory_hotplug.h
+++ b/include/linux/memory_hotplug.h
@@ -295,8 +295,6 @@ extern int walk_memory_range(unsigned long start_pfn, unsigned long end_pfn,
 		void *arg, int (*func)(struct memory_block *, void *));
 extern int add_memory(int nid, u64 start, u64 size);
 extern int add_memory_resource(int nid, struct resource *resource, bool online);
-extern int zone_for_memory(int nid, u64 start, u64 size, int zone_default,
-		bool for_device);
 extern int arch_add_memory(int nid, u64 start, u64 size, bool want_memblock);
 extern void move_pfn_range_to_zone(struct zone *zone, unsigned long start_pfn,
 		unsigned long nr_pages);
diff --git a/mm/memory_hotplug.c b/mm/memory_hotplug.c
index 2013e54178ad..074fee084a47 100644
--- a/mm/memory_hotplug.c
+++ b/mm/memory_hotplug.c
@@ -300,180 +300,6 @@ void __init register_page_bootmem_info_node(struct pglist_data *pgdat)
 }
 #endif /* CONFIG_HAVE_BOOTMEM_INFO_NODE */
 
-static void __meminit grow_zone_span(struct zone *zone, unsigned long start_pfn,
-				     unsigned long end_pfn)
-{
-	unsigned long old_zone_end_pfn;
-
-	zone_span_writelock(zone);
-
-	old_zone_end_pfn = zone_end_pfn(zone);
-	if (zone_is_empty(zone) || start_pfn < zone->zone_start_pfn)
-		zone->zone_start_pfn = start_pfn;
-
-	zone->spanned_pages = max(old_zone_end_pfn, end_pfn) -
-				zone->zone_start_pfn;
-
-	zone_span_writeunlock(zone);
-}
-
-static void resize_zone(struct zone *zone, unsigned long start_pfn,
-		unsigned long end_pfn)
-{
-	zone_span_writelock(zone);
-
-	if (end_pfn - start_pfn) {
-		zone->zone_start_pfn = start_pfn;
-		zone->spanned_pages = end_pfn - start_pfn;
-	} else {
-		/*
-		 * make it consist as free_area_init_core(),
-		 * if spanned_pages = 0, then keep start_pfn = 0
-		 */
-		zone->zone_start_pfn = 0;
-		zone->spanned_pages = 0;
-	}
-
-	zone_span_writeunlock(zone);
-}
-
-static void fix_zone_id(struct zone *zone, unsigned long start_pfn,
-		unsigned long end_pfn)
-{
-	enum zone_type zid = zone_idx(zone);
-	int nid = zone->zone_pgdat->node_id;
-	unsigned long pfn;
-
-	for (pfn = start_pfn; pfn < end_pfn; pfn++)
-		set_page_links(pfn_to_page(pfn), zid, nid, pfn);
-}
-
-static void __ref ensure_zone_is_initialized(struct zone *zone,
-			unsigned long start_pfn, unsigned long num_pages)
-{
-	if (!zone_is_initialized(zone))
-		init_currently_empty_zone(zone, start_pfn, num_pages);
-}
-
-static int __meminit move_pfn_range_left(struct zone *z1, struct zone *z2,
-		unsigned long start_pfn, unsigned long end_pfn)
-{
-	unsigned long flags;
-	unsigned long z1_start_pfn;
-
-	ensure_zone_is_initialized(z1, start_pfn, end_pfn - start_pfn);
-
-	pgdat_resize_lock(z1->zone_pgdat, &flags);
-
-	/* can't move pfns which are higher than @z2 */
-	if (end_pfn > zone_end_pfn(z2))
-		goto out_fail;
-	/* the move out part must be at the left most of @z2 */
-	if (start_pfn > z2->zone_start_pfn)
-		goto out_fail;
-	/* must included/overlap */
-	if (end_pfn <= z2->zone_start_pfn)
-		goto out_fail;
-
-	/* use start_pfn for z1's start_pfn if z1 is empty */
-	if (!zone_is_empty(z1))
-		z1_start_pfn = z1->zone_start_pfn;
-	else
-		z1_start_pfn = start_pfn;
-
-	resize_zone(z1, z1_start_pfn, end_pfn);
-	resize_zone(z2, end_pfn, zone_end_pfn(z2));
-
-	pgdat_resize_unlock(z1->zone_pgdat, &flags);
-
-	fix_zone_id(z1, start_pfn, end_pfn);
-
-	return 0;
-out_fail:
-	pgdat_resize_unlock(z1->zone_pgdat, &flags);
-	return -1;
-}
-
-static int __meminit move_pfn_range_right(struct zone *z1, struct zone *z2,
-		unsigned long start_pfn, unsigned long end_pfn)
-{
-	unsigned long flags;
-	unsigned long z2_end_pfn;
-
-	ensure_zone_is_initialized(z2, start_pfn, end_pfn - start_pfn);
-
-	pgdat_resize_lock(z1->zone_pgdat, &flags);
-
-	/* can't move pfns which are lower than @z1 */
-	if (z1->zone_start_pfn > start_pfn)
-		goto out_fail;
-	/* the move out part mast at the right most of @z1 */
-	if (zone_end_pfn(z1) >  end_pfn)
-		goto out_fail;
-	/* must included/overlap */
-	if (start_pfn >= zone_end_pfn(z1))
-		goto out_fail;
-
-	/* use end_pfn for z2's end_pfn if z2 is empty */
-	if (!zone_is_empty(z2))
-		z2_end_pfn = zone_end_pfn(z2);
-	else
-		z2_end_pfn = end_pfn;
-
-	resize_zone(z1, z1->zone_start_pfn, start_pfn);
-	resize_zone(z2, start_pfn, z2_end_pfn);
-
-	pgdat_resize_unlock(z1->zone_pgdat, &flags);
-
-	fix_zone_id(z2, start_pfn, end_pfn);
-
-	return 0;
-out_fail:
-	pgdat_resize_unlock(z1->zone_pgdat, &flags);
-	return -1;
-}
-
-static void __meminit grow_pgdat_span(struct pglist_data *pgdat, unsigned long start_pfn,
-				      unsigned long end_pfn)
-{
-	unsigned long old_pgdat_end_pfn = pgdat_end_pfn(pgdat);
-
-	if (!pgdat->node_spanned_pages || start_pfn < pgdat->node_start_pfn)
-		pgdat->node_start_pfn = start_pfn;
-
-	pgdat->node_spanned_pages = max(old_pgdat_end_pfn, end_pfn) -
-					pgdat->node_start_pfn;
-}
-
-static int __meminit __add_zone(struct zone *zone, unsigned long phys_start_pfn)
-{
-	struct pglist_data *pgdat = zone->zone_pgdat;
-	int nr_pages = PAGES_PER_SECTION;
-	int nid = pgdat->node_id;
-	int zone_type;
-	unsigned long flags, pfn;
-
-	zone_type = zone - pgdat->node_zones;
-	ensure_zone_is_initialized(zone, phys_start_pfn, nr_pages);
-
-	pgdat_resize_lock(zone->zone_pgdat, &flags);
-	grow_zone_span(zone, phys_start_pfn, phys_start_pfn + nr_pages);
-	grow_pgdat_span(zone->zone_pgdat, phys_start_pfn,
-			phys_start_pfn + nr_pages);
-	pgdat_resize_unlock(zone->zone_pgdat, &flags);
-	memmap_init_zone(nr_pages, nid, zone_type,
-			 phys_start_pfn, MEMMAP_HOTPLUG);
-
-	/* online_page_range is called later and expects pages reserved */
-	for (pfn = phys_start_pfn; pfn < phys_start_pfn + nr_pages; pfn++) {
-		if (!pfn_valid(pfn))
-			continue;
-
-		SetPageReserved(pfn_to_page(pfn));
-	}
-	return 0;
-}
-
 static int __meminit __add_section(int nid, unsigned long phys_start_pfn,
 		bool want_memblock)
 {
@@ -1356,39 +1182,6 @@ static int check_hotplug_memory_range(u64 start, u64 size)
 	return 0;
 }
 
-/*
- * If movable zone has already been setup, newly added memory should be check.
- * If its address is higher than movable zone, it should be added as movable.
- * Without this check, movable zone may overlap with other zone.
- */
-static int should_add_memory_movable(int nid, u64 start, u64 size)
-{
-	unsigned long start_pfn = start >> PAGE_SHIFT;
-	pg_data_t *pgdat = NODE_DATA(nid);
-	struct zone *movable_zone = pgdat->node_zones + ZONE_MOVABLE;
-
-	if (zone_is_empty(movable_zone))
-		return 0;
-
-	if (movable_zone->zone_start_pfn <= start_pfn)
-		return 1;
-
-	return 0;
-}
-
-int zone_for_memory(int nid, u64 start, u64 size, int zone_default,
-		bool for_device)
-{
-#ifdef CONFIG_ZONE_DEVICE
-	if (for_device)
-		return ZONE_DEVICE;
-#endif
-	if (should_add_memory_movable(nid, start, size))
-		return ZONE_MOVABLE;
-
-	return zone_default;
-}
-
 static int online_memory_block(struct memory_block *mem, void *arg)
 {
 	return device_online(&mem->dev);
-- 
2.11.0

^ permalink raw reply	[flat|nested] 74+ messages in thread

* [PATCH 14/14] mm, memory_hotplug: remove unused cruft after memory hotplug rework
@ 2017-05-15  8:58   ` Michal Hocko
  0 siblings, 0 replies; 74+ messages in thread
From: Michal Hocko @ 2017-05-15  8:58 UTC (permalink / raw)
  To: Andrew Morton
  Cc: linux-mm, Mel Gorman, Vlastimil Babka, Andrea Arcangeli,
	Jerome Glisse, Reza Arbab, Yasuaki Ishimatsu, qiuxishi,
	Kani Toshimitsu, slaoub, Joonsoo Kim, Andi Kleen, David Rientjes,
	Daniel Kiper, Igor Mammedov, Vitaly Kuznetsov, LKML,
	Michal Hocko

From: Michal Hocko <mhocko@suse.com>

zone_for_memory doesn't have any user anymore as well as the whole zone
shifting infrastructure so drop them all.

This shouldn't introduce any functional changes.

Acked-by: Vlastimil Babka <vbabka@suse.cz>
Signed-off-by: Michal Hocko <mhocko@suse.com>
---
 include/linux/memory_hotplug.h |   2 -
 mm/memory_hotplug.c            | 207 -----------------------------------------
 2 files changed, 209 deletions(-)

diff --git a/include/linux/memory_hotplug.h b/include/linux/memory_hotplug.h
index e1193c44aed1..aec88657ec49 100644
--- a/include/linux/memory_hotplug.h
+++ b/include/linux/memory_hotplug.h
@@ -295,8 +295,6 @@ extern int walk_memory_range(unsigned long start_pfn, unsigned long end_pfn,
 		void *arg, int (*func)(struct memory_block *, void *));
 extern int add_memory(int nid, u64 start, u64 size);
 extern int add_memory_resource(int nid, struct resource *resource, bool online);
-extern int zone_for_memory(int nid, u64 start, u64 size, int zone_default,
-		bool for_device);
 extern int arch_add_memory(int nid, u64 start, u64 size, bool want_memblock);
 extern void move_pfn_range_to_zone(struct zone *zone, unsigned long start_pfn,
 		unsigned long nr_pages);
diff --git a/mm/memory_hotplug.c b/mm/memory_hotplug.c
index 2013e54178ad..074fee084a47 100644
--- a/mm/memory_hotplug.c
+++ b/mm/memory_hotplug.c
@@ -300,180 +300,6 @@ void __init register_page_bootmem_info_node(struct pglist_data *pgdat)
 }
 #endif /* CONFIG_HAVE_BOOTMEM_INFO_NODE */
 
-static void __meminit grow_zone_span(struct zone *zone, unsigned long start_pfn,
-				     unsigned long end_pfn)
-{
-	unsigned long old_zone_end_pfn;
-
-	zone_span_writelock(zone);
-
-	old_zone_end_pfn = zone_end_pfn(zone);
-	if (zone_is_empty(zone) || start_pfn < zone->zone_start_pfn)
-		zone->zone_start_pfn = start_pfn;
-
-	zone->spanned_pages = max(old_zone_end_pfn, end_pfn) -
-				zone->zone_start_pfn;
-
-	zone_span_writeunlock(zone);
-}
-
-static void resize_zone(struct zone *zone, unsigned long start_pfn,
-		unsigned long end_pfn)
-{
-	zone_span_writelock(zone);
-
-	if (end_pfn - start_pfn) {
-		zone->zone_start_pfn = start_pfn;
-		zone->spanned_pages = end_pfn - start_pfn;
-	} else {
-		/*
-		 * make it consist as free_area_init_core(),
-		 * if spanned_pages = 0, then keep start_pfn = 0
-		 */
-		zone->zone_start_pfn = 0;
-		zone->spanned_pages = 0;
-	}
-
-	zone_span_writeunlock(zone);
-}
-
-static void fix_zone_id(struct zone *zone, unsigned long start_pfn,
-		unsigned long end_pfn)
-{
-	enum zone_type zid = zone_idx(zone);
-	int nid = zone->zone_pgdat->node_id;
-	unsigned long pfn;
-
-	for (pfn = start_pfn; pfn < end_pfn; pfn++)
-		set_page_links(pfn_to_page(pfn), zid, nid, pfn);
-}
-
-static void __ref ensure_zone_is_initialized(struct zone *zone,
-			unsigned long start_pfn, unsigned long num_pages)
-{
-	if (!zone_is_initialized(zone))
-		init_currently_empty_zone(zone, start_pfn, num_pages);
-}
-
-static int __meminit move_pfn_range_left(struct zone *z1, struct zone *z2,
-		unsigned long start_pfn, unsigned long end_pfn)
-{
-	unsigned long flags;
-	unsigned long z1_start_pfn;
-
-	ensure_zone_is_initialized(z1, start_pfn, end_pfn - start_pfn);
-
-	pgdat_resize_lock(z1->zone_pgdat, &flags);
-
-	/* can't move pfns which are higher than @z2 */
-	if (end_pfn > zone_end_pfn(z2))
-		goto out_fail;
-	/* the move out part must be at the left most of @z2 */
-	if (start_pfn > z2->zone_start_pfn)
-		goto out_fail;
-	/* must included/overlap */
-	if (end_pfn <= z2->zone_start_pfn)
-		goto out_fail;
-
-	/* use start_pfn for z1's start_pfn if z1 is empty */
-	if (!zone_is_empty(z1))
-		z1_start_pfn = z1->zone_start_pfn;
-	else
-		z1_start_pfn = start_pfn;
-
-	resize_zone(z1, z1_start_pfn, end_pfn);
-	resize_zone(z2, end_pfn, zone_end_pfn(z2));
-
-	pgdat_resize_unlock(z1->zone_pgdat, &flags);
-
-	fix_zone_id(z1, start_pfn, end_pfn);
-
-	return 0;
-out_fail:
-	pgdat_resize_unlock(z1->zone_pgdat, &flags);
-	return -1;
-}
-
-static int __meminit move_pfn_range_right(struct zone *z1, struct zone *z2,
-		unsigned long start_pfn, unsigned long end_pfn)
-{
-	unsigned long flags;
-	unsigned long z2_end_pfn;
-
-	ensure_zone_is_initialized(z2, start_pfn, end_pfn - start_pfn);
-
-	pgdat_resize_lock(z1->zone_pgdat, &flags);
-
-	/* can't move pfns which are lower than @z1 */
-	if (z1->zone_start_pfn > start_pfn)
-		goto out_fail;
-	/* the move out part mast at the right most of @z1 */
-	if (zone_end_pfn(z1) >  end_pfn)
-		goto out_fail;
-	/* must included/overlap */
-	if (start_pfn >= zone_end_pfn(z1))
-		goto out_fail;
-
-	/* use end_pfn for z2's end_pfn if z2 is empty */
-	if (!zone_is_empty(z2))
-		z2_end_pfn = zone_end_pfn(z2);
-	else
-		z2_end_pfn = end_pfn;
-
-	resize_zone(z1, z1->zone_start_pfn, start_pfn);
-	resize_zone(z2, start_pfn, z2_end_pfn);
-
-	pgdat_resize_unlock(z1->zone_pgdat, &flags);
-
-	fix_zone_id(z2, start_pfn, end_pfn);
-
-	return 0;
-out_fail:
-	pgdat_resize_unlock(z1->zone_pgdat, &flags);
-	return -1;
-}
-
-static void __meminit grow_pgdat_span(struct pglist_data *pgdat, unsigned long start_pfn,
-				      unsigned long end_pfn)
-{
-	unsigned long old_pgdat_end_pfn = pgdat_end_pfn(pgdat);
-
-	if (!pgdat->node_spanned_pages || start_pfn < pgdat->node_start_pfn)
-		pgdat->node_start_pfn = start_pfn;
-
-	pgdat->node_spanned_pages = max(old_pgdat_end_pfn, end_pfn) -
-					pgdat->node_start_pfn;
-}
-
-static int __meminit __add_zone(struct zone *zone, unsigned long phys_start_pfn)
-{
-	struct pglist_data *pgdat = zone->zone_pgdat;
-	int nr_pages = PAGES_PER_SECTION;
-	int nid = pgdat->node_id;
-	int zone_type;
-	unsigned long flags, pfn;
-
-	zone_type = zone - pgdat->node_zones;
-	ensure_zone_is_initialized(zone, phys_start_pfn, nr_pages);
-
-	pgdat_resize_lock(zone->zone_pgdat, &flags);
-	grow_zone_span(zone, phys_start_pfn, phys_start_pfn + nr_pages);
-	grow_pgdat_span(zone->zone_pgdat, phys_start_pfn,
-			phys_start_pfn + nr_pages);
-	pgdat_resize_unlock(zone->zone_pgdat, &flags);
-	memmap_init_zone(nr_pages, nid, zone_type,
-			 phys_start_pfn, MEMMAP_HOTPLUG);
-
-	/* online_page_range is called later and expects pages reserved */
-	for (pfn = phys_start_pfn; pfn < phys_start_pfn + nr_pages; pfn++) {
-		if (!pfn_valid(pfn))
-			continue;
-
-		SetPageReserved(pfn_to_page(pfn));
-	}
-	return 0;
-}
-
 static int __meminit __add_section(int nid, unsigned long phys_start_pfn,
 		bool want_memblock)
 {
@@ -1356,39 +1182,6 @@ static int check_hotplug_memory_range(u64 start, u64 size)
 	return 0;
 }
 
-/*
- * If movable zone has already been setup, newly added memory should be check.
- * If its address is higher than movable zone, it should be added as movable.
- * Without this check, movable zone may overlap with other zone.
- */
-static int should_add_memory_movable(int nid, u64 start, u64 size)
-{
-	unsigned long start_pfn = start >> PAGE_SHIFT;
-	pg_data_t *pgdat = NODE_DATA(nid);
-	struct zone *movable_zone = pgdat->node_zones + ZONE_MOVABLE;
-
-	if (zone_is_empty(movable_zone))
-		return 0;
-
-	if (movable_zone->zone_start_pfn <= start_pfn)
-		return 1;
-
-	return 0;
-}
-
-int zone_for_memory(int nid, u64 start, u64 size, int zone_default,
-		bool for_device)
-{
-#ifdef CONFIG_ZONE_DEVICE
-	if (for_device)
-		return ZONE_DEVICE;
-#endif
-	if (should_add_memory_movable(nid, start, size))
-		return ZONE_MOVABLE;
-
-	return zone_default;
-}
-
 static int online_memory_block(struct memory_block *mem, void *arg)
 {
 	return device_online(&mem->dev);
-- 
2.11.0

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

^ permalink raw reply	[flat|nested] 74+ messages in thread

* Re: [PATCH 07/14] mm: consider zone which is not fully populated to have holes
  2017-05-15  8:58   ` Michal Hocko
@ 2017-05-18 16:14     ` Vlastimil Babka
  -1 siblings, 0 replies; 74+ messages in thread
From: Vlastimil Babka @ 2017-05-18 16:14 UTC (permalink / raw)
  To: Michal Hocko, Andrew Morton
  Cc: linux-mm, Mel Gorman, Andrea Arcangeli, Jerome Glisse,
	Reza Arbab, Yasuaki Ishimatsu, qiuxishi, Kani Toshimitsu, slaoub,
	Joonsoo Kim, Andi Kleen, David Rientjes, Daniel Kiper,
	Igor Mammedov, Vitaly Kuznetsov, LKML, Michal Hocko

On 05/15/2017 10:58 AM, Michal Hocko wrote:
> From: Michal Hocko <mhocko@suse.com>
> 
> __pageblock_pfn_to_page has two users currently, set_zone_contiguous
> which checks whether the given zone contains holes and
> pageblock_pfn_to_page which then carefully returns a first valid
> page from the given pfn range for the given zone. This doesn't handle
> zones which are not fully populated though. Memory pageblocks can be
> offlined or might not have been onlined yet. In such a case the zone
> should be considered to have holes otherwise pfn walkers can touch
> and play with offline pages.
> 
> Current callers of pageblock_pfn_to_page in compaction seem to work
> properly right now because they only isolate PageBuddy
> (isolate_freepages_block) or PageLRU resp. __PageMovable
> (isolate_migratepages_block) which will be always false for these pages.
> It would be safer to skip these pages altogether, though.
> 
> In order to do this patch adds a new memory section state
> (SECTION_IS_ONLINE) which is set in memory_present (during boot
> time) or in online_pages_range during the memory hotplug. Similarly
> offline_mem_sections clears the bit and it is called when the memory
> range is offlined.
> 
> pfn_to_online_page helper is then added which check the mem section and
> only returns a page if it is onlined already.
> 
> Use the new helper in __pageblock_pfn_to_page and skip the whole page
> block in such a case.
> 
> Changes since v3
> - clarify pfn_valid semantic - requested by Joonsoo
> 
> Signed-off-by: Michal Hocko <mhocko@suse.com>
> ---
>  include/linux/memory_hotplug.h | 21 ++++++++++++++++++++
>  include/linux/mmzone.h         | 35 ++++++++++++++++++++++++++------
>  mm/memory_hotplug.c            |  3 +++
>  mm/page_alloc.c                |  5 ++++-
>  mm/sparse.c                    | 45 +++++++++++++++++++++++++++++++++++++++++-
>  5 files changed, 101 insertions(+), 8 deletions(-)
> 
> diff --git a/include/linux/memory_hotplug.h b/include/linux/memory_hotplug.h
> index 3c8cf86201c3..fc1c873504eb 100644
> --- a/include/linux/memory_hotplug.h
> +++ b/include/linux/memory_hotplug.h
> @@ -14,6 +14,19 @@ struct memory_block;
>  struct resource;
>  
>  #ifdef CONFIG_MEMORY_HOTPLUG
> +/*
> + * Return page for the valid pfn only if the page is online. All pfn
> + * walkers which rely on the fully initialized page->flags and others
> + * should use this rather than pfn_valid && pfn_to_page
> + */
> +#define pfn_to_online_page(pfn)				\
> +({							\
> +	struct page *___page = NULL;			\
> +							\
> +	if (online_section_nr(pfn_to_section_nr(pfn)))	\
> +		___page = pfn_to_page(pfn);		\
> +	___page;					\
> +})

This seems to be already assuming pfn_valid() to be true. There's no
"pfn_to_section_nr(pfn) >= NR_MEM_SECTIONS" check and the comment
suggests as such, but...

>  /*
>   * Types for free bootmem stored in page->lru.next. These have to be in
> @@ -203,6 +216,14 @@ extern void set_zone_contiguous(struct zone *zone);
>  extern void clear_zone_contiguous(struct zone *zone);
>  
>  #else /* ! CONFIG_MEMORY_HOTPLUG */
> +#define pfn_to_online_page(pfn)			\
> +({						\
> +	struct page *___page = NULL;		\
> +	if (pfn_valid(pfn))			\
> +		___page = pfn_to_page(pfn);	\

This includes the pfn_valid() check itself. Why the discrepancy?
Somebody might develop code with !HOTPLUG and forget the check, and then
it starts breaking with HOTPLUG?

> +	___page;				\
> + })
> +
>  /*
>   * Stub functions for when hotplug is off
>   */

...

> diff --git a/mm/memory_hotplug.c b/mm/memory_hotplug.c
> index 05796ee974f7..c3a146028ba6 100644
> --- a/mm/memory_hotplug.c
> +++ b/mm/memory_hotplug.c
> @@ -929,6 +929,9 @@ static int online_pages_range(unsigned long start_pfn, unsigned long nr_pages,
>  	unsigned long i;
>  	unsigned long onlined_pages = *(unsigned long *)arg;
>  	struct page *page;
> +
> +	online_mem_sections(start_pfn, start_pfn + nr_pages);

Shouldn't this be moved *below* the loop that initializes struct pages?
In the offline case you do mark sections offline before "tearing" struct
pages, so that should be symmetric.

> +
>  	if (PageReserved(pfn_to_page(start_pfn)))
>  		for (i = 0; i < nr_pages; i++) {
>  			page = pfn_to_page(start_pfn + i);
> diff --git a/mm/page_alloc.c b/mm/page_alloc.c
> index c1670f090107..7e5151a7dd7b 100644
> --- a/mm/page_alloc.c
> +++ b/mm/page_alloc.c
> @@ -1353,7 +1353,9 @@ struct page *__pageblock_pfn_to_page(unsigned long start_pfn,
>  	if (!pfn_valid(start_pfn) || !pfn_valid(end_pfn))
>  		return NULL;
>  
> -	start_page = pfn_to_page(start_pfn);
> +	start_page = pfn_to_online_page(start_pfn);
> +	if (!start_page)
> +		return NULL;
>  
>  	if (page_zone(start_page) != zone)
>  		return NULL;
> @@ -7671,6 +7673,7 @@ __offline_isolated_pages(unsigned long start_pfn, unsigned long end_pfn)
>  			break;
>  	if (pfn == end_pfn)
>  		return;
> +	offline_mem_sections(pfn, end_pfn);
>  	zone = page_zone(pfn_to_page(pfn));
>  	spin_lock_irqsave(&zone->lock, flags);
>  	pfn = start_pfn;

^ permalink raw reply	[flat|nested] 74+ messages in thread

* Re: [PATCH 07/14] mm: consider zone which is not fully populated to have holes
@ 2017-05-18 16:14     ` Vlastimil Babka
  0 siblings, 0 replies; 74+ messages in thread
From: Vlastimil Babka @ 2017-05-18 16:14 UTC (permalink / raw)
  To: Michal Hocko, Andrew Morton
  Cc: linux-mm, Mel Gorman, Andrea Arcangeli, Jerome Glisse,
	Reza Arbab, Yasuaki Ishimatsu, qiuxishi, Kani Toshimitsu, slaoub,
	Joonsoo Kim, Andi Kleen, David Rientjes, Daniel Kiper,
	Igor Mammedov, Vitaly Kuznetsov, LKML, Michal Hocko

On 05/15/2017 10:58 AM, Michal Hocko wrote:
> From: Michal Hocko <mhocko@suse.com>
> 
> __pageblock_pfn_to_page has two users currently, set_zone_contiguous
> which checks whether the given zone contains holes and
> pageblock_pfn_to_page which then carefully returns a first valid
> page from the given pfn range for the given zone. This doesn't handle
> zones which are not fully populated though. Memory pageblocks can be
> offlined or might not have been onlined yet. In such a case the zone
> should be considered to have holes otherwise pfn walkers can touch
> and play with offline pages.
> 
> Current callers of pageblock_pfn_to_page in compaction seem to work
> properly right now because they only isolate PageBuddy
> (isolate_freepages_block) or PageLRU resp. __PageMovable
> (isolate_migratepages_block) which will be always false for these pages.
> It would be safer to skip these pages altogether, though.
> 
> In order to do this patch adds a new memory section state
> (SECTION_IS_ONLINE) which is set in memory_present (during boot
> time) or in online_pages_range during the memory hotplug. Similarly
> offline_mem_sections clears the bit and it is called when the memory
> range is offlined.
> 
> pfn_to_online_page helper is then added which check the mem section and
> only returns a page if it is onlined already.
> 
> Use the new helper in __pageblock_pfn_to_page and skip the whole page
> block in such a case.
> 
> Changes since v3
> - clarify pfn_valid semantic - requested by Joonsoo
> 
> Signed-off-by: Michal Hocko <mhocko@suse.com>
> ---
>  include/linux/memory_hotplug.h | 21 ++++++++++++++++++++
>  include/linux/mmzone.h         | 35 ++++++++++++++++++++++++++------
>  mm/memory_hotplug.c            |  3 +++
>  mm/page_alloc.c                |  5 ++++-
>  mm/sparse.c                    | 45 +++++++++++++++++++++++++++++++++++++++++-
>  5 files changed, 101 insertions(+), 8 deletions(-)
> 
> diff --git a/include/linux/memory_hotplug.h b/include/linux/memory_hotplug.h
> index 3c8cf86201c3..fc1c873504eb 100644
> --- a/include/linux/memory_hotplug.h
> +++ b/include/linux/memory_hotplug.h
> @@ -14,6 +14,19 @@ struct memory_block;
>  struct resource;
>  
>  #ifdef CONFIG_MEMORY_HOTPLUG
> +/*
> + * Return page for the valid pfn only if the page is online. All pfn
> + * walkers which rely on the fully initialized page->flags and others
> + * should use this rather than pfn_valid && pfn_to_page
> + */
> +#define pfn_to_online_page(pfn)				\
> +({							\
> +	struct page *___page = NULL;			\
> +							\
> +	if (online_section_nr(pfn_to_section_nr(pfn)))	\
> +		___page = pfn_to_page(pfn);		\
> +	___page;					\
> +})

This seems to be already assuming pfn_valid() to be true. There's no
"pfn_to_section_nr(pfn) >= NR_MEM_SECTIONS" check and the comment
suggests as such, but...

>  /*
>   * Types for free bootmem stored in page->lru.next. These have to be in
> @@ -203,6 +216,14 @@ extern void set_zone_contiguous(struct zone *zone);
>  extern void clear_zone_contiguous(struct zone *zone);
>  
>  #else /* ! CONFIG_MEMORY_HOTPLUG */
> +#define pfn_to_online_page(pfn)			\
> +({						\
> +	struct page *___page = NULL;		\
> +	if (pfn_valid(pfn))			\
> +		___page = pfn_to_page(pfn);	\

This includes the pfn_valid() check itself. Why the discrepancy?
Somebody might develop code with !HOTPLUG and forget the check, and then
it starts breaking with HOTPLUG?

> +	___page;				\
> + })
> +
>  /*
>   * Stub functions for when hotplug is off
>   */

...

> diff --git a/mm/memory_hotplug.c b/mm/memory_hotplug.c
> index 05796ee974f7..c3a146028ba6 100644
> --- a/mm/memory_hotplug.c
> +++ b/mm/memory_hotplug.c
> @@ -929,6 +929,9 @@ static int online_pages_range(unsigned long start_pfn, unsigned long nr_pages,
>  	unsigned long i;
>  	unsigned long onlined_pages = *(unsigned long *)arg;
>  	struct page *page;
> +
> +	online_mem_sections(start_pfn, start_pfn + nr_pages);

Shouldn't this be moved *below* the loop that initializes struct pages?
In the offline case you do mark sections offline before "tearing" struct
pages, so that should be symmetric.

> +
>  	if (PageReserved(pfn_to_page(start_pfn)))
>  		for (i = 0; i < nr_pages; i++) {
>  			page = pfn_to_page(start_pfn + i);
> diff --git a/mm/page_alloc.c b/mm/page_alloc.c
> index c1670f090107..7e5151a7dd7b 100644
> --- a/mm/page_alloc.c
> +++ b/mm/page_alloc.c
> @@ -1353,7 +1353,9 @@ struct page *__pageblock_pfn_to_page(unsigned long start_pfn,
>  	if (!pfn_valid(start_pfn) || !pfn_valid(end_pfn))
>  		return NULL;
>  
> -	start_page = pfn_to_page(start_pfn);
> +	start_page = pfn_to_online_page(start_pfn);
> +	if (!start_page)
> +		return NULL;
>  
>  	if (page_zone(start_page) != zone)
>  		return NULL;
> @@ -7671,6 +7673,7 @@ __offline_isolated_pages(unsigned long start_pfn, unsigned long end_pfn)
>  			break;
>  	if (pfn == end_pfn)
>  		return;
> +	offline_mem_sections(pfn, end_pfn);
>  	zone = page_zone(pfn_to_page(pfn));
>  	spin_lock_irqsave(&zone->lock, flags);
>  	pfn = start_pfn;

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

^ permalink raw reply	[flat|nested] 74+ messages in thread

* Re: [PATCH 07/14] mm: consider zone which is not fully populated to have holes
  2017-05-18 16:14     ` Vlastimil Babka
@ 2017-05-18 16:42       ` Michal Hocko
  -1 siblings, 0 replies; 74+ messages in thread
From: Michal Hocko @ 2017-05-18 16:42 UTC (permalink / raw)
  To: Vlastimil Babka
  Cc: Andrew Morton, linux-mm, Mel Gorman, Andrea Arcangeli,
	Jerome Glisse, Reza Arbab, Yasuaki Ishimatsu, qiuxishi,
	Kani Toshimitsu, slaoub, Joonsoo Kim, Andi Kleen, David Rientjes,
	Daniel Kiper, Igor Mammedov, Vitaly Kuznetsov, LKML

On Thu 18-05-17 18:14:39, Vlastimil Babka wrote:
> On 05/15/2017 10:58 AM, Michal Hocko wrote:
[...]
> >  #ifdef CONFIG_MEMORY_HOTPLUG
> > +/*
> > + * Return page for the valid pfn only if the page is online. All pfn
> > + * walkers which rely on the fully initialized page->flags and others
> > + * should use this rather than pfn_valid && pfn_to_page
> > + */
> > +#define pfn_to_online_page(pfn)				\
> > +({							\
> > +	struct page *___page = NULL;			\
> > +							\
> > +	if (online_section_nr(pfn_to_section_nr(pfn)))	\
> > +		___page = pfn_to_page(pfn);		\
> > +	___page;					\
> > +})
> 
> This seems to be already assuming pfn_valid() to be true. There's no
> "pfn_to_section_nr(pfn) >= NR_MEM_SECTIONS" check and the comment
> suggests as such, but...

Yes, we should check the validity of the section number. We do not have
to check whether the section is valid because online sections are a
subset of those that are valid.

> > diff --git a/mm/memory_hotplug.c b/mm/memory_hotplug.c
> > index 05796ee974f7..c3a146028ba6 100644
> > --- a/mm/memory_hotplug.c
> > +++ b/mm/memory_hotplug.c
> > @@ -929,6 +929,9 @@ static int online_pages_range(unsigned long start_pfn, unsigned long nr_pages,
> >  	unsigned long i;
> >  	unsigned long onlined_pages = *(unsigned long *)arg;
> >  	struct page *page;
> > +
> > +	online_mem_sections(start_pfn, start_pfn + nr_pages);
> 
> Shouldn't this be moved *below* the loop that initializes struct pages?
> In the offline case you do mark sections offline before "tearing" struct
> pages, so that should be symmetric.

You are right! Andrew, could you fold the following intot the patch?
---
>From 0550b61203d6970b47fd79f5e6372dccd143cbec Mon Sep 17 00:00:00 2001
From: Michal Hocko <mhocko@suse.com>
Date: Thu, 18 May 2017 18:38:24 +0200
Subject: [PATCH] fold me "mm: consider zone which is not fully populated to
 have holes"

- check valid section number in pfn_to_online_page - Vlastimil
- mark sections online after all struct pages are initialized in
  online_pages_range - Vlastimil

Signed-off-by: Michal Hocko <mhocko@suse.com>
---
 include/linux/memory_hotplug.h | 3 ++-
 mm/memory_hotplug.c            | 5 +++--
 2 files changed, 5 insertions(+), 3 deletions(-)

diff --git a/include/linux/memory_hotplug.h b/include/linux/memory_hotplug.h
index aec88657ec49..9e0249d0f5e4 100644
--- a/include/linux/memory_hotplug.h
+++ b/include/linux/memory_hotplug.h
@@ -22,8 +22,9 @@ struct resource;
 #define pfn_to_online_page(pfn)				\
 ({							\
 	struct page *___page = NULL;			\
+	unsigned long ___nr = pfn_to_section_nr(pfn);	\
 							\
-	if (online_section_nr(pfn_to_section_nr(pfn)))	\
+	if (___nr < NR_MEM_SECTIONS && online_section_nr(___nr))\
 		___page = pfn_to_page(pfn);		\
 	___page;					\
 })
diff --git a/mm/memory_hotplug.c b/mm/memory_hotplug.c
index de30822642c6..a31b547c11f0 100644
--- a/mm/memory_hotplug.c
+++ b/mm/memory_hotplug.c
@@ -746,14 +746,15 @@ static int online_pages_range(unsigned long start_pfn, unsigned long nr_pages,
 	unsigned long onlined_pages = *(unsigned long *)arg;
 	struct page *page;
 
-	online_mem_sections(start_pfn, start_pfn + nr_pages);
-
 	if (PageReserved(pfn_to_page(start_pfn)))
 		for (i = 0; i < nr_pages; i++) {
 			page = pfn_to_page(start_pfn + i);
 			(*online_page_callback)(page);
 			onlined_pages++;
 		}
+
+	online_mem_sections(start_pfn, start_pfn + nr_pages);
+
 	*(unsigned long *)arg = onlined_pages;
 	return 0;
 }
-- 
2.11.0


-- 
Michal Hocko
SUSE Labs

^ permalink raw reply	[flat|nested] 74+ messages in thread

* Re: [PATCH 07/14] mm: consider zone which is not fully populated to have holes
@ 2017-05-18 16:42       ` Michal Hocko
  0 siblings, 0 replies; 74+ messages in thread
From: Michal Hocko @ 2017-05-18 16:42 UTC (permalink / raw)
  To: Vlastimil Babka
  Cc: Andrew Morton, linux-mm, Mel Gorman, Andrea Arcangeli,
	Jerome Glisse, Reza Arbab, Yasuaki Ishimatsu, qiuxishi,
	Kani Toshimitsu, slaoub, Joonsoo Kim, Andi Kleen, David Rientjes,
	Daniel Kiper, Igor Mammedov, Vitaly Kuznetsov, LKML

On Thu 18-05-17 18:14:39, Vlastimil Babka wrote:
> On 05/15/2017 10:58 AM, Michal Hocko wrote:
[...]
> >  #ifdef CONFIG_MEMORY_HOTPLUG
> > +/*
> > + * Return page for the valid pfn only if the page is online. All pfn
> > + * walkers which rely on the fully initialized page->flags and others
> > + * should use this rather than pfn_valid && pfn_to_page
> > + */
> > +#define pfn_to_online_page(pfn)				\
> > +({							\
> > +	struct page *___page = NULL;			\
> > +							\
> > +	if (online_section_nr(pfn_to_section_nr(pfn)))	\
> > +		___page = pfn_to_page(pfn);		\
> > +	___page;					\
> > +})
> 
> This seems to be already assuming pfn_valid() to be true. There's no
> "pfn_to_section_nr(pfn) >= NR_MEM_SECTIONS" check and the comment
> suggests as such, but...

Yes, we should check the validity of the section number. We do not have
to check whether the section is valid because online sections are a
subset of those that are valid.

> > diff --git a/mm/memory_hotplug.c b/mm/memory_hotplug.c
> > index 05796ee974f7..c3a146028ba6 100644
> > --- a/mm/memory_hotplug.c
> > +++ b/mm/memory_hotplug.c
> > @@ -929,6 +929,9 @@ static int online_pages_range(unsigned long start_pfn, unsigned long nr_pages,
> >  	unsigned long i;
> >  	unsigned long onlined_pages = *(unsigned long *)arg;
> >  	struct page *page;
> > +
> > +	online_mem_sections(start_pfn, start_pfn + nr_pages);
> 
> Shouldn't this be moved *below* the loop that initializes struct pages?
> In the offline case you do mark sections offline before "tearing" struct
> pages, so that should be symmetric.

You are right! Andrew, could you fold the following intot the patch?
---

^ permalink raw reply	[flat|nested] 74+ messages in thread

* Re: [PATCH 07/14] mm: consider zone which is not fully populated to have holes
  2017-05-18 16:42       ` Michal Hocko
@ 2017-05-19  7:21         ` Vlastimil Babka
  -1 siblings, 0 replies; 74+ messages in thread
From: Vlastimil Babka @ 2017-05-19  7:21 UTC (permalink / raw)
  To: Michal Hocko
  Cc: Andrew Morton, linux-mm, Mel Gorman, Andrea Arcangeli,
	Jerome Glisse, Reza Arbab, Yasuaki Ishimatsu, qiuxishi,
	Kani Toshimitsu, slaoub, Joonsoo Kim, Andi Kleen, David Rientjes,
	Daniel Kiper, Igor Mammedov, Vitaly Kuznetsov, LKML

On 05/18/2017 06:42 PM, Michal Hocko wrote:
> On Thu 18-05-17 18:14:39, Vlastimil Babka wrote:
>> On 05/15/2017 10:58 AM, Michal Hocko wrote:
> [...]
>>>  #ifdef CONFIG_MEMORY_HOTPLUG
>>> +/*
>>> + * Return page for the valid pfn only if the page is online. All pfn
>>> + * walkers which rely on the fully initialized page->flags and others
>>> + * should use this rather than pfn_valid && pfn_to_page
>>> + */
>>> +#define pfn_to_online_page(pfn)				\
>>> +({							\
>>> +	struct page *___page = NULL;			\
>>> +							\
>>> +	if (online_section_nr(pfn_to_section_nr(pfn)))	\
>>> +		___page = pfn_to_page(pfn);		\
>>> +	___page;					\
>>> +})
>>
>> This seems to be already assuming pfn_valid() to be true. There's no
>> "pfn_to_section_nr(pfn) >= NR_MEM_SECTIONS" check and the comment
>> suggests as such, but...
> 
> Yes, we should check the validity of the section number. We do not have
> to check whether the section is valid because online sections are a
> subset of those that are valid.
> 
>>> diff --git a/mm/memory_hotplug.c b/mm/memory_hotplug.c
>>> index 05796ee974f7..c3a146028ba6 100644
>>> --- a/mm/memory_hotplug.c
>>> +++ b/mm/memory_hotplug.c
>>> @@ -929,6 +929,9 @@ static int online_pages_range(unsigned long start_pfn, unsigned long nr_pages,
>>>  	unsigned long i;
>>>  	unsigned long onlined_pages = *(unsigned long *)arg;
>>>  	struct page *page;
>>> +
>>> +	online_mem_sections(start_pfn, start_pfn + nr_pages);
>>
>> Shouldn't this be moved *below* the loop that initializes struct pages?
>> In the offline case you do mark sections offline before "tearing" struct
>> pages, so that should be symmetric.
> 
> You are right! Andrew, could you fold the following intot the patch?
> ---
> From 0550b61203d6970b47fd79f5e6372dccd143cbec Mon Sep 17 00:00:00 2001
> From: Michal Hocko <mhocko@suse.com>
> Date: Thu, 18 May 2017 18:38:24 +0200
> Subject: [PATCH] fold me "mm: consider zone which is not fully populated to
>  have holes"
> 
> - check valid section number in pfn_to_online_page - Vlastimil
> - mark sections online after all struct pages are initialized in
>   online_pages_range - Vlastimil
> 
> Signed-off-by: Michal Hocko <mhocko@suse.com>

Both the patch and fix:

Acked-by: Vlastimil Babka <vbabka@suse.cz>

^ permalink raw reply	[flat|nested] 74+ messages in thread

* Re: [PATCH 07/14] mm: consider zone which is not fully populated to have holes
@ 2017-05-19  7:21         ` Vlastimil Babka
  0 siblings, 0 replies; 74+ messages in thread
From: Vlastimil Babka @ 2017-05-19  7:21 UTC (permalink / raw)
  To: Michal Hocko
  Cc: Andrew Morton, linux-mm, Mel Gorman, Andrea Arcangeli,
	Jerome Glisse, Reza Arbab, Yasuaki Ishimatsu, qiuxishi,
	Kani Toshimitsu, slaoub, Joonsoo Kim, Andi Kleen, David Rientjes,
	Daniel Kiper, Igor Mammedov, Vitaly Kuznetsov, LKML

On 05/18/2017 06:42 PM, Michal Hocko wrote:
> On Thu 18-05-17 18:14:39, Vlastimil Babka wrote:
>> On 05/15/2017 10:58 AM, Michal Hocko wrote:
> [...]
>>>  #ifdef CONFIG_MEMORY_HOTPLUG
>>> +/*
>>> + * Return page for the valid pfn only if the page is online. All pfn
>>> + * walkers which rely on the fully initialized page->flags and others
>>> + * should use this rather than pfn_valid && pfn_to_page
>>> + */
>>> +#define pfn_to_online_page(pfn)				\
>>> +({							\
>>> +	struct page *___page = NULL;			\
>>> +							\
>>> +	if (online_section_nr(pfn_to_section_nr(pfn)))	\
>>> +		___page = pfn_to_page(pfn);		\
>>> +	___page;					\
>>> +})
>>
>> This seems to be already assuming pfn_valid() to be true. There's no
>> "pfn_to_section_nr(pfn) >= NR_MEM_SECTIONS" check and the comment
>> suggests as such, but...
> 
> Yes, we should check the validity of the section number. We do not have
> to check whether the section is valid because online sections are a
> subset of those that are valid.
> 
>>> diff --git a/mm/memory_hotplug.c b/mm/memory_hotplug.c
>>> index 05796ee974f7..c3a146028ba6 100644
>>> --- a/mm/memory_hotplug.c
>>> +++ b/mm/memory_hotplug.c
>>> @@ -929,6 +929,9 @@ static int online_pages_range(unsigned long start_pfn, unsigned long nr_pages,
>>>  	unsigned long i;
>>>  	unsigned long onlined_pages = *(unsigned long *)arg;
>>>  	struct page *page;
>>> +
>>> +	online_mem_sections(start_pfn, start_pfn + nr_pages);
>>
>> Shouldn't this be moved *below* the loop that initializes struct pages?
>> In the offline case you do mark sections offline before "tearing" struct
>> pages, so that should be symmetric.
> 
> You are right! Andrew, could you fold the following intot the patch?
> ---
> From 0550b61203d6970b47fd79f5e6372dccd143cbec Mon Sep 17 00:00:00 2001
> From: Michal Hocko <mhocko@suse.com>
> Date: Thu, 18 May 2017 18:38:24 +0200
> Subject: [PATCH] fold me "mm: consider zone which is not fully populated to
>  have holes"
> 
> - check valid section number in pfn_to_online_page - Vlastimil
> - mark sections online after all struct pages are initialized in
>   online_pages_range - Vlastimil
> 
> Signed-off-by: Michal Hocko <mhocko@suse.com>

Both the patch and fix:

Acked-by: Vlastimil Babka <vbabka@suse.cz>

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

^ permalink raw reply	[flat|nested] 74+ messages in thread

* Re: [PATCH 08/14] mm, compaction: skip over holes in __reset_isolation_suitable
  2017-05-15  8:58   ` Michal Hocko
@ 2017-05-19  7:23     ` Vlastimil Babka
  -1 siblings, 0 replies; 74+ messages in thread
From: Vlastimil Babka @ 2017-05-19  7:23 UTC (permalink / raw)
  To: Michal Hocko, Andrew Morton
  Cc: linux-mm, Mel Gorman, Andrea Arcangeli, Jerome Glisse,
	Reza Arbab, Yasuaki Ishimatsu, qiuxishi, Kani Toshimitsu, slaoub,
	Joonsoo Kim, Andi Kleen, David Rientjes, Daniel Kiper,
	Igor Mammedov, Vitaly Kuznetsov, LKML, Michal Hocko

On 05/15/2017 10:58 AM, Michal Hocko wrote:
> From: Michal Hocko <mhocko@suse.com>
> 
> __reset_isolation_suitable walks the whole zone pfn range and it tries
> to jump over holes by checking the zone for each page. It might still
> stumble over offline pages, though. Skip those by checking
> pfn_to_online_page()
> 
> Signed-off-by: Michal Hocko <mhocko@suse.com>

Acked-by: Vlastimil Babka <vbabka@suse.cz>

> ---
>  mm/compaction.c | 5 ++---
>  1 file changed, 2 insertions(+), 3 deletions(-)
> 
> diff --git a/mm/compaction.c b/mm/compaction.c
> index 613c59e928cb..fb548e4c7bd4 100644
> --- a/mm/compaction.c
> +++ b/mm/compaction.c
> @@ -236,10 +236,9 @@ static void __reset_isolation_suitable(struct zone *zone)
>  
>  		cond_resched();
>  
> -		if (!pfn_valid(pfn))
> +		page = pfn_to_online_page(pfn);
> +		if (!page)
>  			continue;
> -
> -		page = pfn_to_page(pfn);
>  		if (zone != page_zone(page))
>  			continue;
>  
> 

^ permalink raw reply	[flat|nested] 74+ messages in thread

* Re: [PATCH 08/14] mm, compaction: skip over holes in __reset_isolation_suitable
@ 2017-05-19  7:23     ` Vlastimil Babka
  0 siblings, 0 replies; 74+ messages in thread
From: Vlastimil Babka @ 2017-05-19  7:23 UTC (permalink / raw)
  To: Michal Hocko, Andrew Morton
  Cc: linux-mm, Mel Gorman, Andrea Arcangeli, Jerome Glisse,
	Reza Arbab, Yasuaki Ishimatsu, qiuxishi, Kani Toshimitsu, slaoub,
	Joonsoo Kim, Andi Kleen, David Rientjes, Daniel Kiper,
	Igor Mammedov, Vitaly Kuznetsov, LKML, Michal Hocko

On 05/15/2017 10:58 AM, Michal Hocko wrote:
> From: Michal Hocko <mhocko@suse.com>
> 
> __reset_isolation_suitable walks the whole zone pfn range and it tries
> to jump over holes by checking the zone for each page. It might still
> stumble over offline pages, though. Skip those by checking
> pfn_to_online_page()
> 
> Signed-off-by: Michal Hocko <mhocko@suse.com>

Acked-by: Vlastimil Babka <vbabka@suse.cz>

> ---
>  mm/compaction.c | 5 ++---
>  1 file changed, 2 insertions(+), 3 deletions(-)
> 
> diff --git a/mm/compaction.c b/mm/compaction.c
> index 613c59e928cb..fb548e4c7bd4 100644
> --- a/mm/compaction.c
> +++ b/mm/compaction.c
> @@ -236,10 +236,9 @@ static void __reset_isolation_suitable(struct zone *zone)
>  
>  		cond_resched();
>  
> -		if (!pfn_valid(pfn))
> +		page = pfn_to_online_page(pfn);
> +		if (!page)
>  			continue;
> -
> -		page = pfn_to_page(pfn);
>  		if (zone != page_zone(page))
>  			continue;
>  
> 

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

^ permalink raw reply	[flat|nested] 74+ messages in thread

* Re: [PATCH 09/14] mm: __first_valid_page skip over offline pages
  2017-05-15  8:58   ` Michal Hocko
@ 2017-05-19  8:05     ` Vlastimil Babka
  -1 siblings, 0 replies; 74+ messages in thread
From: Vlastimil Babka @ 2017-05-19  8:05 UTC (permalink / raw)
  To: Michal Hocko, Andrew Morton
  Cc: linux-mm, Mel Gorman, Andrea Arcangeli, Jerome Glisse,
	Reza Arbab, Yasuaki Ishimatsu, qiuxishi, Kani Toshimitsu, slaoub,
	Joonsoo Kim, Andi Kleen, David Rientjes, Daniel Kiper,
	Igor Mammedov, Vitaly Kuznetsov, LKML, Michal Hocko

On 05/15/2017 10:58 AM, Michal Hocko wrote:
> From: Michal Hocko <mhocko@suse.com>
> 
> __first_valid_page skips over invalid pfns in the range but it might
> still stumble over offline pages. At least start_isolate_page_range
> will mark those set_migratetype_isolate. This doesn't represent
> any immediate AFAICS because alloc_contig_range will fail to isolate
> those pages but it relies on not fully initialized page which will
> become a problem later when we stop associating offline pages to zones.
> Use pfn_to_online_page to handle this.
> 
> This is more a preparatory patch than a fix.
> 
> Signed-off-by: Michal Hocko <mhocko@suse.com>

Iterating over single pages when the whole section is offline seems
rather wasteful, but it should be really rare, so whatever.

Acked-by: Vlastimil Babka <vbabka@suse.cz>

> ---
>  mm/page_isolation.c | 26 ++++++++++++++++++--------
>  1 file changed, 18 insertions(+), 8 deletions(-)
> 
> diff --git a/mm/page_isolation.c b/mm/page_isolation.c
> index 5092e4ef00c8..3606104893e0 100644
> --- a/mm/page_isolation.c
> +++ b/mm/page_isolation.c
> @@ -138,12 +138,18 @@ static inline struct page *
>  __first_valid_page(unsigned long pfn, unsigned long nr_pages)
>  {
>  	int i;
> -	for (i = 0; i < nr_pages; i++)
> -		if (pfn_valid_within(pfn + i))
> -			break;
> -	if (unlikely(i == nr_pages))
> -		return NULL;
> -	return pfn_to_page(pfn + i);
> +
> +	for (i = 0; i < nr_pages; i++) {
> +		struct page *page;
> +
> +		if (!pfn_valid_within(pfn + i))
> +			continue;
> +		page = pfn_to_online_page(pfn + i);
> +		if (!page)
> +			continue;
> +		return page;
> +	}
> +	return NULL;
>  }
>  
>  /*
> @@ -184,8 +190,12 @@ int start_isolate_page_range(unsigned long start_pfn, unsigned long end_pfn,
>  undo:
>  	for (pfn = start_pfn;
>  	     pfn < undo_pfn;
> -	     pfn += pageblock_nr_pages)
> -		unset_migratetype_isolate(pfn_to_page(pfn), migratetype);
> +	     pfn += pageblock_nr_pages) {
> +		struct page *page = pfn_to_online_page(pfn);
> +		if (!page)
> +			continue;
> +		unset_migratetype_isolate(page, migratetype);
> +	}
>  
>  	return -EBUSY;
>  }
> 

^ permalink raw reply	[flat|nested] 74+ messages in thread

* Re: [PATCH 09/14] mm: __first_valid_page skip over offline pages
@ 2017-05-19  8:05     ` Vlastimil Babka
  0 siblings, 0 replies; 74+ messages in thread
From: Vlastimil Babka @ 2017-05-19  8:05 UTC (permalink / raw)
  To: Michal Hocko, Andrew Morton
  Cc: linux-mm, Mel Gorman, Andrea Arcangeli, Jerome Glisse,
	Reza Arbab, Yasuaki Ishimatsu, qiuxishi, Kani Toshimitsu, slaoub,
	Joonsoo Kim, Andi Kleen, David Rientjes, Daniel Kiper,
	Igor Mammedov, Vitaly Kuznetsov, LKML, Michal Hocko

On 05/15/2017 10:58 AM, Michal Hocko wrote:
> From: Michal Hocko <mhocko@suse.com>
> 
> __first_valid_page skips over invalid pfns in the range but it might
> still stumble over offline pages. At least start_isolate_page_range
> will mark those set_migratetype_isolate. This doesn't represent
> any immediate AFAICS because alloc_contig_range will fail to isolate
> those pages but it relies on not fully initialized page which will
> become a problem later when we stop associating offline pages to zones.
> Use pfn_to_online_page to handle this.
> 
> This is more a preparatory patch than a fix.
> 
> Signed-off-by: Michal Hocko <mhocko@suse.com>

Iterating over single pages when the whole section is offline seems
rather wasteful, but it should be really rare, so whatever.

Acked-by: Vlastimil Babka <vbabka@suse.cz>

> ---
>  mm/page_isolation.c | 26 ++++++++++++++++++--------
>  1 file changed, 18 insertions(+), 8 deletions(-)
> 
> diff --git a/mm/page_isolation.c b/mm/page_isolation.c
> index 5092e4ef00c8..3606104893e0 100644
> --- a/mm/page_isolation.c
> +++ b/mm/page_isolation.c
> @@ -138,12 +138,18 @@ static inline struct page *
>  __first_valid_page(unsigned long pfn, unsigned long nr_pages)
>  {
>  	int i;
> -	for (i = 0; i < nr_pages; i++)
> -		if (pfn_valid_within(pfn + i))
> -			break;
> -	if (unlikely(i == nr_pages))
> -		return NULL;
> -	return pfn_to_page(pfn + i);
> +
> +	for (i = 0; i < nr_pages; i++) {
> +		struct page *page;
> +
> +		if (!pfn_valid_within(pfn + i))
> +			continue;
> +		page = pfn_to_online_page(pfn + i);
> +		if (!page)
> +			continue;
> +		return page;
> +	}
> +	return NULL;
>  }
>  
>  /*
> @@ -184,8 +190,12 @@ int start_isolate_page_range(unsigned long start_pfn, unsigned long end_pfn,
>  undo:
>  	for (pfn = start_pfn;
>  	     pfn < undo_pfn;
> -	     pfn += pageblock_nr_pages)
> -		unset_migratetype_isolate(pfn_to_page(pfn), migratetype);
> +	     pfn += pageblock_nr_pages) {
> +		struct page *page = pfn_to_online_page(pfn);
> +		if (!page)
> +			continue;
> +		unset_migratetype_isolate(page, migratetype);
> +	}
>  
>  	return -EBUSY;
>  }
> 

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

^ permalink raw reply	[flat|nested] 74+ messages in thread

* Re: [PATCH 10/14] mm, vmstat: skip reporting offline pages in pagetypeinfo
  2017-05-15  8:58   ` Michal Hocko
@ 2017-05-19  8:17     ` Vlastimil Babka
  -1 siblings, 0 replies; 74+ messages in thread
From: Vlastimil Babka @ 2017-05-19  8:17 UTC (permalink / raw)
  To: Michal Hocko, Andrew Morton
  Cc: linux-mm, Mel Gorman, Andrea Arcangeli, Jerome Glisse,
	Reza Arbab, Yasuaki Ishimatsu, qiuxishi, Kani Toshimitsu, slaoub,
	Joonsoo Kim, Andi Kleen, David Rientjes, Daniel Kiper,
	Igor Mammedov, Vitaly Kuznetsov, LKML, Michal Hocko

On 05/15/2017 10:58 AM, Michal Hocko wrote:
> From: Michal Hocko <mhocko@suse.com>
> 
> pagetypeinfo_showblockcount_print skips over invalid pfns but it would
> report pages which are offline because those have a valid pfn. Their
> migrate type is misleading at best. Now that we have pfn_to_online_page()
> we can use it instead of pfn_valid() and fix this.
> 
> Noticed-by: Joonsoo Kim <js1304@gmail.com>
> Signed-off-by: Michal Hocko <mhocko@suse.com>

(with the followup fix)

Acked-by: Vlastimil Babka <vbabka@suse.cz>

> ---
>  mm/vmstat.c | 4 +---
>  1 file changed, 1 insertion(+), 3 deletions(-)
> 
> diff --git a/mm/vmstat.c b/mm/vmstat.c
> index 571d3ec05566..c432e581f9a9 100644
> --- a/mm/vmstat.c
> +++ b/mm/vmstat.c
> @@ -1223,11 +1223,9 @@ static void pagetypeinfo_showblockcount_print(struct seq_file *m,
>  	for (pfn = start_pfn; pfn < end_pfn; pfn += pageblock_nr_pages) {
>  		struct page *page;
>  
> -		if (!pfn_valid(pfn))
> +		if (!pfn_to_online_page(pfn))
>  			continue;
>  
> -		page = pfn_to_page(pfn);
> -
>  		/* Watch for unexpected holes punched in the memmap */
>  		if (!memmap_valid_within(pfn, page, zone))
>  			continue;
> 

^ permalink raw reply	[flat|nested] 74+ messages in thread

* Re: [PATCH 10/14] mm, vmstat: skip reporting offline pages in pagetypeinfo
@ 2017-05-19  8:17     ` Vlastimil Babka
  0 siblings, 0 replies; 74+ messages in thread
From: Vlastimil Babka @ 2017-05-19  8:17 UTC (permalink / raw)
  To: Michal Hocko, Andrew Morton
  Cc: linux-mm, Mel Gorman, Andrea Arcangeli, Jerome Glisse,
	Reza Arbab, Yasuaki Ishimatsu, qiuxishi, Kani Toshimitsu, slaoub,
	Joonsoo Kim, Andi Kleen, David Rientjes, Daniel Kiper,
	Igor Mammedov, Vitaly Kuznetsov, LKML, Michal Hocko

On 05/15/2017 10:58 AM, Michal Hocko wrote:
> From: Michal Hocko <mhocko@suse.com>
> 
> pagetypeinfo_showblockcount_print skips over invalid pfns but it would
> report pages which are offline because those have a valid pfn. Their
> migrate type is misleading at best. Now that we have pfn_to_online_page()
> we can use it instead of pfn_valid() and fix this.
> 
> Noticed-by: Joonsoo Kim <js1304@gmail.com>
> Signed-off-by: Michal Hocko <mhocko@suse.com>

(with the followup fix)

Acked-by: Vlastimil Babka <vbabka@suse.cz>

> ---
>  mm/vmstat.c | 4 +---
>  1 file changed, 1 insertion(+), 3 deletions(-)
> 
> diff --git a/mm/vmstat.c b/mm/vmstat.c
> index 571d3ec05566..c432e581f9a9 100644
> --- a/mm/vmstat.c
> +++ b/mm/vmstat.c
> @@ -1223,11 +1223,9 @@ static void pagetypeinfo_showblockcount_print(struct seq_file *m,
>  	for (pfn = start_pfn; pfn < end_pfn; pfn += pageblock_nr_pages) {
>  		struct page *page;
>  
> -		if (!pfn_valid(pfn))
> +		if (!pfn_to_online_page(pfn))
>  			continue;
>  
> -		page = pfn_to_page(pfn);
> -
>  		/* Watch for unexpected holes punched in the memmap */
>  		if (!memmap_valid_within(pfn, page, zone))
>  			continue;
> 

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

^ permalink raw reply	[flat|nested] 74+ messages in thread

* Re: [PATCH 11/14] mm, memory_hotplug: do not associate hotadded memory to zones until online
  2017-05-15  8:58   ` Michal Hocko
@ 2017-05-19  8:56     ` Vlastimil Babka
  -1 siblings, 0 replies; 74+ messages in thread
From: Vlastimil Babka @ 2017-05-19  8:56 UTC (permalink / raw)
  To: Michal Hocko, Andrew Morton
  Cc: linux-mm, Mel Gorman, Andrea Arcangeli, Jerome Glisse,
	Reza Arbab, Yasuaki Ishimatsu, qiuxishi, Kani Toshimitsu, slaoub,
	Joonsoo Kim, Andi Kleen, David Rientjes, Daniel Kiper,
	Igor Mammedov, Vitaly Kuznetsov, LKML, Michal Hocko,
	Dan Williams, Heiko Carstens, Martin Schwidefsky

On 05/15/2017 10:58 AM, Michal Hocko wrote:
> From: Michal Hocko <mhocko@suse.com>
> 
> The current memory hotplug implementation relies on having all the
> struct pages associate with a zone/node during the physical hotplug phase
> (arch_add_memory->__add_pages->__add_section->__add_zone). In the vast
> majority of cases this means that they are added to ZONE_NORMAL. This
> has been so since 9d99aaa31f59 ("[PATCH] x86_64: Support memory hotadd
> without sparsemem") and it wasn't a big deal back then because movable
> onlining didn't exist yet.
> 
> Much later memory hotplug wanted to (ab)use ZONE_MOVABLE for movable
> onlining 511c2aba8f07 ("mm, memory-hotplug: dynamic configure movable
> memory and portion memory") and then things got more complicated. Rather
> than reconsidering the zone association which was no longer needed
> (because the memory hotplug already depended on SPARSEMEM) a convoluted
> semantic of zone shifting has been developed. Only the currently last
> memblock or the one adjacent to the zone_movable can be onlined movable.
> This essentially means that the online type changes as the new memblocks
> are added.
> 
> Let's simulate memory hot online manually
> $ echo 0x100000000 > /sys/devices/system/memory/probe
> $ grep . /sys/devices/system/memory/memory32/valid_zones
> Normal Movable
> 
> $ echo $((0x100000000+(128<<20))) > /sys/devices/system/memory/probe
> $ grep . /sys/devices/system/memory/memory3?/valid_zones
> /sys/devices/system/memory/memory32/valid_zones:Normal
> /sys/devices/system/memory/memory33/valid_zones:Normal Movable
> 
> $ echo $((0x100000000+2*(128<<20))) > /sys/devices/system/memory/probe
> $ grep . /sys/devices/system/memory/memory3?/valid_zones
> /sys/devices/system/memory/memory32/valid_zones:Normal
> /sys/devices/system/memory/memory33/valid_zones:Normal
> /sys/devices/system/memory/memory34/valid_zones:Normal Movable
> 
> $ echo online_movable > /sys/devices/system/memory/memory34/state
> $ grep . /sys/devices/system/memory/memory3?/valid_zones
> /sys/devices/system/memory/memory32/valid_zones:Normal
> /sys/devices/system/memory/memory33/valid_zones:Normal Movable
> /sys/devices/system/memory/memory34/valid_zones:Movable Normal
> 
> This is an awkward semantic because an udev event is sent as soon as the
> block is onlined and an udev handler might want to online it based on
> some policy (e.g. association with a node) but it will inherently race
> with new blocks showing up.
> 
> This patch changes the physical online phase to not associate pages
> with any zone at all. All the pages are just marked reserved and wait
> for the onlining phase to be associated with the zone as per the online
> request. There are only two requirements
> 	- existing ZONE_NORMAL and ZONE_MOVABLE cannot overlap
> 	- ZONE_NORMAL precedes ZONE_MOVABLE in physical addresses
> the later on is not an inherent requirement and can be changed in the
> future. It preserves the current behavior and made the code slightly
> simpler. This is subject to change in future.
> 
> This means that the same physical online steps as above will lead to the
> following state:
> Normal Movable
> 
> /sys/devices/system/memory/memory32/valid_zones:Normal Movable
> /sys/devices/system/memory/memory33/valid_zones:Normal Movable
> 
> /sys/devices/system/memory/memory32/valid_zones:Normal Movable
> /sys/devices/system/memory/memory33/valid_zones:Normal Movable
> /sys/devices/system/memory/memory34/valid_zones:Normal Movable
> 
> /sys/devices/system/memory/memory32/valid_zones:Normal Movable
> /sys/devices/system/memory/memory33/valid_zones:Normal Movable
> /sys/devices/system/memory/memory34/valid_zones:Movable
> 
> Implementation:
> The current move_pfn_range is reimplemented to check the above
> requirements (allow_online_pfn_range) and then updates the respective
> zone (move_pfn_range_to_zone), the pgdat and links all the pages in the
> pfn range with the zone/node. __add_pages is updated to not require the
> zone and only initializes sections in the range. This allowed to
> simplify the arch_add_memory code (s390 could get rid of quite some
> of code).
> 
> devm_memremap_pages is the only user of arch_add_memory which relies
> on the zone association because it only hooks into the memory hotplug
> only half way. It uses it to associate the new memory with ZONE_DEVICE
> but doesn't allow it to be {on,off}lined via sysfs. This means that this
> particular code path has to call move_pfn_range_to_zone explicitly.
> 
> The original zone shifting code is kept in place and will be removed in
> the follow up patch for an easier review.
> 
> Please note that this patch also changes the original behavior when
> offlining a memory block adjacent to another zone (Normal vs. Movable)
> used to allow to change its movable type. This will be handled later.
> 
> Changes since v1
> - we have to associate the page with the node early (in __add_section),
>   because pfn_to_node depends on struct page containing this
>   information - based on testing by Reza Arbab
> - resize_{zone,pgdat}_range has to check whether they are popoulated -
>   Reza Arbab
> - fix devm_memremap_pages to use pfn rather than physical address -
>   Jérôme Glisse
> - move_pfn_range has to check for intersection with zone_movable rather
>   than to rely on allow_online_pfn_range(MMOP_ONLINE_MOVABLE) for
>   MMOP_ONLINE_KEEP
> 
> Changes since v2
> - fix show_valid_zones nr_pages calculation
> - allow_online_pfn_range has to check managed pages rather than present
> - zone_intersects fix bogus check
> - fix zone_intersects + few nits as per Vlastimil
> 
> Cc: Martin Schwidefsky <schwidefsky@de.ibm.com>
> Cc: linux-arch@vger.kernel.org
> Tested-by: Dan Williams <dan.j.williams@intel.com>
> Acked-by: Heiko Carstens <heiko.carstens@de.ibm.com> # For s390 bits
> Tested-by: Reza Arbab <arbab@linux.vnet.ibm.com>
> Signed-off-by: Michal Hocko <mhocko@suse.com>

Acked-by: Vlastimil Babka <vbabka@suse.cz>

^ permalink raw reply	[flat|nested] 74+ messages in thread

* Re: [PATCH 11/14] mm, memory_hotplug: do not associate hotadded memory to zones until online
@ 2017-05-19  8:56     ` Vlastimil Babka
  0 siblings, 0 replies; 74+ messages in thread
From: Vlastimil Babka @ 2017-05-19  8:56 UTC (permalink / raw)
  To: Michal Hocko, Andrew Morton
  Cc: linux-mm, Mel Gorman, Andrea Arcangeli, Jerome Glisse,
	Reza Arbab, Yasuaki Ishimatsu, qiuxishi, Kani Toshimitsu, slaoub,
	Joonsoo Kim, Andi Kleen, David Rientjes, Daniel Kiper,
	Igor Mammedov, Vitaly Kuznetsov, LKML, Michal Hocko,
	Dan Williams, Heiko Carstens, Martin Schwidefsky

On 05/15/2017 10:58 AM, Michal Hocko wrote:
> From: Michal Hocko <mhocko@suse.com>
> 
> The current memory hotplug implementation relies on having all the
> struct pages associate with a zone/node during the physical hotplug phase
> (arch_add_memory->__add_pages->__add_section->__add_zone). In the vast
> majority of cases this means that they are added to ZONE_NORMAL. This
> has been so since 9d99aaa31f59 ("[PATCH] x86_64: Support memory hotadd
> without sparsemem") and it wasn't a big deal back then because movable
> onlining didn't exist yet.
> 
> Much later memory hotplug wanted to (ab)use ZONE_MOVABLE for movable
> onlining 511c2aba8f07 ("mm, memory-hotplug: dynamic configure movable
> memory and portion memory") and then things got more complicated. Rather
> than reconsidering the zone association which was no longer needed
> (because the memory hotplug already depended on SPARSEMEM) a convoluted
> semantic of zone shifting has been developed. Only the currently last
> memblock or the one adjacent to the zone_movable can be onlined movable.
> This essentially means that the online type changes as the new memblocks
> are added.
> 
> Let's simulate memory hot online manually
> $ echo 0x100000000 > /sys/devices/system/memory/probe
> $ grep . /sys/devices/system/memory/memory32/valid_zones
> Normal Movable
> 
> $ echo $((0x100000000+(128<<20))) > /sys/devices/system/memory/probe
> $ grep . /sys/devices/system/memory/memory3?/valid_zones
> /sys/devices/system/memory/memory32/valid_zones:Normal
> /sys/devices/system/memory/memory33/valid_zones:Normal Movable
> 
> $ echo $((0x100000000+2*(128<<20))) > /sys/devices/system/memory/probe
> $ grep . /sys/devices/system/memory/memory3?/valid_zones
> /sys/devices/system/memory/memory32/valid_zones:Normal
> /sys/devices/system/memory/memory33/valid_zones:Normal
> /sys/devices/system/memory/memory34/valid_zones:Normal Movable
> 
> $ echo online_movable > /sys/devices/system/memory/memory34/state
> $ grep . /sys/devices/system/memory/memory3?/valid_zones
> /sys/devices/system/memory/memory32/valid_zones:Normal
> /sys/devices/system/memory/memory33/valid_zones:Normal Movable
> /sys/devices/system/memory/memory34/valid_zones:Movable Normal
> 
> This is an awkward semantic because an udev event is sent as soon as the
> block is onlined and an udev handler might want to online it based on
> some policy (e.g. association with a node) but it will inherently race
> with new blocks showing up.
> 
> This patch changes the physical online phase to not associate pages
> with any zone at all. All the pages are just marked reserved and wait
> for the onlining phase to be associated with the zone as per the online
> request. There are only two requirements
> 	- existing ZONE_NORMAL and ZONE_MOVABLE cannot overlap
> 	- ZONE_NORMAL precedes ZONE_MOVABLE in physical addresses
> the later on is not an inherent requirement and can be changed in the
> future. It preserves the current behavior and made the code slightly
> simpler. This is subject to change in future.
> 
> This means that the same physical online steps as above will lead to the
> following state:
> Normal Movable
> 
> /sys/devices/system/memory/memory32/valid_zones:Normal Movable
> /sys/devices/system/memory/memory33/valid_zones:Normal Movable
> 
> /sys/devices/system/memory/memory32/valid_zones:Normal Movable
> /sys/devices/system/memory/memory33/valid_zones:Normal Movable
> /sys/devices/system/memory/memory34/valid_zones:Normal Movable
> 
> /sys/devices/system/memory/memory32/valid_zones:Normal Movable
> /sys/devices/system/memory/memory33/valid_zones:Normal Movable
> /sys/devices/system/memory/memory34/valid_zones:Movable
> 
> Implementation:
> The current move_pfn_range is reimplemented to check the above
> requirements (allow_online_pfn_range) and then updates the respective
> zone (move_pfn_range_to_zone), the pgdat and links all the pages in the
> pfn range with the zone/node. __add_pages is updated to not require the
> zone and only initializes sections in the range. This allowed to
> simplify the arch_add_memory code (s390 could get rid of quite some
> of code).
> 
> devm_memremap_pages is the only user of arch_add_memory which relies
> on the zone association because it only hooks into the memory hotplug
> only half way. It uses it to associate the new memory with ZONE_DEVICE
> but doesn't allow it to be {on,off}lined via sysfs. This means that this
> particular code path has to call move_pfn_range_to_zone explicitly.
> 
> The original zone shifting code is kept in place and will be removed in
> the follow up patch for an easier review.
> 
> Please note that this patch also changes the original behavior when
> offlining a memory block adjacent to another zone (Normal vs. Movable)
> used to allow to change its movable type. This will be handled later.
> 
> Changes since v1
> - we have to associate the page with the node early (in __add_section),
>   because pfn_to_node depends on struct page containing this
>   information - based on testing by Reza Arbab
> - resize_{zone,pgdat}_range has to check whether they are popoulated -
>   Reza Arbab
> - fix devm_memremap_pages to use pfn rather than physical address -
>   JA(C)rA'me Glisse
> - move_pfn_range has to check for intersection with zone_movable rather
>   than to rely on allow_online_pfn_range(MMOP_ONLINE_MOVABLE) for
>   MMOP_ONLINE_KEEP
> 
> Changes since v2
> - fix show_valid_zones nr_pages calculation
> - allow_online_pfn_range has to check managed pages rather than present
> - zone_intersects fix bogus check
> - fix zone_intersects + few nits as per Vlastimil
> 
> Cc: Martin Schwidefsky <schwidefsky@de.ibm.com>
> Cc: linux-arch@vger.kernel.org
> Tested-by: Dan Williams <dan.j.williams@intel.com>
> Acked-by: Heiko Carstens <heiko.carstens@de.ibm.com> # For s390 bits
> Tested-by: Reza Arbab <arbab@linux.vnet.ibm.com>
> Signed-off-by: Michal Hocko <mhocko@suse.com>

Acked-by: Vlastimil Babka <vbabka@suse.cz>

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

^ permalink raw reply	[flat|nested] 74+ messages in thread

* Re: [PATCH -v4 0/14] mm: make movable onlining suck less
  2017-05-15  8:58 ` Michal Hocko
@ 2017-06-09  9:51   ` Wei Yang
  -1 siblings, 0 replies; 74+ messages in thread
From: Wei Yang @ 2017-06-09  9:51 UTC (permalink / raw)
  To: Michal Hocko
  Cc: Andrew Morton, Linux-MM, Mel Gorman, Vlastimil Babka,
	Andrea Arcangeli, Jerome Glisse, Reza Arbab, Yasuaki Ishimatsu,
	qiuxishi, Kani Toshimitsu, slaoub, Joonsoo Kim, Andi Kleen,
	David Rientjes, Daniel Kiper, Igor Mammedov, Vitaly Kuznetsov,
	LKML, Balbir Singh, Dan Williams, Heiko Carstens,
	Martin Schwidefsky, Michal Hocko, Tobias Regnery,
	Yasuaki Ishimatsu

Hi, Michal

I am not that familiar with hotplug and trying to catch up the issue
 and your solution.

One potential issue I found is we don't check the physical boundary
when add_memory_resource().

For example, on x86-64, only 64T physical memory is supported currently.
Looks it is expanded after 5-level pagetable is introduced. While there is
still some limitations on this. But we don't check the boundary I think.

During the bootup, this is ensured by the max_pfn which is guaranteed to
be under MAX_ARCH_PFN. I don't see some limitation on this when doing
 hotplug.

Here is my very simple check on this, while it seems not every arch has
 MAX_ARCH_PFN. Willing to hear from you.

diff --git a/mm/memory_hotplug.c b/mm/memory_hotplug.c
index 6fa7208bcd56..27541566f9ac 100644
--- a/mm/memory_hotplug.c
+++ b/mm/memory_hotplug.c
@@ -1355,7 +1355,12 @@ int __ref add_memory_resource(int nid, struct
resource *res, bool online)
        int ret;

        start = res->start;
+       if (start > (MAX_ARCH_PFN << PAGE_SHIFT))
+               return -EINVAL;
+
        size = resource_size(res);
+       if ((start + size) > (MAX_ARCH_PFN << PAGE_SHIFT))
+               size = (MAX_ARCH_PFN << PAGE_SHIFT) - start;

        ret = check_hotplug_memory_range(start, size);
        if (ret)

On Mon, May 15, 2017 at 4:58 PM, Michal Hocko <mhocko@kernel.org> wrote:
> Hi,
> The last version of this series has been posted here [1]. The timing wasn't
> all that great so this is mostly a resubmit. I've added one additional patch
> to fix another pfn walker noticed by Joonsoo (this is patch 11) and also
> added a clarification for pfn_valid() and offline pages.
>
> There is still a lot of work on top - namely this implementation doesn't
> support reonlining to a different zone on the zones boundaries but I
> will do that in a separate series because this one is getting quite
> large already and it should work reasonably well now.
>
> Joonsoo had some worries about pfn_valid and suggested to change its
> semantic to return false on offline holes but I would be rally worried
> to change a established semantic used by a lot of code and so I have
> introuduced pfn_to_online_page helper instead. If this is seen as a
> controversial point I would rather drop pfn_to_online_page and related
> patches as they are not stictly necessary because the code would be
> similarly broken as now wrt. offline holes.
>
> This is a rebase on top of the current mmotm tree (mmotm-2017-05-12-15-53)
> and the full series is in
> git://git.kernel.org/pub/scm/linux/kernel/git/mhocko/mm.git try
> attempts/rewrite-mem_hotplug branch.
>
> Motivation:
> Movable onlining is a real hack with many downsides - mainly
> reintroduction of lowmem/highmem issues we used to have on 32b systems -
> but it is the only way to make the memory hotremove more reliable which
> is something that people are asking for.
>
> The current semantic of memory movable onlinening is really cumbersome,
> however. The main reason for this is that the udev driven approach is
> basically unusable because udev races with the memory probing while only
> the last memory block or the one adjacent to the existing zone_movable
> are allowed to be onlined movable. In short the criterion for the
> successful online_movable changes under udev's feet. A reliable udev
> approach would require a 2 phase approach where the first successful
> movable online would have to check all the previous blocks and online
> them in descending order. This is hard to be considered sane.
>
> This patchset aims at making the onlining semantic more usable. First of
> all it allows to online memory movable as long as it doesn't clash with
> the existing ZONE_NORMAL. That means that ZONE_NORMAL and ZONE_MOVABLE
> cannot overlap. Currently I preserve the original ordering semantic so
> the zone always precedes the movable zone but I have plans to remove this
> restriction in future because it is not really necessary.
>
> First 3 patches are cleanups which should be ready to be merged right
> away (unless I have missed something subtle of course).
>
> Patch 4 deals with ZONE_DEVICE dependencies down the __add_pages path.
>
> Patch 5 deals with implicit assumptions of register_one_node on pgdat
> initialization.
>
> Patches 6-10 deal with offline holes in the zone for pfn walkers. I
> hope I got all of them right but people familiar with compaction should
> double check this.
>
> Patch 11 is the core of the change. In order to make it easier to review
> I have tried it to be as minimalistic as possible and the large code
> removal is moved to patch 14.
>
> Patch 12 is a trivial follow up cleanup. Patch 13 fixes sparse warnings
> and finally patch 14 removes the unused code.
>
> I have tested the patches in kvm:
> # qemu-system-x86_64 -enable-kvm -monitor pty -m 2G,slots=4,maxmem=4G -numa node,mem=1G -numa node,mem=1G ...
>
> and then probed the additional memory by
> (qemu) object_add memory-backend-ram,id=mem1,size=1G
> (qemu) device_add pc-dimm,id=dimm1,memdev=mem1
>
> Then I have used this simple script to probe the memory block by hand
> # cat probe_memblock.sh
> #!/bin/sh
>
> BLOCK_NR=$1
>
> # echo $((0x100000000+$BLOCK_NR*(128<<20))) > /sys/devices/system/memory/probe
>
> # for i in $(seq 10); do sh probe_memblock.sh $i; done
> # grep . /sys/devices/system/memory/memory3?/valid_zones 2>/dev/null
> /sys/devices/system/memory/memory33/valid_zones:Normal Movable
> /sys/devices/system/memory/memory34/valid_zones:Normal Movable
> /sys/devices/system/memory/memory35/valid_zones:Normal Movable
> /sys/devices/system/memory/memory36/valid_zones:Normal Movable
> /sys/devices/system/memory/memory37/valid_zones:Normal Movable
> /sys/devices/system/memory/memory38/valid_zones:Normal Movable
> /sys/devices/system/memory/memory39/valid_zones:Normal Movable
>
> The main difference to the original implementation is that all new
> memblocks can be both online_kernel and online_movable initially
> because there is no clash obviously. For the comparison the original
> implementation would have
>
> /sys/devices/system/memory/memory33/valid_zones:Normal
> /sys/devices/system/memory/memory34/valid_zones:Normal
> /sys/devices/system/memory/memory35/valid_zones:Normal
> /sys/devices/system/memory/memory36/valid_zones:Normal
> /sys/devices/system/memory/memory37/valid_zones:Normal
> /sys/devices/system/memory/memory38/valid_zones:Normal
> /sys/devices/system/memory/memory39/valid_zones:Normal Movable
>
> Now
> # echo online_movable > /sys/devices/system/memory/memory34/state
> # grep . /sys/devices/system/memory/memory3?/valid_zones 2>/dev/null
> /sys/devices/system/memory/memory33/valid_zones:Normal Movable
> /sys/devices/system/memory/memory34/valid_zones:Movable
> /sys/devices/system/memory/memory35/valid_zones:Movable
> /sys/devices/system/memory/memory36/valid_zones:Movable
> /sys/devices/system/memory/memory37/valid_zones:Movable
> /sys/devices/system/memory/memory38/valid_zones:Movable
> /sys/devices/system/memory/memory39/valid_zones:Movable
>
> Block 33 can still be online both kernel and movable while all
> the remaining can be only movable.
> /proc/zonelist says
> Node 0, zone   Normal
>   pages free     0
>         min      0
>         low      0
>         high     0
>         spanned  0
>         present  0
> --
> Node 0, zone  Movable
>   pages free     32753
>         min      85
>         low      117
>         high     149
>         spanned  32768
>         present  32768
>
> A new memblock at a lower address will result in a new memblock (32)
> which will still allow both Normal and Movable.
> # sh probe_memblock.sh 0
> # grep . /sys/devices/system/memory/memory3[2-5]/valid_zones 2>/dev/null
> /sys/devices/system/memory/memory32/valid_zones:Normal Movable
> /sys/devices/system/memory/memory33/valid_zones:Normal Movable
> /sys/devices/system/memory/memory34/valid_zones:Movable
> /sys/devices/system/memory/memory35/valid_zones:Movable
>
> and online_kernel will convert it to the zone normal properly
> while 33 can be still onlined both ways.
> # echo online_kernel > /sys/devices/system/memory/memory32/state
> # grep . /sys/devices/system/memory/memory3[2-5]/valid_zones 2>/dev/null
> /sys/devices/system/memory/memory32/valid_zones:Normal
> /sys/devices/system/memory/memory33/valid_zones:Normal Movable
> /sys/devices/system/memory/memory34/valid_zones:Movable
> /sys/devices/system/memory/memory35/valid_zones:Movable
>
> /proc/zoneinfo will now tell
> Node 0, zone   Normal
>   pages free     65441
>         min      165
>         low      230
>         high     295
>         spanned  65536
>         present  65536
> --
> Node 0, zone  Movable
>   pages free     32740
>         min      82
>         low      114
>         high     146
>         spanned  32768
>         present  32768
>
> so both zones have one memblock spanned and present.
>
> Onlining 39 should associate this block to the movable zone
> # echo online > /sys/devices/system/memory/memory39/state
>
> /proc/zoneinfo will now tell
> Node 0, zone   Normal
>   pages free     32765
>         min      80
>         low      112
>         high     144
>         spanned  32768
>         present  32768
> --
> Node 0, zone  Movable
>   pages free     65501
>         min      160
>         low      225
>         high     290
>         spanned  196608
>         present  65536
>
> so we will have a movable zone which spans 6 memblocks, 2 present and 4
> representing a hole.
>
> Offlining both movable blocks will lead to the zone with no present
> pages which is the expected behavior I believe.
> # echo offline > /sys/devices/system/memory/memory39/state
> # echo offline > /sys/devices/system/memory/memory34/state
> # grep -A6 "Movable\|Normal" /proc/zoneinfo
> Node 0, zone   Normal
>   pages free     32735
>         min      90
>         low      122
>         high     154
>         spanned  32768
>         present  32768
> --
> Node 0, zone  Movable
>   pages free     0
>         min      0
>         low      0
>         high     0
>         spanned  196608
>         present  0
>
> Any thoughts, complains, suggestions?
>
> As a bonus we will get a nice cleanup in the memory hotplug codebase.
>  arch/ia64/mm/init.c            |  11 +-
>  arch/powerpc/mm/mem.c          |  12 +-
>  arch/s390/mm/init.c            |  32 +--
>  arch/sh/mm/init.c              |  10 +-
>  arch/x86/mm/init_32.c          |   7 +-
>  arch/x86/mm/init_64.c          |  11 +-
>  drivers/base/memory.c          |  79 +++----
>  drivers/base/node.c            |  58 ++----
>  include/linux/memory_hotplug.h |  40 +++-
>  include/linux/mmzone.h         |  57 +++++-
>  include/linux/node.h           |  35 +++-
>  kernel/memremap.c              |   6 +-
>  mm/compaction.c                |   5 +-
>  mm/memory_hotplug.c            | 455 ++++++++++++++---------------------------
>  mm/page_alloc.c                |  13 +-
>  mm/page_isolation.c            |  26 ++-
>  mm/sparse.c                    |  48 ++++-
>  mm/vmstat.c                    |   4 +-
>  18 files changed, 417 insertions(+), 492 deletions(-)
>
> Shortlog says:
> Michal Hocko (14):
>       mm: remove return value from init_currently_empty_zone
>       mm, memory_hotplug: use node instead of zone in can_online_high_movable
>       mm: drop page_initialized check from get_nid_for_pfn
>       mm, memory_hotplug: get rid of is_zone_device_section
>       mm, memory_hotplug: split up register_one_node
>       mm, memory_hotplug: consider offline memblocks removable
>       mm: consider zone which is not fully populated to have holes
>       mm, compaction: skip over holes in __reset_isolation_suitable
>       mm: __first_valid_page skip over offline pages
>       mm, vmstat: skip reporting offline pages in pagetypeinfo
>       mm, memory_hotplug: do not associate hotadded memory to zones until online
>       mm, memory_hotplug: replace for_device by want_memblock in arch_add_memory
>       mm, memory_hotplug: fix the section mismatch warning
>       mm, memory_hotplug: remove unused cruft after memory hotplug rework
>
> [1] http://lkml.kernel.org/r/20170421120512.23960-1-mhocko@kernel.org
>
>

^ permalink raw reply	[flat|nested] 74+ messages in thread

* Re: [PATCH -v4 0/14] mm: make movable onlining suck less
@ 2017-06-09  9:51   ` Wei Yang
  0 siblings, 0 replies; 74+ messages in thread
From: Wei Yang @ 2017-06-09  9:51 UTC (permalink / raw)
  To: Michal Hocko
  Cc: Andrew Morton, Linux-MM, Mel Gorman, Vlastimil Babka,
	Andrea Arcangeli, Jerome Glisse, Reza Arbab, Yasuaki Ishimatsu,
	qiuxishi, Kani Toshimitsu, slaoub, Joonsoo Kim, Andi Kleen,
	David Rientjes, Daniel Kiper, Igor Mammedov, Vitaly Kuznetsov,
	LKML, Balbir Singh, Dan Williams, Heiko Carstens,
	Martin Schwidefsky, Michal Hocko, Tobias Regnery,
	Yasuaki Ishimatsu

Hi, Michal

I am not that familiar with hotplug and trying to catch up the issue
 and your solution.

One potential issue I found is we don't check the physical boundary
when add_memory_resource().

For example, on x86-64, only 64T physical memory is supported currently.
Looks it is expanded after 5-level pagetable is introduced. While there is
still some limitations on this. But we don't check the boundary I think.

During the bootup, this is ensured by the max_pfn which is guaranteed to
be under MAX_ARCH_PFN. I don't see some limitation on this when doing
 hotplug.

Here is my very simple check on this, while it seems not every arch has
 MAX_ARCH_PFN. Willing to hear from you.

diff --git a/mm/memory_hotplug.c b/mm/memory_hotplug.c
index 6fa7208bcd56..27541566f9ac 100644
--- a/mm/memory_hotplug.c
+++ b/mm/memory_hotplug.c
@@ -1355,7 +1355,12 @@ int __ref add_memory_resource(int nid, struct
resource *res, bool online)
        int ret;

        start = res->start;
+       if (start > (MAX_ARCH_PFN << PAGE_SHIFT))
+               return -EINVAL;
+
        size = resource_size(res);
+       if ((start + size) > (MAX_ARCH_PFN << PAGE_SHIFT))
+               size = (MAX_ARCH_PFN << PAGE_SHIFT) - start;

        ret = check_hotplug_memory_range(start, size);
        if (ret)

On Mon, May 15, 2017 at 4:58 PM, Michal Hocko <mhocko@kernel.org> wrote:
> Hi,
> The last version of this series has been posted here [1]. The timing wasn't
> all that great so this is mostly a resubmit. I've added one additional patch
> to fix another pfn walker noticed by Joonsoo (this is patch 11) and also
> added a clarification for pfn_valid() and offline pages.
>
> There is still a lot of work on top - namely this implementation doesn't
> support reonlining to a different zone on the zones boundaries but I
> will do that in a separate series because this one is getting quite
> large already and it should work reasonably well now.
>
> Joonsoo had some worries about pfn_valid and suggested to change its
> semantic to return false on offline holes but I would be rally worried
> to change a established semantic used by a lot of code and so I have
> introuduced pfn_to_online_page helper instead. If this is seen as a
> controversial point I would rather drop pfn_to_online_page and related
> patches as they are not stictly necessary because the code would be
> similarly broken as now wrt. offline holes.
>
> This is a rebase on top of the current mmotm tree (mmotm-2017-05-12-15-53)
> and the full series is in
> git://git.kernel.org/pub/scm/linux/kernel/git/mhocko/mm.git try
> attempts/rewrite-mem_hotplug branch.
>
> Motivation:
> Movable onlining is a real hack with many downsides - mainly
> reintroduction of lowmem/highmem issues we used to have on 32b systems -
> but it is the only way to make the memory hotremove more reliable which
> is something that people are asking for.
>
> The current semantic of memory movable onlinening is really cumbersome,
> however. The main reason for this is that the udev driven approach is
> basically unusable because udev races with the memory probing while only
> the last memory block or the one adjacent to the existing zone_movable
> are allowed to be onlined movable. In short the criterion for the
> successful online_movable changes under udev's feet. A reliable udev
> approach would require a 2 phase approach where the first successful
> movable online would have to check all the previous blocks and online
> them in descending order. This is hard to be considered sane.
>
> This patchset aims at making the onlining semantic more usable. First of
> all it allows to online memory movable as long as it doesn't clash with
> the existing ZONE_NORMAL. That means that ZONE_NORMAL and ZONE_MOVABLE
> cannot overlap. Currently I preserve the original ordering semantic so
> the zone always precedes the movable zone but I have plans to remove this
> restriction in future because it is not really necessary.
>
> First 3 patches are cleanups which should be ready to be merged right
> away (unless I have missed something subtle of course).
>
> Patch 4 deals with ZONE_DEVICE dependencies down the __add_pages path.
>
> Patch 5 deals with implicit assumptions of register_one_node on pgdat
> initialization.
>
> Patches 6-10 deal with offline holes in the zone for pfn walkers. I
> hope I got all of them right but people familiar with compaction should
> double check this.
>
> Patch 11 is the core of the change. In order to make it easier to review
> I have tried it to be as minimalistic as possible and the large code
> removal is moved to patch 14.
>
> Patch 12 is a trivial follow up cleanup. Patch 13 fixes sparse warnings
> and finally patch 14 removes the unused code.
>
> I have tested the patches in kvm:
> # qemu-system-x86_64 -enable-kvm -monitor pty -m 2G,slots=4,maxmem=4G -numa node,mem=1G -numa node,mem=1G ...
>
> and then probed the additional memory by
> (qemu) object_add memory-backend-ram,id=mem1,size=1G
> (qemu) device_add pc-dimm,id=dimm1,memdev=mem1
>
> Then I have used this simple script to probe the memory block by hand
> # cat probe_memblock.sh
> #!/bin/sh
>
> BLOCK_NR=$1
>
> # echo $((0x100000000+$BLOCK_NR*(128<<20))) > /sys/devices/system/memory/probe
>
> # for i in $(seq 10); do sh probe_memblock.sh $i; done
> # grep . /sys/devices/system/memory/memory3?/valid_zones 2>/dev/null
> /sys/devices/system/memory/memory33/valid_zones:Normal Movable
> /sys/devices/system/memory/memory34/valid_zones:Normal Movable
> /sys/devices/system/memory/memory35/valid_zones:Normal Movable
> /sys/devices/system/memory/memory36/valid_zones:Normal Movable
> /sys/devices/system/memory/memory37/valid_zones:Normal Movable
> /sys/devices/system/memory/memory38/valid_zones:Normal Movable
> /sys/devices/system/memory/memory39/valid_zones:Normal Movable
>
> The main difference to the original implementation is that all new
> memblocks can be both online_kernel and online_movable initially
> because there is no clash obviously. For the comparison the original
> implementation would have
>
> /sys/devices/system/memory/memory33/valid_zones:Normal
> /sys/devices/system/memory/memory34/valid_zones:Normal
> /sys/devices/system/memory/memory35/valid_zones:Normal
> /sys/devices/system/memory/memory36/valid_zones:Normal
> /sys/devices/system/memory/memory37/valid_zones:Normal
> /sys/devices/system/memory/memory38/valid_zones:Normal
> /sys/devices/system/memory/memory39/valid_zones:Normal Movable
>
> Now
> # echo online_movable > /sys/devices/system/memory/memory34/state
> # grep . /sys/devices/system/memory/memory3?/valid_zones 2>/dev/null
> /sys/devices/system/memory/memory33/valid_zones:Normal Movable
> /sys/devices/system/memory/memory34/valid_zones:Movable
> /sys/devices/system/memory/memory35/valid_zones:Movable
> /sys/devices/system/memory/memory36/valid_zones:Movable
> /sys/devices/system/memory/memory37/valid_zones:Movable
> /sys/devices/system/memory/memory38/valid_zones:Movable
> /sys/devices/system/memory/memory39/valid_zones:Movable
>
> Block 33 can still be online both kernel and movable while all
> the remaining can be only movable.
> /proc/zonelist says
> Node 0, zone   Normal
>   pages free     0
>         min      0
>         low      0
>         high     0
>         spanned  0
>         present  0
> --
> Node 0, zone  Movable
>   pages free     32753
>         min      85
>         low      117
>         high     149
>         spanned  32768
>         present  32768
>
> A new memblock at a lower address will result in a new memblock (32)
> which will still allow both Normal and Movable.
> # sh probe_memblock.sh 0
> # grep . /sys/devices/system/memory/memory3[2-5]/valid_zones 2>/dev/null
> /sys/devices/system/memory/memory32/valid_zones:Normal Movable
> /sys/devices/system/memory/memory33/valid_zones:Normal Movable
> /sys/devices/system/memory/memory34/valid_zones:Movable
> /sys/devices/system/memory/memory35/valid_zones:Movable
>
> and online_kernel will convert it to the zone normal properly
> while 33 can be still onlined both ways.
> # echo online_kernel > /sys/devices/system/memory/memory32/state
> # grep . /sys/devices/system/memory/memory3[2-5]/valid_zones 2>/dev/null
> /sys/devices/system/memory/memory32/valid_zones:Normal
> /sys/devices/system/memory/memory33/valid_zones:Normal Movable
> /sys/devices/system/memory/memory34/valid_zones:Movable
> /sys/devices/system/memory/memory35/valid_zones:Movable
>
> /proc/zoneinfo will now tell
> Node 0, zone   Normal
>   pages free     65441
>         min      165
>         low      230
>         high     295
>         spanned  65536
>         present  65536
> --
> Node 0, zone  Movable
>   pages free     32740
>         min      82
>         low      114
>         high     146
>         spanned  32768
>         present  32768
>
> so both zones have one memblock spanned and present.
>
> Onlining 39 should associate this block to the movable zone
> # echo online > /sys/devices/system/memory/memory39/state
>
> /proc/zoneinfo will now tell
> Node 0, zone   Normal
>   pages free     32765
>         min      80
>         low      112
>         high     144
>         spanned  32768
>         present  32768
> --
> Node 0, zone  Movable
>   pages free     65501
>         min      160
>         low      225
>         high     290
>         spanned  196608
>         present  65536
>
> so we will have a movable zone which spans 6 memblocks, 2 present and 4
> representing a hole.
>
> Offlining both movable blocks will lead to the zone with no present
> pages which is the expected behavior I believe.
> # echo offline > /sys/devices/system/memory/memory39/state
> # echo offline > /sys/devices/system/memory/memory34/state
> # grep -A6 "Movable\|Normal" /proc/zoneinfo
> Node 0, zone   Normal
>   pages free     32735
>         min      90
>         low      122
>         high     154
>         spanned  32768
>         present  32768
> --
> Node 0, zone  Movable
>   pages free     0
>         min      0
>         low      0
>         high     0
>         spanned  196608
>         present  0
>
> Any thoughts, complains, suggestions?
>
> As a bonus we will get a nice cleanup in the memory hotplug codebase.
>  arch/ia64/mm/init.c            |  11 +-
>  arch/powerpc/mm/mem.c          |  12 +-
>  arch/s390/mm/init.c            |  32 +--
>  arch/sh/mm/init.c              |  10 +-
>  arch/x86/mm/init_32.c          |   7 +-
>  arch/x86/mm/init_64.c          |  11 +-
>  drivers/base/memory.c          |  79 +++----
>  drivers/base/node.c            |  58 ++----
>  include/linux/memory_hotplug.h |  40 +++-
>  include/linux/mmzone.h         |  57 +++++-
>  include/linux/node.h           |  35 +++-
>  kernel/memremap.c              |   6 +-
>  mm/compaction.c                |   5 +-
>  mm/memory_hotplug.c            | 455 ++++++++++++++---------------------------
>  mm/page_alloc.c                |  13 +-
>  mm/page_isolation.c            |  26 ++-
>  mm/sparse.c                    |  48 ++++-
>  mm/vmstat.c                    |   4 +-
>  18 files changed, 417 insertions(+), 492 deletions(-)
>
> Shortlog says:
> Michal Hocko (14):
>       mm: remove return value from init_currently_empty_zone
>       mm, memory_hotplug: use node instead of zone in can_online_high_movable
>       mm: drop page_initialized check from get_nid_for_pfn
>       mm, memory_hotplug: get rid of is_zone_device_section
>       mm, memory_hotplug: split up register_one_node
>       mm, memory_hotplug: consider offline memblocks removable
>       mm: consider zone which is not fully populated to have holes
>       mm, compaction: skip over holes in __reset_isolation_suitable
>       mm: __first_valid_page skip over offline pages
>       mm, vmstat: skip reporting offline pages in pagetypeinfo
>       mm, memory_hotplug: do not associate hotadded memory to zones until online
>       mm, memory_hotplug: replace for_device by want_memblock in arch_add_memory
>       mm, memory_hotplug: fix the section mismatch warning
>       mm, memory_hotplug: remove unused cruft after memory hotplug rework
>
> [1] http://lkml.kernel.org/r/20170421120512.23960-1-mhocko@kernel.org
>
>

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

^ permalink raw reply	[flat|nested] 74+ messages in thread

* Re: [PATCH -v4 0/14] mm: make movable onlining suck less
  2017-06-09  9:51   ` Wei Yang
@ 2017-06-09 10:49     ` Michal Hocko
  -1 siblings, 0 replies; 74+ messages in thread
From: Michal Hocko @ 2017-06-09 10:49 UTC (permalink / raw)
  To: Wei Yang
  Cc: Andrew Morton, Linux-MM, Mel Gorman, Vlastimil Babka,
	Andrea Arcangeli, Jerome Glisse, Reza Arbab, Yasuaki Ishimatsu,
	qiuxishi, Kani Toshimitsu, slaoub, Joonsoo Kim, Andi Kleen,
	David Rientjes, Daniel Kiper, Igor Mammedov, Vitaly Kuznetsov,
	LKML, Balbir Singh, Dan Williams, Heiko Carstens,
	Martin Schwidefsky, Tobias Regnery, Yasuaki Ishimatsu

On Fri 09-06-17 17:51:24, Wei Yang wrote:
> Hi, Michal
> 
> I am not that familiar with hotplug and trying to catch up the issue
>  and your solution.
> 
> One potential issue I found is we don't check the physical boundary
> when add_memory_resource().
> 
> For example, on x86-64, only 64T physical memory is supported currently.
> Looks it is expanded after 5-level pagetable is introduced. While there is
> still some limitations on this. But we don't check the boundary I think.
> 
> During the bootup, this is ensured by the max_pfn which is guaranteed to
> be under MAX_ARCH_PFN. I don't see some limitation on this when doing
>  hotplug.

This might be true and I would have to double check but this rework
doesn't change anything in that regards. Or do I miss something?
-- 
Michal Hocko
SUSE Labs

^ permalink raw reply	[flat|nested] 74+ messages in thread

* Re: [PATCH -v4 0/14] mm: make movable onlining suck less
@ 2017-06-09 10:49     ` Michal Hocko
  0 siblings, 0 replies; 74+ messages in thread
From: Michal Hocko @ 2017-06-09 10:49 UTC (permalink / raw)
  To: Wei Yang
  Cc: Andrew Morton, Linux-MM, Mel Gorman, Vlastimil Babka,
	Andrea Arcangeli, Jerome Glisse, Reza Arbab, Yasuaki Ishimatsu,
	qiuxishi, Kani Toshimitsu, slaoub, Joonsoo Kim, Andi Kleen,
	David Rientjes, Daniel Kiper, Igor Mammedov, Vitaly Kuznetsov,
	LKML, Balbir Singh, Dan Williams, Heiko Carstens,
	Martin Schwidefsky, Tobias Regnery, Yasuaki Ishimatsu

On Fri 09-06-17 17:51:24, Wei Yang wrote:
> Hi, Michal
> 
> I am not that familiar with hotplug and trying to catch up the issue
>  and your solution.
> 
> One potential issue I found is we don't check the physical boundary
> when add_memory_resource().
> 
> For example, on x86-64, only 64T physical memory is supported currently.
> Looks it is expanded after 5-level pagetable is introduced. While there is
> still some limitations on this. But we don't check the boundary I think.
> 
> During the bootup, this is ensured by the max_pfn which is guaranteed to
> be under MAX_ARCH_PFN. I don't see some limitation on this when doing
>  hotplug.

This might be true and I would have to double check but this rework
doesn't change anything in that regards. Or do I miss something?
-- 
Michal Hocko
SUSE Labs

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

^ permalink raw reply	[flat|nested] 74+ messages in thread

* Re: [PATCH -v4 0/14] mm: make movable onlining suck less
  2017-06-09 10:49     ` Michal Hocko
@ 2017-06-10  2:20       ` Wei Yang
  -1 siblings, 0 replies; 74+ messages in thread
From: Wei Yang @ 2017-06-10  2:20 UTC (permalink / raw)
  To: Michal Hocko
  Cc: Andrew Morton, Linux-MM, Mel Gorman, Vlastimil Babka,
	Andrea Arcangeli, Jerome Glisse, Reza Arbab, Yasuaki Ishimatsu,
	qiuxishi, Kani Toshimitsu, slaoub, Joonsoo Kim, Andi Kleen,
	David Rientjes, Daniel Kiper, Igor Mammedov, Vitaly Kuznetsov,
	LKML, Balbir Singh, Dan Williams, Heiko Carstens,
	Martin Schwidefsky, Tobias Regnery, Yasuaki Ishimatsu

On Fri, Jun 9, 2017 at 6:49 PM, Michal Hocko <mhocko@kernel.org> wrote:
> On Fri 09-06-17 17:51:24, Wei Yang wrote:
>> Hi, Michal
>>
>> I am not that familiar with hotplug and trying to catch up the issue
>>  and your solution.
>>
>> One potential issue I found is we don't check the physical boundary
>> when add_memory_resource().
>>
>> For example, on x86-64, only 64T physical memory is supported currently.
>> Looks it is expanded after 5-level pagetable is introduced. While there is
>> still some limitations on this. But we don't check the boundary I think.
>>
>> During the bootup, this is ensured by the max_pfn which is guaranteed to
>> be under MAX_ARCH_PFN. I don't see some limitation on this when doing
>>  hotplug.
>
> This might be true and I would have to double check but this rework
> doesn't change anything in that regards. Or do I miss something?

Ah, yes, I believe your patch set don't touch this area.

This is just related to hotplug.

> --
> Michal Hocko
> SUSE Labs

^ permalink raw reply	[flat|nested] 74+ messages in thread

* Re: [PATCH -v4 0/14] mm: make movable onlining suck less
@ 2017-06-10  2:20       ` Wei Yang
  0 siblings, 0 replies; 74+ messages in thread
From: Wei Yang @ 2017-06-10  2:20 UTC (permalink / raw)
  To: Michal Hocko
  Cc: Andrew Morton, Linux-MM, Mel Gorman, Vlastimil Babka,
	Andrea Arcangeli, Jerome Glisse, Reza Arbab, Yasuaki Ishimatsu,
	qiuxishi, Kani Toshimitsu, slaoub, Joonsoo Kim, Andi Kleen,
	David Rientjes, Daniel Kiper, Igor Mammedov, Vitaly Kuznetsov,
	LKML, Balbir Singh, Dan Williams, Heiko Carstens,
	Martin Schwidefsky, Tobias Regnery, Yasuaki Ishimatsu

On Fri, Jun 9, 2017 at 6:49 PM, Michal Hocko <mhocko@kernel.org> wrote:
> On Fri 09-06-17 17:51:24, Wei Yang wrote:
>> Hi, Michal
>>
>> I am not that familiar with hotplug and trying to catch up the issue
>>  and your solution.
>>
>> One potential issue I found is we don't check the physical boundary
>> when add_memory_resource().
>>
>> For example, on x86-64, only 64T physical memory is supported currently.
>> Looks it is expanded after 5-level pagetable is introduced. While there is
>> still some limitations on this. But we don't check the boundary I think.
>>
>> During the bootup, this is ensured by the max_pfn which is guaranteed to
>> be under MAX_ARCH_PFN. I don't see some limitation on this when doing
>>  hotplug.
>
> This might be true and I would have to double check but this rework
> doesn't change anything in that regards. Or do I miss something?

Ah, yes, I believe your patch set don't touch this area.

This is just related to hotplug.

> --
> Michal Hocko
> SUSE Labs

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

^ permalink raw reply	[flat|nested] 74+ messages in thread

* Re: [PATCH 04/14] mm, memory_hotplug: get rid of is_zone_device_section
  2017-05-15  8:58   ` Michal Hocko
@ 2017-06-10  9:56     ` Wei Yang
  -1 siblings, 0 replies; 74+ messages in thread
From: Wei Yang @ 2017-06-10  9:56 UTC (permalink / raw)
  To: Michal Hocko
  Cc: Andrew Morton, Linux-MM, Mel Gorman, Vlastimil Babka,
	Andrea Arcangeli, Jerome Glisse, Reza Arbab, Yasuaki Ishimatsu,
	qiuxishi, Kani Toshimitsu, slaoub, Joonsoo Kim, Andi Kleen,
	David Rientjes, Daniel Kiper, Igor Mammedov, Vitaly Kuznetsov,
	LKML, Michal Hocko, Dan Williams

On Mon, May 15, 2017 at 4:58 PM, Michal Hocko <mhocko@kernel.org> wrote:
> From: Michal Hocko <mhocko@suse.com>
>
> device memory hotplug hooks into regular memory hotplug only half way.
> It needs memory sections to track struct pages but there is no
> need/desire to associate those sections with memory blocks and export
> them to the userspace via sysfs because they cannot be onlined anyway.
>
> This is currently expressed by for_device argument to arch_add_memory
> which then makes sure to associate the given memory range with
> ZONE_DEVICE. register_new_memory then relies on is_zone_device_section
> to distinguish special memory hotplug from the regular one. While this
> works now, later patches in this series want to move __add_zone outside
> of arch_add_memory path so we have to come up with something else.
>
> Add want_memblock down the __add_pages path and use it to control
> whether the section->memblock association should be done. arch_add_memory
> then just trivially want memblock for everything but for_device hotplug.
>
> remove_memory_section doesn't need is_zone_device_section either. We can
> simply skip all the memblock specific cleanup if there is no memblock
> for the given section.
>
> This shouldn't introduce any functional change.
>

Hmm... one question about the memory_block behavior.

In case one memory_block contains more than one memory section.
If one section is "device zone", the whole memory_block is not visible
in sysfs. Or until the whole memory_block is full, the sysfs is visible.

This is the known behavior right?

And in this case, the memory_block could be found. Would this introduce
some problem when remove_memory_section()?


> Changes since v1
> - return 0 if want_memblock == 0 from __add_section as per Jerome Glisse
>
> Changes since v2
> - fix remove_memory_section unlock on find_memory_block failure
>   as per Jerome - spotted by Evgeny Baskakov
>
> Tested-by: Dan Williams <dan.j.williams@intel.com>
> Acked-by: Vlastimil Babka <vbabka@suse.cz>
> Signed-off-by: Michal Hocko <mhocko@suse.com>
> ---
>  arch/ia64/mm/init.c            |  2 +-
>  arch/powerpc/mm/mem.c          |  2 +-
>  arch/s390/mm/init.c            |  2 +-
>  arch/sh/mm/init.c              |  2 +-
>  arch/x86/mm/init_32.c          |  2 +-
>  arch/x86/mm/init_64.c          |  2 +-
>  drivers/base/memory.c          | 23 +++++++++--------------
>  include/linux/memory_hotplug.h |  2 +-
>  mm/memory_hotplug.c            |  9 ++++++---
>  9 files changed, 22 insertions(+), 24 deletions(-)
>
> diff --git a/arch/ia64/mm/init.c b/arch/ia64/mm/init.c
> index 8f3efa682ee8..39e2aeb4669d 100644
> --- a/arch/ia64/mm/init.c
> +++ b/arch/ia64/mm/init.c
> @@ -658,7 +658,7 @@ int arch_add_memory(int nid, u64 start, u64 size, bool for_device)
>
>         zone = pgdat->node_zones +
>                 zone_for_memory(nid, start, size, ZONE_NORMAL, for_device);
> -       ret = __add_pages(nid, zone, start_pfn, nr_pages);
> +       ret = __add_pages(nid, zone, start_pfn, nr_pages, !for_device);
>
>         if (ret)
>                 printk("%s: Problem encountered in __add_pages() as ret=%d\n",
> diff --git a/arch/powerpc/mm/mem.c b/arch/powerpc/mm/mem.c
> index 9ee536ec0739..e6b2e6618b6c 100644
> --- a/arch/powerpc/mm/mem.c
> +++ b/arch/powerpc/mm/mem.c
> @@ -151,7 +151,7 @@ int arch_add_memory(int nid, u64 start, u64 size, bool for_device)
>         zone = pgdata->node_zones +
>                 zone_for_memory(nid, start, size, 0, for_device);
>
> -       return __add_pages(nid, zone, start_pfn, nr_pages);
> +       return __add_pages(nid, zone, start_pfn, nr_pages, !for_device);
>  }
>
>  #ifdef CONFIG_MEMORY_HOTREMOVE
> diff --git a/arch/s390/mm/init.c b/arch/s390/mm/init.c
> index ee6a1d3d4983..893cf88cf02d 100644
> --- a/arch/s390/mm/init.c
> +++ b/arch/s390/mm/init.c
> @@ -191,7 +191,7 @@ int arch_add_memory(int nid, u64 start, u64 size, bool for_device)
>                         continue;
>                 nr_pages = (start_pfn + size_pages > zone_end_pfn) ?
>                            zone_end_pfn - start_pfn : size_pages;
> -               rc = __add_pages(nid, zone, start_pfn, nr_pages);
> +               rc = __add_pages(nid, zone, start_pfn, nr_pages, !for_device);
>                 if (rc)
>                         break;
>                 start_pfn += nr_pages;
> diff --git a/arch/sh/mm/init.c b/arch/sh/mm/init.c
> index 75491862d900..a9d57f75ae8c 100644
> --- a/arch/sh/mm/init.c
> +++ b/arch/sh/mm/init.c
> @@ -498,7 +498,7 @@ int arch_add_memory(int nid, u64 start, u64 size, bool for_device)
>         ret = __add_pages(nid, pgdat->node_zones +
>                         zone_for_memory(nid, start, size, ZONE_NORMAL,
>                         for_device),
> -                       start_pfn, nr_pages);
> +                       start_pfn, nr_pages, !for_device);
>         if (unlikely(ret))
>                 printk("%s: Failed, __add_pages() == %d\n", __func__, ret);
>
> diff --git a/arch/x86/mm/init_32.c b/arch/x86/mm/init_32.c
> index 99fb83819a5f..94594b889144 100644
> --- a/arch/x86/mm/init_32.c
> +++ b/arch/x86/mm/init_32.c
> @@ -831,7 +831,7 @@ int arch_add_memory(int nid, u64 start, u64 size, bool for_device)
>         unsigned long start_pfn = start >> PAGE_SHIFT;
>         unsigned long nr_pages = size >> PAGE_SHIFT;
>
> -       return __add_pages(nid, zone, start_pfn, nr_pages);
> +       return __add_pages(nid, zone, start_pfn, nr_pages, !for_device);
>  }
>
>  #ifdef CONFIG_MEMORY_HOTREMOVE
> diff --git a/arch/x86/mm/init_64.c b/arch/x86/mm/init_64.c
> index 41270b96403d..2e004364a373 100644
> --- a/arch/x86/mm/init_64.c
> +++ b/arch/x86/mm/init_64.c
> @@ -697,7 +697,7 @@ int arch_add_memory(int nid, u64 start, u64 size, bool for_device)
>
>         init_memory_mapping(start, start + size);
>
> -       ret = __add_pages(nid, zone, start_pfn, nr_pages);
> +       ret = __add_pages(nid, zone, start_pfn, nr_pages, !for_device);
>         WARN_ON_ONCE(ret);
>
>         /* update max_pfn, max_low_pfn and high_memory */
> diff --git a/drivers/base/memory.c b/drivers/base/memory.c
> index 90225ffee501..f8fd562c3f18 100644
> --- a/drivers/base/memory.c
> +++ b/drivers/base/memory.c
> @@ -685,14 +685,6 @@ static int add_memory_block(int base_section_nr)
>         return 0;
>  }
>
> -static bool is_zone_device_section(struct mem_section *ms)
> -{
> -       struct page *page;
> -
> -       page = sparse_decode_mem_map(ms->section_mem_map, __section_nr(ms));
> -       return is_zone_device_page(page);
> -}
> -
>  /*
>   * need an interface for the VM to add new memory regions,
>   * but without onlining it.
> @@ -702,9 +694,6 @@ int register_new_memory(int nid, struct mem_section *section)
>         int ret = 0;
>         struct memory_block *mem;
>
> -       if (is_zone_device_section(section))
> -               return 0;
> -
>         mutex_lock(&mem_sysfs_mutex);
>
>         mem = find_memory_block(section);
> @@ -741,11 +730,16 @@ static int remove_memory_section(unsigned long node_id,
>  {
>         struct memory_block *mem;
>
> -       if (is_zone_device_section(section))
> -               return 0;
> -
>         mutex_lock(&mem_sysfs_mutex);
> +
> +       /*
> +        * Some users of the memory hotplug do not want/need memblock to
> +        * track all sections. Skip over those.
> +        */
>         mem = find_memory_block(section);
> +       if (!mem)
> +               goto out_unlock;
> +
>         unregister_mem_sect_under_nodes(mem, __section_nr(section));
>
>         mem->section_count--;
> @@ -754,6 +748,7 @@ static int remove_memory_section(unsigned long node_id,
>         else
>                 put_device(&mem->dev);
>
> +out_unlock:
>         mutex_unlock(&mem_sysfs_mutex);
>         return 0;
>  }
> diff --git a/include/linux/memory_hotplug.h b/include/linux/memory_hotplug.h
> index 134a2f69c21a..3c8cf86201c3 100644
> --- a/include/linux/memory_hotplug.h
> +++ b/include/linux/memory_hotplug.h
> @@ -111,7 +111,7 @@ extern int __remove_pages(struct zone *zone, unsigned long start_pfn,
>
>  /* reasonably generic interface to expand the physical pages in a zone  */
>  extern int __add_pages(int nid, struct zone *zone, unsigned long start_pfn,
> -       unsigned long nr_pages);
> +       unsigned long nr_pages, bool want_memblock);
>
>  #ifdef CONFIG_NUMA
>  extern int memory_add_physaddr_to_nid(u64 start);
> diff --git a/mm/memory_hotplug.c b/mm/memory_hotplug.c
> index 6290d34b6331..a95120c56a9a 100644
> --- a/mm/memory_hotplug.c
> +++ b/mm/memory_hotplug.c
> @@ -494,7 +494,7 @@ static int __meminit __add_zone(struct zone *zone, unsigned long phys_start_pfn)
>  }
>
>  static int __meminit __add_section(int nid, struct zone *zone,
> -                                       unsigned long phys_start_pfn)
> +               unsigned long phys_start_pfn, bool want_memblock)
>  {
>         int ret;
>
> @@ -511,6 +511,9 @@ static int __meminit __add_section(int nid, struct zone *zone,
>         if (ret < 0)
>                 return ret;
>
> +       if (!want_memblock)
> +               return 0;
> +
>         return register_new_memory(nid, __pfn_to_section(phys_start_pfn));
>  }
>
> @@ -521,7 +524,7 @@ static int __meminit __add_section(int nid, struct zone *zone,
>   * add the new pages.
>   */
>  int __ref __add_pages(int nid, struct zone *zone, unsigned long phys_start_pfn,
> -                       unsigned long nr_pages)
> +                       unsigned long nr_pages, bool want_memblock)
>  {
>         unsigned long i;
>         int err = 0;
> @@ -549,7 +552,7 @@ int __ref __add_pages(int nid, struct zone *zone, unsigned long phys_start_pfn,
>         }
>
>         for (i = start_sec; i <= end_sec; i++) {
> -               err = __add_section(nid, zone, section_nr_to_pfn(i));
> +               err = __add_section(nid, zone, section_nr_to_pfn(i), want_memblock);
>
>                 /*
>                  * EEXIST is finally dealt with by ioresource collision
> --
> 2.11.0
>

^ permalink raw reply	[flat|nested] 74+ messages in thread

* Re: [PATCH 04/14] mm, memory_hotplug: get rid of is_zone_device_section
@ 2017-06-10  9:56     ` Wei Yang
  0 siblings, 0 replies; 74+ messages in thread
From: Wei Yang @ 2017-06-10  9:56 UTC (permalink / raw)
  To: Michal Hocko
  Cc: Andrew Morton, Linux-MM, Mel Gorman, Vlastimil Babka,
	Andrea Arcangeli, Jerome Glisse, Reza Arbab, Yasuaki Ishimatsu,
	qiuxishi, Kani Toshimitsu, slaoub, Joonsoo Kim, Andi Kleen,
	David Rientjes, Daniel Kiper, Igor Mammedov, Vitaly Kuznetsov,
	LKML, Michal Hocko, Dan Williams

On Mon, May 15, 2017 at 4:58 PM, Michal Hocko <mhocko@kernel.org> wrote:
> From: Michal Hocko <mhocko@suse.com>
>
> device memory hotplug hooks into regular memory hotplug only half way.
> It needs memory sections to track struct pages but there is no
> need/desire to associate those sections with memory blocks and export
> them to the userspace via sysfs because they cannot be onlined anyway.
>
> This is currently expressed by for_device argument to arch_add_memory
> which then makes sure to associate the given memory range with
> ZONE_DEVICE. register_new_memory then relies on is_zone_device_section
> to distinguish special memory hotplug from the regular one. While this
> works now, later patches in this series want to move __add_zone outside
> of arch_add_memory path so we have to come up with something else.
>
> Add want_memblock down the __add_pages path and use it to control
> whether the section->memblock association should be done. arch_add_memory
> then just trivially want memblock for everything but for_device hotplug.
>
> remove_memory_section doesn't need is_zone_device_section either. We can
> simply skip all the memblock specific cleanup if there is no memblock
> for the given section.
>
> This shouldn't introduce any functional change.
>

Hmm... one question about the memory_block behavior.

In case one memory_block contains more than one memory section.
If one section is "device zone", the whole memory_block is not visible
in sysfs. Or until the whole memory_block is full, the sysfs is visible.

This is the known behavior right?

And in this case, the memory_block could be found. Would this introduce
some problem when remove_memory_section()?


> Changes since v1
> - return 0 if want_memblock == 0 from __add_section as per Jerome Glisse
>
> Changes since v2
> - fix remove_memory_section unlock on find_memory_block failure
>   as per Jerome - spotted by Evgeny Baskakov
>
> Tested-by: Dan Williams <dan.j.williams@intel.com>
> Acked-by: Vlastimil Babka <vbabka@suse.cz>
> Signed-off-by: Michal Hocko <mhocko@suse.com>
> ---
>  arch/ia64/mm/init.c            |  2 +-
>  arch/powerpc/mm/mem.c          |  2 +-
>  arch/s390/mm/init.c            |  2 +-
>  arch/sh/mm/init.c              |  2 +-
>  arch/x86/mm/init_32.c          |  2 +-
>  arch/x86/mm/init_64.c          |  2 +-
>  drivers/base/memory.c          | 23 +++++++++--------------
>  include/linux/memory_hotplug.h |  2 +-
>  mm/memory_hotplug.c            |  9 ++++++---
>  9 files changed, 22 insertions(+), 24 deletions(-)
>
> diff --git a/arch/ia64/mm/init.c b/arch/ia64/mm/init.c
> index 8f3efa682ee8..39e2aeb4669d 100644
> --- a/arch/ia64/mm/init.c
> +++ b/arch/ia64/mm/init.c
> @@ -658,7 +658,7 @@ int arch_add_memory(int nid, u64 start, u64 size, bool for_device)
>
>         zone = pgdat->node_zones +
>                 zone_for_memory(nid, start, size, ZONE_NORMAL, for_device);
> -       ret = __add_pages(nid, zone, start_pfn, nr_pages);
> +       ret = __add_pages(nid, zone, start_pfn, nr_pages, !for_device);
>
>         if (ret)
>                 printk("%s: Problem encountered in __add_pages() as ret=%d\n",
> diff --git a/arch/powerpc/mm/mem.c b/arch/powerpc/mm/mem.c
> index 9ee536ec0739..e6b2e6618b6c 100644
> --- a/arch/powerpc/mm/mem.c
> +++ b/arch/powerpc/mm/mem.c
> @@ -151,7 +151,7 @@ int arch_add_memory(int nid, u64 start, u64 size, bool for_device)
>         zone = pgdata->node_zones +
>                 zone_for_memory(nid, start, size, 0, for_device);
>
> -       return __add_pages(nid, zone, start_pfn, nr_pages);
> +       return __add_pages(nid, zone, start_pfn, nr_pages, !for_device);
>  }
>
>  #ifdef CONFIG_MEMORY_HOTREMOVE
> diff --git a/arch/s390/mm/init.c b/arch/s390/mm/init.c
> index ee6a1d3d4983..893cf88cf02d 100644
> --- a/arch/s390/mm/init.c
> +++ b/arch/s390/mm/init.c
> @@ -191,7 +191,7 @@ int arch_add_memory(int nid, u64 start, u64 size, bool for_device)
>                         continue;
>                 nr_pages = (start_pfn + size_pages > zone_end_pfn) ?
>                            zone_end_pfn - start_pfn : size_pages;
> -               rc = __add_pages(nid, zone, start_pfn, nr_pages);
> +               rc = __add_pages(nid, zone, start_pfn, nr_pages, !for_device);
>                 if (rc)
>                         break;
>                 start_pfn += nr_pages;
> diff --git a/arch/sh/mm/init.c b/arch/sh/mm/init.c
> index 75491862d900..a9d57f75ae8c 100644
> --- a/arch/sh/mm/init.c
> +++ b/arch/sh/mm/init.c
> @@ -498,7 +498,7 @@ int arch_add_memory(int nid, u64 start, u64 size, bool for_device)
>         ret = __add_pages(nid, pgdat->node_zones +
>                         zone_for_memory(nid, start, size, ZONE_NORMAL,
>                         for_device),
> -                       start_pfn, nr_pages);
> +                       start_pfn, nr_pages, !for_device);
>         if (unlikely(ret))
>                 printk("%s: Failed, __add_pages() == %d\n", __func__, ret);
>
> diff --git a/arch/x86/mm/init_32.c b/arch/x86/mm/init_32.c
> index 99fb83819a5f..94594b889144 100644
> --- a/arch/x86/mm/init_32.c
> +++ b/arch/x86/mm/init_32.c
> @@ -831,7 +831,7 @@ int arch_add_memory(int nid, u64 start, u64 size, bool for_device)
>         unsigned long start_pfn = start >> PAGE_SHIFT;
>         unsigned long nr_pages = size >> PAGE_SHIFT;
>
> -       return __add_pages(nid, zone, start_pfn, nr_pages);
> +       return __add_pages(nid, zone, start_pfn, nr_pages, !for_device);
>  }
>
>  #ifdef CONFIG_MEMORY_HOTREMOVE
> diff --git a/arch/x86/mm/init_64.c b/arch/x86/mm/init_64.c
> index 41270b96403d..2e004364a373 100644
> --- a/arch/x86/mm/init_64.c
> +++ b/arch/x86/mm/init_64.c
> @@ -697,7 +697,7 @@ int arch_add_memory(int nid, u64 start, u64 size, bool for_device)
>
>         init_memory_mapping(start, start + size);
>
> -       ret = __add_pages(nid, zone, start_pfn, nr_pages);
> +       ret = __add_pages(nid, zone, start_pfn, nr_pages, !for_device);
>         WARN_ON_ONCE(ret);
>
>         /* update max_pfn, max_low_pfn and high_memory */
> diff --git a/drivers/base/memory.c b/drivers/base/memory.c
> index 90225ffee501..f8fd562c3f18 100644
> --- a/drivers/base/memory.c
> +++ b/drivers/base/memory.c
> @@ -685,14 +685,6 @@ static int add_memory_block(int base_section_nr)
>         return 0;
>  }
>
> -static bool is_zone_device_section(struct mem_section *ms)
> -{
> -       struct page *page;
> -
> -       page = sparse_decode_mem_map(ms->section_mem_map, __section_nr(ms));
> -       return is_zone_device_page(page);
> -}
> -
>  /*
>   * need an interface for the VM to add new memory regions,
>   * but without onlining it.
> @@ -702,9 +694,6 @@ int register_new_memory(int nid, struct mem_section *section)
>         int ret = 0;
>         struct memory_block *mem;
>
> -       if (is_zone_device_section(section))
> -               return 0;
> -
>         mutex_lock(&mem_sysfs_mutex);
>
>         mem = find_memory_block(section);
> @@ -741,11 +730,16 @@ static int remove_memory_section(unsigned long node_id,
>  {
>         struct memory_block *mem;
>
> -       if (is_zone_device_section(section))
> -               return 0;
> -
>         mutex_lock(&mem_sysfs_mutex);
> +
> +       /*
> +        * Some users of the memory hotplug do not want/need memblock to
> +        * track all sections. Skip over those.
> +        */
>         mem = find_memory_block(section);
> +       if (!mem)
> +               goto out_unlock;
> +
>         unregister_mem_sect_under_nodes(mem, __section_nr(section));
>
>         mem->section_count--;
> @@ -754,6 +748,7 @@ static int remove_memory_section(unsigned long node_id,
>         else
>                 put_device(&mem->dev);
>
> +out_unlock:
>         mutex_unlock(&mem_sysfs_mutex);
>         return 0;
>  }
> diff --git a/include/linux/memory_hotplug.h b/include/linux/memory_hotplug.h
> index 134a2f69c21a..3c8cf86201c3 100644
> --- a/include/linux/memory_hotplug.h
> +++ b/include/linux/memory_hotplug.h
> @@ -111,7 +111,7 @@ extern int __remove_pages(struct zone *zone, unsigned long start_pfn,
>
>  /* reasonably generic interface to expand the physical pages in a zone  */
>  extern int __add_pages(int nid, struct zone *zone, unsigned long start_pfn,
> -       unsigned long nr_pages);
> +       unsigned long nr_pages, bool want_memblock);
>
>  #ifdef CONFIG_NUMA
>  extern int memory_add_physaddr_to_nid(u64 start);
> diff --git a/mm/memory_hotplug.c b/mm/memory_hotplug.c
> index 6290d34b6331..a95120c56a9a 100644
> --- a/mm/memory_hotplug.c
> +++ b/mm/memory_hotplug.c
> @@ -494,7 +494,7 @@ static int __meminit __add_zone(struct zone *zone, unsigned long phys_start_pfn)
>  }
>
>  static int __meminit __add_section(int nid, struct zone *zone,
> -                                       unsigned long phys_start_pfn)
> +               unsigned long phys_start_pfn, bool want_memblock)
>  {
>         int ret;
>
> @@ -511,6 +511,9 @@ static int __meminit __add_section(int nid, struct zone *zone,
>         if (ret < 0)
>                 return ret;
>
> +       if (!want_memblock)
> +               return 0;
> +
>         return register_new_memory(nid, __pfn_to_section(phys_start_pfn));
>  }
>
> @@ -521,7 +524,7 @@ static int __meminit __add_section(int nid, struct zone *zone,
>   * add the new pages.
>   */
>  int __ref __add_pages(int nid, struct zone *zone, unsigned long phys_start_pfn,
> -                       unsigned long nr_pages)
> +                       unsigned long nr_pages, bool want_memblock)
>  {
>         unsigned long i;
>         int err = 0;
> @@ -549,7 +552,7 @@ int __ref __add_pages(int nid, struct zone *zone, unsigned long phys_start_pfn,
>         }
>
>         for (i = start_sec; i <= end_sec; i++) {
> -               err = __add_section(nid, zone, section_nr_to_pfn(i));
> +               err = __add_section(nid, zone, section_nr_to_pfn(i), want_memblock);
>
>                 /*
>                  * EEXIST is finally dealt with by ioresource collision
> --
> 2.11.0
>

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

^ permalink raw reply	[flat|nested] 74+ messages in thread

* Re: [PATCH 04/14] mm, memory_hotplug: get rid of is_zone_device_section
  2017-06-10  9:56     ` Wei Yang
@ 2017-06-10 14:58       ` Wei Yang
  -1 siblings, 0 replies; 74+ messages in thread
From: Wei Yang @ 2017-06-10 14:58 UTC (permalink / raw)
  To: Michal Hocko
  Cc: Andrew Morton, Linux-MM, Mel Gorman, Vlastimil Babka,
	Andrea Arcangeli, Jerome Glisse, Reza Arbab, Yasuaki Ishimatsu,
	qiuxishi, Kani Toshimitsu, slaoub, Joonsoo Kim, Andi Kleen,
	David Rientjes, Daniel Kiper, Igor Mammedov, Vitaly Kuznetsov,
	LKML, Michal Hocko, Dan Williams

On Sat, Jun 10, 2017 at 5:56 PM, Wei Yang <richard.weiyang@gmail.com> wrote:
> On Mon, May 15, 2017 at 4:58 PM, Michal Hocko <mhocko@kernel.org> wrote:
>> From: Michal Hocko <mhocko@suse.com>
>>
>> device memory hotplug hooks into regular memory hotplug only half way.
>> It needs memory sections to track struct pages but there is no
>> need/desire to associate those sections with memory blocks and export
>> them to the userspace via sysfs because they cannot be onlined anyway.
>>
>> This is currently expressed by for_device argument to arch_add_memory
>> which then makes sure to associate the given memory range with
>> ZONE_DEVICE. register_new_memory then relies on is_zone_device_section
>> to distinguish special memory hotplug from the regular one. While this
>> works now, later patches in this series want to move __add_zone outside
>> of arch_add_memory path so we have to come up with something else.
>>
>> Add want_memblock down the __add_pages path and use it to control
>> whether the section->memblock association should be done. arch_add_memory
>> then just trivially want memblock for everything but for_device hotplug.
>>
>> remove_memory_section doesn't need is_zone_device_section either. We can
>> simply skip all the memblock specific cleanup if there is no memblock
>> for the given section.
>>
>> This shouldn't introduce any functional change.
>>
>
> Hmm... one question about the memory_block behavior.
>
> In case one memory_block contains more than one memory section.
> If one section is "device zone", the whole memory_block is not visible
> in sysfs. Or until the whole memory_block is full, the sysfs is visible.
>

Ok, I made a mistake here. The memory_block device is visible in this
case, while the sysfs link between memory_block and node is not visible
for the whole memory_block device.

BTW, current register_mem_sect_under_node() will create the sysfs
link between memory_block and node for each pfn, while actually
we only need one link between them. If I am correct.

If you think it is fine, I would like to change this one to create the link
on section base.

> This is the known behavior right?
>
> And in this case, the memory_block could be found. Would this introduce
> some problem when remove_memory_section()?
>
>
>> Changes since v1
>> - return 0 if want_memblock == 0 from __add_section as per Jerome Glisse
>>
>> Changes since v2
>> - fix remove_memory_section unlock on find_memory_block failure
>>   as per Jerome - spotted by Evgeny Baskakov
>>
>> Tested-by: Dan Williams <dan.j.williams@intel.com>
>> Acked-by: Vlastimil Babka <vbabka@suse.cz>
>> Signed-off-by: Michal Hocko <mhocko@suse.com>
>> ---
>>  arch/ia64/mm/init.c            |  2 +-
>>  arch/powerpc/mm/mem.c          |  2 +-
>>  arch/s390/mm/init.c            |  2 +-
>>  arch/sh/mm/init.c              |  2 +-
>>  arch/x86/mm/init_32.c          |  2 +-
>>  arch/x86/mm/init_64.c          |  2 +-
>>  drivers/base/memory.c          | 23 +++++++++--------------
>>  include/linux/memory_hotplug.h |  2 +-
>>  mm/memory_hotplug.c            |  9 ++++++---
>>  9 files changed, 22 insertions(+), 24 deletions(-)
>>
>> diff --git a/arch/ia64/mm/init.c b/arch/ia64/mm/init.c
>> index 8f3efa682ee8..39e2aeb4669d 100644
>> --- a/arch/ia64/mm/init.c
>> +++ b/arch/ia64/mm/init.c
>> @@ -658,7 +658,7 @@ int arch_add_memory(int nid, u64 start, u64 size, bool for_device)
>>
>>         zone = pgdat->node_zones +
>>                 zone_for_memory(nid, start, size, ZONE_NORMAL, for_device);
>> -       ret = __add_pages(nid, zone, start_pfn, nr_pages);
>> +       ret = __add_pages(nid, zone, start_pfn, nr_pages, !for_device);
>>
>>         if (ret)
>>                 printk("%s: Problem encountered in __add_pages() as ret=%d\n",
>> diff --git a/arch/powerpc/mm/mem.c b/arch/powerpc/mm/mem.c
>> index 9ee536ec0739..e6b2e6618b6c 100644
>> --- a/arch/powerpc/mm/mem.c
>> +++ b/arch/powerpc/mm/mem.c
>> @@ -151,7 +151,7 @@ int arch_add_memory(int nid, u64 start, u64 size, bool for_device)
>>         zone = pgdata->node_zones +
>>                 zone_for_memory(nid, start, size, 0, for_device);
>>
>> -       return __add_pages(nid, zone, start_pfn, nr_pages);
>> +       return __add_pages(nid, zone, start_pfn, nr_pages, !for_device);
>>  }
>>
>>  #ifdef CONFIG_MEMORY_HOTREMOVE
>> diff --git a/arch/s390/mm/init.c b/arch/s390/mm/init.c
>> index ee6a1d3d4983..893cf88cf02d 100644
>> --- a/arch/s390/mm/init.c
>> +++ b/arch/s390/mm/init.c
>> @@ -191,7 +191,7 @@ int arch_add_memory(int nid, u64 start, u64 size, bool for_device)
>>                         continue;
>>                 nr_pages = (start_pfn + size_pages > zone_end_pfn) ?
>>                            zone_end_pfn - start_pfn : size_pages;
>> -               rc = __add_pages(nid, zone, start_pfn, nr_pages);
>> +               rc = __add_pages(nid, zone, start_pfn, nr_pages, !for_device);
>>                 if (rc)
>>                         break;
>>                 start_pfn += nr_pages;
>> diff --git a/arch/sh/mm/init.c b/arch/sh/mm/init.c
>> index 75491862d900..a9d57f75ae8c 100644
>> --- a/arch/sh/mm/init.c
>> +++ b/arch/sh/mm/init.c
>> @@ -498,7 +498,7 @@ int arch_add_memory(int nid, u64 start, u64 size, bool for_device)
>>         ret = __add_pages(nid, pgdat->node_zones +
>>                         zone_for_memory(nid, start, size, ZONE_NORMAL,
>>                         for_device),
>> -                       start_pfn, nr_pages);
>> +                       start_pfn, nr_pages, !for_device);
>>         if (unlikely(ret))
>>                 printk("%s: Failed, __add_pages() == %d\n", __func__, ret);
>>
>> diff --git a/arch/x86/mm/init_32.c b/arch/x86/mm/init_32.c
>> index 99fb83819a5f..94594b889144 100644
>> --- a/arch/x86/mm/init_32.c
>> +++ b/arch/x86/mm/init_32.c
>> @@ -831,7 +831,7 @@ int arch_add_memory(int nid, u64 start, u64 size, bool for_device)
>>         unsigned long start_pfn = start >> PAGE_SHIFT;
>>         unsigned long nr_pages = size >> PAGE_SHIFT;
>>
>> -       return __add_pages(nid, zone, start_pfn, nr_pages);
>> +       return __add_pages(nid, zone, start_pfn, nr_pages, !for_device);
>>  }
>>
>>  #ifdef CONFIG_MEMORY_HOTREMOVE
>> diff --git a/arch/x86/mm/init_64.c b/arch/x86/mm/init_64.c
>> index 41270b96403d..2e004364a373 100644
>> --- a/arch/x86/mm/init_64.c
>> +++ b/arch/x86/mm/init_64.c
>> @@ -697,7 +697,7 @@ int arch_add_memory(int nid, u64 start, u64 size, bool for_device)
>>
>>         init_memory_mapping(start, start + size);
>>
>> -       ret = __add_pages(nid, zone, start_pfn, nr_pages);
>> +       ret = __add_pages(nid, zone, start_pfn, nr_pages, !for_device);
>>         WARN_ON_ONCE(ret);
>>
>>         /* update max_pfn, max_low_pfn and high_memory */
>> diff --git a/drivers/base/memory.c b/drivers/base/memory.c
>> index 90225ffee501..f8fd562c3f18 100644
>> --- a/drivers/base/memory.c
>> +++ b/drivers/base/memory.c
>> @@ -685,14 +685,6 @@ static int add_memory_block(int base_section_nr)
>>         return 0;
>>  }
>>
>> -static bool is_zone_device_section(struct mem_section *ms)
>> -{
>> -       struct page *page;
>> -
>> -       page = sparse_decode_mem_map(ms->section_mem_map, __section_nr(ms));
>> -       return is_zone_device_page(page);
>> -}
>> -
>>  /*
>>   * need an interface for the VM to add new memory regions,
>>   * but without onlining it.
>> @@ -702,9 +694,6 @@ int register_new_memory(int nid, struct mem_section *section)
>>         int ret = 0;
>>         struct memory_block *mem;
>>
>> -       if (is_zone_device_section(section))
>> -               return 0;
>> -
>>         mutex_lock(&mem_sysfs_mutex);
>>
>>         mem = find_memory_block(section);
>> @@ -741,11 +730,16 @@ static int remove_memory_section(unsigned long node_id,
>>  {
>>         struct memory_block *mem;
>>
>> -       if (is_zone_device_section(section))
>> -               return 0;
>> -
>>         mutex_lock(&mem_sysfs_mutex);
>> +
>> +       /*
>> +        * Some users of the memory hotplug do not want/need memblock to
>> +        * track all sections. Skip over those.
>> +        */
>>         mem = find_memory_block(section);
>> +       if (!mem)
>> +               goto out_unlock;
>> +
>>         unregister_mem_sect_under_nodes(mem, __section_nr(section));
>>
>>         mem->section_count--;
>> @@ -754,6 +748,7 @@ static int remove_memory_section(unsigned long node_id,
>>         else
>>                 put_device(&mem->dev);
>>
>> +out_unlock:
>>         mutex_unlock(&mem_sysfs_mutex);
>>         return 0;
>>  }
>> diff --git a/include/linux/memory_hotplug.h b/include/linux/memory_hotplug.h
>> index 134a2f69c21a..3c8cf86201c3 100644
>> --- a/include/linux/memory_hotplug.h
>> +++ b/include/linux/memory_hotplug.h
>> @@ -111,7 +111,7 @@ extern int __remove_pages(struct zone *zone, unsigned long start_pfn,
>>
>>  /* reasonably generic interface to expand the physical pages in a zone  */
>>  extern int __add_pages(int nid, struct zone *zone, unsigned long start_pfn,
>> -       unsigned long nr_pages);
>> +       unsigned long nr_pages, bool want_memblock);
>>
>>  #ifdef CONFIG_NUMA
>>  extern int memory_add_physaddr_to_nid(u64 start);
>> diff --git a/mm/memory_hotplug.c b/mm/memory_hotplug.c
>> index 6290d34b6331..a95120c56a9a 100644
>> --- a/mm/memory_hotplug.c
>> +++ b/mm/memory_hotplug.c
>> @@ -494,7 +494,7 @@ static int __meminit __add_zone(struct zone *zone, unsigned long phys_start_pfn)
>>  }
>>
>>  static int __meminit __add_section(int nid, struct zone *zone,
>> -                                       unsigned long phys_start_pfn)
>> +               unsigned long phys_start_pfn, bool want_memblock)
>>  {
>>         int ret;
>>
>> @@ -511,6 +511,9 @@ static int __meminit __add_section(int nid, struct zone *zone,
>>         if (ret < 0)
>>                 return ret;
>>
>> +       if (!want_memblock)
>> +               return 0;
>> +
>>         return register_new_memory(nid, __pfn_to_section(phys_start_pfn));
>>  }
>>
>> @@ -521,7 +524,7 @@ static int __meminit __add_section(int nid, struct zone *zone,
>>   * add the new pages.
>>   */
>>  int __ref __add_pages(int nid, struct zone *zone, unsigned long phys_start_pfn,
>> -                       unsigned long nr_pages)
>> +                       unsigned long nr_pages, bool want_memblock)
>>  {
>>         unsigned long i;
>>         int err = 0;
>> @@ -549,7 +552,7 @@ int __ref __add_pages(int nid, struct zone *zone, unsigned long phys_start_pfn,
>>         }
>>
>>         for (i = start_sec; i <= end_sec; i++) {
>> -               err = __add_section(nid, zone, section_nr_to_pfn(i));
>> +               err = __add_section(nid, zone, section_nr_to_pfn(i), want_memblock);
>>
>>                 /*
>>                  * EEXIST is finally dealt with by ioresource collision
>> --
>> 2.11.0
>>

^ permalink raw reply	[flat|nested] 74+ messages in thread

* Re: [PATCH 04/14] mm, memory_hotplug: get rid of is_zone_device_section
@ 2017-06-10 14:58       ` Wei Yang
  0 siblings, 0 replies; 74+ messages in thread
From: Wei Yang @ 2017-06-10 14:58 UTC (permalink / raw)
  To: Michal Hocko
  Cc: Andrew Morton, Linux-MM, Mel Gorman, Vlastimil Babka,
	Andrea Arcangeli, Jerome Glisse, Reza Arbab, Yasuaki Ishimatsu,
	qiuxishi, Kani Toshimitsu, slaoub, Joonsoo Kim, Andi Kleen,
	David Rientjes, Daniel Kiper, Igor Mammedov, Vitaly Kuznetsov,
	LKML, Michal Hocko, Dan Williams

On Sat, Jun 10, 2017 at 5:56 PM, Wei Yang <richard.weiyang@gmail.com> wrote:
> On Mon, May 15, 2017 at 4:58 PM, Michal Hocko <mhocko@kernel.org> wrote:
>> From: Michal Hocko <mhocko@suse.com>
>>
>> device memory hotplug hooks into regular memory hotplug only half way.
>> It needs memory sections to track struct pages but there is no
>> need/desire to associate those sections with memory blocks and export
>> them to the userspace via sysfs because they cannot be onlined anyway.
>>
>> This is currently expressed by for_device argument to arch_add_memory
>> which then makes sure to associate the given memory range with
>> ZONE_DEVICE. register_new_memory then relies on is_zone_device_section
>> to distinguish special memory hotplug from the regular one. While this
>> works now, later patches in this series want to move __add_zone outside
>> of arch_add_memory path so we have to come up with something else.
>>
>> Add want_memblock down the __add_pages path and use it to control
>> whether the section->memblock association should be done. arch_add_memory
>> then just trivially want memblock for everything but for_device hotplug.
>>
>> remove_memory_section doesn't need is_zone_device_section either. We can
>> simply skip all the memblock specific cleanup if there is no memblock
>> for the given section.
>>
>> This shouldn't introduce any functional change.
>>
>
> Hmm... one question about the memory_block behavior.
>
> In case one memory_block contains more than one memory section.
> If one section is "device zone", the whole memory_block is not visible
> in sysfs. Or until the whole memory_block is full, the sysfs is visible.
>

Ok, I made a mistake here. The memory_block device is visible in this
case, while the sysfs link between memory_block and node is not visible
for the whole memory_block device.

BTW, current register_mem_sect_under_node() will create the sysfs
link between memory_block and node for each pfn, while actually
we only need one link between them. If I am correct.

If you think it is fine, I would like to change this one to create the link
on section base.

> This is the known behavior right?
>
> And in this case, the memory_block could be found. Would this introduce
> some problem when remove_memory_section()?
>
>
>> Changes since v1
>> - return 0 if want_memblock == 0 from __add_section as per Jerome Glisse
>>
>> Changes since v2
>> - fix remove_memory_section unlock on find_memory_block failure
>>   as per Jerome - spotted by Evgeny Baskakov
>>
>> Tested-by: Dan Williams <dan.j.williams@intel.com>
>> Acked-by: Vlastimil Babka <vbabka@suse.cz>
>> Signed-off-by: Michal Hocko <mhocko@suse.com>
>> ---
>>  arch/ia64/mm/init.c            |  2 +-
>>  arch/powerpc/mm/mem.c          |  2 +-
>>  arch/s390/mm/init.c            |  2 +-
>>  arch/sh/mm/init.c              |  2 +-
>>  arch/x86/mm/init_32.c          |  2 +-
>>  arch/x86/mm/init_64.c          |  2 +-
>>  drivers/base/memory.c          | 23 +++++++++--------------
>>  include/linux/memory_hotplug.h |  2 +-
>>  mm/memory_hotplug.c            |  9 ++++++---
>>  9 files changed, 22 insertions(+), 24 deletions(-)
>>
>> diff --git a/arch/ia64/mm/init.c b/arch/ia64/mm/init.c
>> index 8f3efa682ee8..39e2aeb4669d 100644
>> --- a/arch/ia64/mm/init.c
>> +++ b/arch/ia64/mm/init.c
>> @@ -658,7 +658,7 @@ int arch_add_memory(int nid, u64 start, u64 size, bool for_device)
>>
>>         zone = pgdat->node_zones +
>>                 zone_for_memory(nid, start, size, ZONE_NORMAL, for_device);
>> -       ret = __add_pages(nid, zone, start_pfn, nr_pages);
>> +       ret = __add_pages(nid, zone, start_pfn, nr_pages, !for_device);
>>
>>         if (ret)
>>                 printk("%s: Problem encountered in __add_pages() as ret=%d\n",
>> diff --git a/arch/powerpc/mm/mem.c b/arch/powerpc/mm/mem.c
>> index 9ee536ec0739..e6b2e6618b6c 100644
>> --- a/arch/powerpc/mm/mem.c
>> +++ b/arch/powerpc/mm/mem.c
>> @@ -151,7 +151,7 @@ int arch_add_memory(int nid, u64 start, u64 size, bool for_device)
>>         zone = pgdata->node_zones +
>>                 zone_for_memory(nid, start, size, 0, for_device);
>>
>> -       return __add_pages(nid, zone, start_pfn, nr_pages);
>> +       return __add_pages(nid, zone, start_pfn, nr_pages, !for_device);
>>  }
>>
>>  #ifdef CONFIG_MEMORY_HOTREMOVE
>> diff --git a/arch/s390/mm/init.c b/arch/s390/mm/init.c
>> index ee6a1d3d4983..893cf88cf02d 100644
>> --- a/arch/s390/mm/init.c
>> +++ b/arch/s390/mm/init.c
>> @@ -191,7 +191,7 @@ int arch_add_memory(int nid, u64 start, u64 size, bool for_device)
>>                         continue;
>>                 nr_pages = (start_pfn + size_pages > zone_end_pfn) ?
>>                            zone_end_pfn - start_pfn : size_pages;
>> -               rc = __add_pages(nid, zone, start_pfn, nr_pages);
>> +               rc = __add_pages(nid, zone, start_pfn, nr_pages, !for_device);
>>                 if (rc)
>>                         break;
>>                 start_pfn += nr_pages;
>> diff --git a/arch/sh/mm/init.c b/arch/sh/mm/init.c
>> index 75491862d900..a9d57f75ae8c 100644
>> --- a/arch/sh/mm/init.c
>> +++ b/arch/sh/mm/init.c
>> @@ -498,7 +498,7 @@ int arch_add_memory(int nid, u64 start, u64 size, bool for_device)
>>         ret = __add_pages(nid, pgdat->node_zones +
>>                         zone_for_memory(nid, start, size, ZONE_NORMAL,
>>                         for_device),
>> -                       start_pfn, nr_pages);
>> +                       start_pfn, nr_pages, !for_device);
>>         if (unlikely(ret))
>>                 printk("%s: Failed, __add_pages() == %d\n", __func__, ret);
>>
>> diff --git a/arch/x86/mm/init_32.c b/arch/x86/mm/init_32.c
>> index 99fb83819a5f..94594b889144 100644
>> --- a/arch/x86/mm/init_32.c
>> +++ b/arch/x86/mm/init_32.c
>> @@ -831,7 +831,7 @@ int arch_add_memory(int nid, u64 start, u64 size, bool for_device)
>>         unsigned long start_pfn = start >> PAGE_SHIFT;
>>         unsigned long nr_pages = size >> PAGE_SHIFT;
>>
>> -       return __add_pages(nid, zone, start_pfn, nr_pages);
>> +       return __add_pages(nid, zone, start_pfn, nr_pages, !for_device);
>>  }
>>
>>  #ifdef CONFIG_MEMORY_HOTREMOVE
>> diff --git a/arch/x86/mm/init_64.c b/arch/x86/mm/init_64.c
>> index 41270b96403d..2e004364a373 100644
>> --- a/arch/x86/mm/init_64.c
>> +++ b/arch/x86/mm/init_64.c
>> @@ -697,7 +697,7 @@ int arch_add_memory(int nid, u64 start, u64 size, bool for_device)
>>
>>         init_memory_mapping(start, start + size);
>>
>> -       ret = __add_pages(nid, zone, start_pfn, nr_pages);
>> +       ret = __add_pages(nid, zone, start_pfn, nr_pages, !for_device);
>>         WARN_ON_ONCE(ret);
>>
>>         /* update max_pfn, max_low_pfn and high_memory */
>> diff --git a/drivers/base/memory.c b/drivers/base/memory.c
>> index 90225ffee501..f8fd562c3f18 100644
>> --- a/drivers/base/memory.c
>> +++ b/drivers/base/memory.c
>> @@ -685,14 +685,6 @@ static int add_memory_block(int base_section_nr)
>>         return 0;
>>  }
>>
>> -static bool is_zone_device_section(struct mem_section *ms)
>> -{
>> -       struct page *page;
>> -
>> -       page = sparse_decode_mem_map(ms->section_mem_map, __section_nr(ms));
>> -       return is_zone_device_page(page);
>> -}
>> -
>>  /*
>>   * need an interface for the VM to add new memory regions,
>>   * but without onlining it.
>> @@ -702,9 +694,6 @@ int register_new_memory(int nid, struct mem_section *section)
>>         int ret = 0;
>>         struct memory_block *mem;
>>
>> -       if (is_zone_device_section(section))
>> -               return 0;
>> -
>>         mutex_lock(&mem_sysfs_mutex);
>>
>>         mem = find_memory_block(section);
>> @@ -741,11 +730,16 @@ static int remove_memory_section(unsigned long node_id,
>>  {
>>         struct memory_block *mem;
>>
>> -       if (is_zone_device_section(section))
>> -               return 0;
>> -
>>         mutex_lock(&mem_sysfs_mutex);
>> +
>> +       /*
>> +        * Some users of the memory hotplug do not want/need memblock to
>> +        * track all sections. Skip over those.
>> +        */
>>         mem = find_memory_block(section);
>> +       if (!mem)
>> +               goto out_unlock;
>> +
>>         unregister_mem_sect_under_nodes(mem, __section_nr(section));
>>
>>         mem->section_count--;
>> @@ -754,6 +748,7 @@ static int remove_memory_section(unsigned long node_id,
>>         else
>>                 put_device(&mem->dev);
>>
>> +out_unlock:
>>         mutex_unlock(&mem_sysfs_mutex);
>>         return 0;
>>  }
>> diff --git a/include/linux/memory_hotplug.h b/include/linux/memory_hotplug.h
>> index 134a2f69c21a..3c8cf86201c3 100644
>> --- a/include/linux/memory_hotplug.h
>> +++ b/include/linux/memory_hotplug.h
>> @@ -111,7 +111,7 @@ extern int __remove_pages(struct zone *zone, unsigned long start_pfn,
>>
>>  /* reasonably generic interface to expand the physical pages in a zone  */
>>  extern int __add_pages(int nid, struct zone *zone, unsigned long start_pfn,
>> -       unsigned long nr_pages);
>> +       unsigned long nr_pages, bool want_memblock);
>>
>>  #ifdef CONFIG_NUMA
>>  extern int memory_add_physaddr_to_nid(u64 start);
>> diff --git a/mm/memory_hotplug.c b/mm/memory_hotplug.c
>> index 6290d34b6331..a95120c56a9a 100644
>> --- a/mm/memory_hotplug.c
>> +++ b/mm/memory_hotplug.c
>> @@ -494,7 +494,7 @@ static int __meminit __add_zone(struct zone *zone, unsigned long phys_start_pfn)
>>  }
>>
>>  static int __meminit __add_section(int nid, struct zone *zone,
>> -                                       unsigned long phys_start_pfn)
>> +               unsigned long phys_start_pfn, bool want_memblock)
>>  {
>>         int ret;
>>
>> @@ -511,6 +511,9 @@ static int __meminit __add_section(int nid, struct zone *zone,
>>         if (ret < 0)
>>                 return ret;
>>
>> +       if (!want_memblock)
>> +               return 0;
>> +
>>         return register_new_memory(nid, __pfn_to_section(phys_start_pfn));
>>  }
>>
>> @@ -521,7 +524,7 @@ static int __meminit __add_section(int nid, struct zone *zone,
>>   * add the new pages.
>>   */
>>  int __ref __add_pages(int nid, struct zone *zone, unsigned long phys_start_pfn,
>> -                       unsigned long nr_pages)
>> +                       unsigned long nr_pages, bool want_memblock)
>>  {
>>         unsigned long i;
>>         int err = 0;
>> @@ -549,7 +552,7 @@ int __ref __add_pages(int nid, struct zone *zone, unsigned long phys_start_pfn,
>>         }
>>
>>         for (i = start_sec; i <= end_sec; i++) {
>> -               err = __add_section(nid, zone, section_nr_to_pfn(i));
>> +               err = __add_section(nid, zone, section_nr_to_pfn(i), want_memblock);
>>
>>                 /*
>>                  * EEXIST is finally dealt with by ioresource collision
>> --
>> 2.11.0
>>

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

^ permalink raw reply	[flat|nested] 74+ messages in thread

* Re: [PATCH 04/14] mm, memory_hotplug: get rid of is_zone_device_section
  2017-06-10 14:58       ` Wei Yang
@ 2017-06-12  6:49         ` Michal Hocko
  -1 siblings, 0 replies; 74+ messages in thread
From: Michal Hocko @ 2017-06-12  6:49 UTC (permalink / raw)
  To: Wei Yang
  Cc: Andrew Morton, Linux-MM, Mel Gorman, Vlastimil Babka,
	Andrea Arcangeli, Jerome Glisse, Reza Arbab, Yasuaki Ishimatsu,
	qiuxishi, Kani Toshimitsu, slaoub, Joonsoo Kim, Andi Kleen,
	David Rientjes, Daniel Kiper, Igor Mammedov, Vitaly Kuznetsov,
	LKML, Dan Williams

On Sat 10-06-17 22:58:21, Wei Yang wrote:
> On Sat, Jun 10, 2017 at 5:56 PM, Wei Yang <richard.weiyang@gmail.com> wrote:
[...]
> > Hmm... one question about the memory_block behavior.
> >
> > In case one memory_block contains more than one memory section.
> > If one section is "device zone", the whole memory_block is not visible
> > in sysfs. Or until the whole memory_block is full, the sysfs is visible.
> >
> 
> Ok, I made a mistake here. The memory_block device is visible in this
> case, while the sysfs link between memory_block and node is not visible
> for the whole memory_block device.

yes the behavior is quite messy

> 
> BTW, current register_mem_sect_under_node() will create the sysfs
> link between memory_block and node for each pfn, while actually
> we only need one link between them. If I am correct.
> 
> If you think it is fine, I would like to change this one to create the link
> on section base.

My longer term plan was to unify all the code to be either memory block
or memory section oriented. The first sounds more logical from the user
visible granularity point of view but there might be some corner cases
which would require to use section based approach. I didn't have time to
study that. If you want to play with that, feel free of course.
-- 
Michal Hocko
SUSE Labs

^ permalink raw reply	[flat|nested] 74+ messages in thread

* Re: [PATCH 04/14] mm, memory_hotplug: get rid of is_zone_device_section
@ 2017-06-12  6:49         ` Michal Hocko
  0 siblings, 0 replies; 74+ messages in thread
From: Michal Hocko @ 2017-06-12  6:49 UTC (permalink / raw)
  To: Wei Yang
  Cc: Andrew Morton, Linux-MM, Mel Gorman, Vlastimil Babka,
	Andrea Arcangeli, Jerome Glisse, Reza Arbab, Yasuaki Ishimatsu,
	qiuxishi, Kani Toshimitsu, slaoub, Joonsoo Kim, Andi Kleen,
	David Rientjes, Daniel Kiper, Igor Mammedov, Vitaly Kuznetsov,
	LKML, Dan Williams

On Sat 10-06-17 22:58:21, Wei Yang wrote:
> On Sat, Jun 10, 2017 at 5:56 PM, Wei Yang <richard.weiyang@gmail.com> wrote:
[...]
> > Hmm... one question about the memory_block behavior.
> >
> > In case one memory_block contains more than one memory section.
> > If one section is "device zone", the whole memory_block is not visible
> > in sysfs. Or until the whole memory_block is full, the sysfs is visible.
> >
> 
> Ok, I made a mistake here. The memory_block device is visible in this
> case, while the sysfs link between memory_block and node is not visible
> for the whole memory_block device.

yes the behavior is quite messy

> 
> BTW, current register_mem_sect_under_node() will create the sysfs
> link between memory_block and node for each pfn, while actually
> we only need one link between them. If I am correct.
> 
> If you think it is fine, I would like to change this one to create the link
> on section base.

My longer term plan was to unify all the code to be either memory block
or memory section oriented. The first sounds more logical from the user
visible granularity point of view but there might be some corner cases
which would require to use section based approach. I didn't have time to
study that. If you want to play with that, feel free of course.
-- 
Michal Hocko
SUSE Labs

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

^ permalink raw reply	[flat|nested] 74+ messages in thread

* Re: [PATCH 04/14] mm, memory_hotplug: get rid of is_zone_device_section
  2017-06-10  9:56     ` Wei Yang
  (?)
  (?)
@ 2017-06-14  6:12     ` Wei Yang
  2017-06-14  6:32         ` Michal Hocko
  -1 siblings, 1 reply; 74+ messages in thread
From: Wei Yang @ 2017-06-14  6:12 UTC (permalink / raw)
  To: Wei Yang
  Cc: Michal Hocko, Andrew Morton, Linux-MM, Mel Gorman,
	Vlastimil Babka, Andrea Arcangeli, Jerome Glisse, Reza Arbab,
	Yasuaki Ishimatsu, qiuxishi, Kani Toshimitsu, slaoub,
	Joonsoo Kim, Andi Kleen, David Rientjes, Daniel Kiper,
	Igor Mammedov, Vitaly Kuznetsov, LKML, Michal Hocko,
	Dan Williams

[-- Attachment #1: Type: text/plain, Size: 10800 bytes --]

On Sat, Jun 10, 2017 at 05:56:02PM +0800, Wei Yang wrote:
>On Mon, May 15, 2017 at 4:58 PM, Michal Hocko <mhocko@kernel.org> wrote:
>> From: Michal Hocko <mhocko@suse.com>
>>
>> device memory hotplug hooks into regular memory hotplug only half way.
>> It needs memory sections to track struct pages but there is no
>> need/desire to associate those sections with memory blocks and export
>> them to the userspace via sysfs because they cannot be onlined anyway.
>>
>> This is currently expressed by for_device argument to arch_add_memory
>> which then makes sure to associate the given memory range with
>> ZONE_DEVICE. register_new_memory then relies on is_zone_device_section
>> to distinguish special memory hotplug from the regular one. While this
>> works now, later patches in this series want to move __add_zone outside
>> of arch_add_memory path so we have to come up with something else.
>>
>> Add want_memblock down the __add_pages path and use it to control
>> whether the section->memblock association should be done. arch_add_memory
>> then just trivially want memblock for everything but for_device hotplug.
>>
>> remove_memory_section doesn't need is_zone_device_section either. We can
>> simply skip all the memblock specific cleanup if there is no memblock
>> for the given section.
>>
>> This shouldn't introduce any functional change.
>>
>
>Hmm... one question about the memory_block behavior.
>
>In case one memory_block contains more than one memory section.
>If one section is "device zone", the whole memory_block is not visible
>in sysfs. Or until the whole memory_block is full, the sysfs is visible.
>
>This is the known behavior right?
>
>And in this case, the memory_block could be found. Would this introduce
>some problem when remove_memory_section()?
>

Hi, Michal

Not sure you missed this one or you think this is fine.

Hmm... this will not happen since we must offline a whole memory_block?
So the memory_hotplug/unplug unit is memory_block instead of mem_section?

>
>> Changes since v1
>> - return 0 if want_memblock == 0 from __add_section as per Jerome Glisse
>>
>> Changes since v2
>> - fix remove_memory_section unlock on find_memory_block failure
>>   as per Jerome - spotted by Evgeny Baskakov
>>
>> Tested-by: Dan Williams <dan.j.williams@intel.com>
>> Acked-by: Vlastimil Babka <vbabka@suse.cz>
>> Signed-off-by: Michal Hocko <mhocko@suse.com>
>> ---
>>  arch/ia64/mm/init.c            |  2 +-
>>  arch/powerpc/mm/mem.c          |  2 +-
>>  arch/s390/mm/init.c            |  2 +-
>>  arch/sh/mm/init.c              |  2 +-
>>  arch/x86/mm/init_32.c          |  2 +-
>>  arch/x86/mm/init_64.c          |  2 +-
>>  drivers/base/memory.c          | 23 +++++++++--------------
>>  include/linux/memory_hotplug.h |  2 +-
>>  mm/memory_hotplug.c            |  9 ++++++---
>>  9 files changed, 22 insertions(+), 24 deletions(-)
>>
>> diff --git a/arch/ia64/mm/init.c b/arch/ia64/mm/init.c
>> index 8f3efa682ee8..39e2aeb4669d 100644
>> --- a/arch/ia64/mm/init.c
>> +++ b/arch/ia64/mm/init.c
>> @@ -658,7 +658,7 @@ int arch_add_memory(int nid, u64 start, u64 size, bool for_device)
>>
>>         zone = pgdat->node_zones +
>>                 zone_for_memory(nid, start, size, ZONE_NORMAL, for_device);
>> -       ret = __add_pages(nid, zone, start_pfn, nr_pages);
>> +       ret = __add_pages(nid, zone, start_pfn, nr_pages, !for_device);
>>
>>         if (ret)
>>                 printk("%s: Problem encountered in __add_pages() as ret=%d\n",
>> diff --git a/arch/powerpc/mm/mem.c b/arch/powerpc/mm/mem.c
>> index 9ee536ec0739..e6b2e6618b6c 100644
>> --- a/arch/powerpc/mm/mem.c
>> +++ b/arch/powerpc/mm/mem.c
>> @@ -151,7 +151,7 @@ int arch_add_memory(int nid, u64 start, u64 size, bool for_device)
>>         zone = pgdata->node_zones +
>>                 zone_for_memory(nid, start, size, 0, for_device);
>>
>> -       return __add_pages(nid, zone, start_pfn, nr_pages);
>> +       return __add_pages(nid, zone, start_pfn, nr_pages, !for_device);
>>  }
>>
>>  #ifdef CONFIG_MEMORY_HOTREMOVE
>> diff --git a/arch/s390/mm/init.c b/arch/s390/mm/init.c
>> index ee6a1d3d4983..893cf88cf02d 100644
>> --- a/arch/s390/mm/init.c
>> +++ b/arch/s390/mm/init.c
>> @@ -191,7 +191,7 @@ int arch_add_memory(int nid, u64 start, u64 size, bool for_device)
>>                         continue;
>>                 nr_pages = (start_pfn + size_pages > zone_end_pfn) ?
>>                            zone_end_pfn - start_pfn : size_pages;
>> -               rc = __add_pages(nid, zone, start_pfn, nr_pages);
>> +               rc = __add_pages(nid, zone, start_pfn, nr_pages, !for_device);
>>                 if (rc)
>>                         break;
>>                 start_pfn += nr_pages;
>> diff --git a/arch/sh/mm/init.c b/arch/sh/mm/init.c
>> index 75491862d900..a9d57f75ae8c 100644
>> --- a/arch/sh/mm/init.c
>> +++ b/arch/sh/mm/init.c
>> @@ -498,7 +498,7 @@ int arch_add_memory(int nid, u64 start, u64 size, bool for_device)
>>         ret = __add_pages(nid, pgdat->node_zones +
>>                         zone_for_memory(nid, start, size, ZONE_NORMAL,
>>                         for_device),
>> -                       start_pfn, nr_pages);
>> +                       start_pfn, nr_pages, !for_device);
>>         if (unlikely(ret))
>>                 printk("%s: Failed, __add_pages() == %d\n", __func__, ret);
>>
>> diff --git a/arch/x86/mm/init_32.c b/arch/x86/mm/init_32.c
>> index 99fb83819a5f..94594b889144 100644
>> --- a/arch/x86/mm/init_32.c
>> +++ b/arch/x86/mm/init_32.c
>> @@ -831,7 +831,7 @@ int arch_add_memory(int nid, u64 start, u64 size, bool for_device)
>>         unsigned long start_pfn = start >> PAGE_SHIFT;
>>         unsigned long nr_pages = size >> PAGE_SHIFT;
>>
>> -       return __add_pages(nid, zone, start_pfn, nr_pages);
>> +       return __add_pages(nid, zone, start_pfn, nr_pages, !for_device);
>>  }
>>
>>  #ifdef CONFIG_MEMORY_HOTREMOVE
>> diff --git a/arch/x86/mm/init_64.c b/arch/x86/mm/init_64.c
>> index 41270b96403d..2e004364a373 100644
>> --- a/arch/x86/mm/init_64.c
>> +++ b/arch/x86/mm/init_64.c
>> @@ -697,7 +697,7 @@ int arch_add_memory(int nid, u64 start, u64 size, bool for_device)
>>
>>         init_memory_mapping(start, start + size);
>>
>> -       ret = __add_pages(nid, zone, start_pfn, nr_pages);
>> +       ret = __add_pages(nid, zone, start_pfn, nr_pages, !for_device);
>>         WARN_ON_ONCE(ret);
>>
>>         /* update max_pfn, max_low_pfn and high_memory */
>> diff --git a/drivers/base/memory.c b/drivers/base/memory.c
>> index 90225ffee501..f8fd562c3f18 100644
>> --- a/drivers/base/memory.c
>> +++ b/drivers/base/memory.c
>> @@ -685,14 +685,6 @@ static int add_memory_block(int base_section_nr)
>>         return 0;
>>  }
>>
>> -static bool is_zone_device_section(struct mem_section *ms)
>> -{
>> -       struct page *page;
>> -
>> -       page = sparse_decode_mem_map(ms->section_mem_map, __section_nr(ms));
>> -       return is_zone_device_page(page);
>> -}
>> -
>>  /*
>>   * need an interface for the VM to add new memory regions,
>>   * but without onlining it.
>> @@ -702,9 +694,6 @@ int register_new_memory(int nid, struct mem_section *section)
>>         int ret = 0;
>>         struct memory_block *mem;
>>
>> -       if (is_zone_device_section(section))
>> -               return 0;
>> -
>>         mutex_lock(&mem_sysfs_mutex);
>>
>>         mem = find_memory_block(section);
>> @@ -741,11 +730,16 @@ static int remove_memory_section(unsigned long node_id,
>>  {
>>         struct memory_block *mem;
>>
>> -       if (is_zone_device_section(section))
>> -               return 0;
>> -
>>         mutex_lock(&mem_sysfs_mutex);
>> +
>> +       /*
>> +        * Some users of the memory hotplug do not want/need memblock to
>> +        * track all sections. Skip over those.
>> +        */
>>         mem = find_memory_block(section);
>> +       if (!mem)
>> +               goto out_unlock;
>> +
>>         unregister_mem_sect_under_nodes(mem, __section_nr(section));
>>
>>         mem->section_count--;
>> @@ -754,6 +748,7 @@ static int remove_memory_section(unsigned long node_id,
>>         else
>>                 put_device(&mem->dev);
>>
>> +out_unlock:
>>         mutex_unlock(&mem_sysfs_mutex);
>>         return 0;
>>  }
>> diff --git a/include/linux/memory_hotplug.h b/include/linux/memory_hotplug.h
>> index 134a2f69c21a..3c8cf86201c3 100644
>> --- a/include/linux/memory_hotplug.h
>> +++ b/include/linux/memory_hotplug.h
>> @@ -111,7 +111,7 @@ extern int __remove_pages(struct zone *zone, unsigned long start_pfn,
>>
>>  /* reasonably generic interface to expand the physical pages in a zone  */
>>  extern int __add_pages(int nid, struct zone *zone, unsigned long start_pfn,
>> -       unsigned long nr_pages);
>> +       unsigned long nr_pages, bool want_memblock);
>>
>>  #ifdef CONFIG_NUMA
>>  extern int memory_add_physaddr_to_nid(u64 start);
>> diff --git a/mm/memory_hotplug.c b/mm/memory_hotplug.c
>> index 6290d34b6331..a95120c56a9a 100644
>> --- a/mm/memory_hotplug.c
>> +++ b/mm/memory_hotplug.c
>> @@ -494,7 +494,7 @@ static int __meminit __add_zone(struct zone *zone, unsigned long phys_start_pfn)
>>  }
>>
>>  static int __meminit __add_section(int nid, struct zone *zone,
>> -                                       unsigned long phys_start_pfn)
>> +               unsigned long phys_start_pfn, bool want_memblock)
>>  {
>>         int ret;
>>
>> @@ -511,6 +511,9 @@ static int __meminit __add_section(int nid, struct zone *zone,
>>         if (ret < 0)
>>                 return ret;
>>
>> +       if (!want_memblock)
>> +               return 0;
>> +
>>         return register_new_memory(nid, __pfn_to_section(phys_start_pfn));
>>  }
>>
>> @@ -521,7 +524,7 @@ static int __meminit __add_section(int nid, struct zone *zone,
>>   * add the new pages.
>>   */
>>  int __ref __add_pages(int nid, struct zone *zone, unsigned long phys_start_pfn,
>> -                       unsigned long nr_pages)
>> +                       unsigned long nr_pages, bool want_memblock)
>>  {
>>         unsigned long i;
>>         int err = 0;
>> @@ -549,7 +552,7 @@ int __ref __add_pages(int nid, struct zone *zone, unsigned long phys_start_pfn,
>>         }
>>
>>         for (i = start_sec; i <= end_sec; i++) {
>> -               err = __add_section(nid, zone, section_nr_to_pfn(i));
>> +               err = __add_section(nid, zone, section_nr_to_pfn(i), want_memblock);
>>
>>                 /*
>>                  * EEXIST is finally dealt with by ioresource collision
>> --
>> 2.11.0
>>

-- 
Wei Yang
Help you, Help me

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

^ permalink raw reply	[flat|nested] 74+ messages in thread

* Re: [PATCH 04/14] mm, memory_hotplug: get rid of is_zone_device_section
  2017-06-12  6:49         ` Michal Hocko
  (?)
@ 2017-06-14  6:17         ` Wei Yang
  -1 siblings, 0 replies; 74+ messages in thread
From: Wei Yang @ 2017-06-14  6:17 UTC (permalink / raw)
  To: Michal Hocko
  Cc: Wei Yang, Andrew Morton, Linux-MM, Mel Gorman, Vlastimil Babka,
	Andrea Arcangeli, Jerome Glisse, Reza Arbab, Yasuaki Ishimatsu,
	qiuxishi, Kani Toshimitsu, slaoub, Joonsoo Kim, Andi Kleen,
	David Rientjes, Daniel Kiper, Igor Mammedov, Vitaly Kuznetsov,
	LKML, Dan Williams

[-- Attachment #1: Type: text/plain, Size: 1751 bytes --]

On Mon, Jun 12, 2017 at 08:49:53AM +0200, Michal Hocko wrote:
>On Sat 10-06-17 22:58:21, Wei Yang wrote:
>> On Sat, Jun 10, 2017 at 5:56 PM, Wei Yang <richard.weiyang@gmail.com> wrote:
>[...]
>> > Hmm... one question about the memory_block behavior.
>> >
>> > In case one memory_block contains more than one memory section.
>> > If one section is "device zone", the whole memory_block is not visible
>> > in sysfs. Or until the whole memory_block is full, the sysfs is visible.
>> >
>> 
>> Ok, I made a mistake here. The memory_block device is visible in this
>> case, while the sysfs link between memory_block and node is not visible
>> for the whole memory_block device.
>
>yes the behavior is quite messy
>
>> 
>> BTW, current register_mem_sect_under_node() will create the sysfs
>> link between memory_block and node for each pfn, while actually
>> we only need one link between them. If I am correct.
>> 
>> If you think it is fine, I would like to change this one to create the link
>> on section base.
>
>My longer term plan was to unify all the code to be either memory block
>or memory section oriented. The first sounds more logical from the user
>visible granularity point of view but there might be some corner cases

This means the granularity of hotplug is memory_block instead of mem_section?

While I see the alignment check of add_memory_resource() is SECTION size.

>which would require to use section based approach. I didn't have time to
>study that. If you want to play with that, feel free of course.

Yep, I am really want to help, while these inter-connected concepts makes me
confused. I need to learn more on these.

>-- 
>Michal Hocko
>SUSE Labs

-- 
Wei Yang
Help you, Help me

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

^ permalink raw reply	[flat|nested] 74+ messages in thread

* Re: [PATCH 04/14] mm, memory_hotplug: get rid of is_zone_device_section
  2017-06-14  6:12     ` Wei Yang
@ 2017-06-14  6:32         ` Michal Hocko
  0 siblings, 0 replies; 74+ messages in thread
From: Michal Hocko @ 2017-06-14  6:32 UTC (permalink / raw)
  To: Wei Yang
  Cc: Andrew Morton, Linux-MM, Mel Gorman, Vlastimil Babka,
	Andrea Arcangeli, Jerome Glisse, Reza Arbab, Yasuaki Ishimatsu,
	qiuxishi, Kani Toshimitsu, slaoub, Joonsoo Kim, Andi Kleen,
	David Rientjes, Daniel Kiper, Igor Mammedov, Vitaly Kuznetsov,
	LKML, Dan Williams

On Wed 14-06-17 14:12:59, Wei Yang wrote:
[...]
> Hi, Michal
> 
> Not sure you missed this one or you think this is fine.
> 
> Hmm... this will not happen since we must offline a whole memory_block?

yes

> So the memory_hotplug/unplug unit is memory_block instead of mem_section?

yes.
-- 
Michal Hocko
SUSE Labs

^ permalink raw reply	[flat|nested] 74+ messages in thread

* Re: [PATCH 04/14] mm, memory_hotplug: get rid of is_zone_device_section
@ 2017-06-14  6:32         ` Michal Hocko
  0 siblings, 0 replies; 74+ messages in thread
From: Michal Hocko @ 2017-06-14  6:32 UTC (permalink / raw)
  To: Wei Yang
  Cc: Andrew Morton, Linux-MM, Mel Gorman, Vlastimil Babka,
	Andrea Arcangeli, Jerome Glisse, Reza Arbab, Yasuaki Ishimatsu,
	qiuxishi, Kani Toshimitsu, slaoub, Joonsoo Kim, Andi Kleen,
	David Rientjes, Daniel Kiper, Igor Mammedov, Vitaly Kuznetsov,
	LKML, Dan Williams

On Wed 14-06-17 14:12:59, Wei Yang wrote:
[...]
> Hi, Michal
> 
> Not sure you missed this one or you think this is fine.
> 
> Hmm... this will not happen since we must offline a whole memory_block?

yes

> So the memory_hotplug/unplug unit is memory_block instead of mem_section?

yes.
-- 
Michal Hocko
SUSE Labs

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

^ permalink raw reply	[flat|nested] 74+ messages in thread

* Re: [PATCH 04/14] mm, memory_hotplug: get rid of is_zone_device_section
  2017-06-14  6:32         ` Michal Hocko
  (?)
@ 2017-06-14  9:12         ` Wei Yang
  2017-06-14  9:24             ` Michal Hocko
  -1 siblings, 1 reply; 74+ messages in thread
From: Wei Yang @ 2017-06-14  9:12 UTC (permalink / raw)
  To: Michal Hocko
  Cc: Wei Yang, Andrew Morton, Linux-MM, Mel Gorman, Vlastimil Babka,
	Andrea Arcangeli, Jerome Glisse, Reza Arbab, Yasuaki Ishimatsu,
	qiuxishi, Kani Toshimitsu, slaoub, Joonsoo Kim, Andi Kleen,
	David Rientjes, Daniel Kiper, Igor Mammedov, Vitaly Kuznetsov,
	LKML, Dan Williams

[-- Attachment #1: Type: text/plain, Size: 618 bytes --]

On Wed, Jun 14, 2017 at 08:32:06AM +0200, Michal Hocko wrote:
>On Wed 14-06-17 14:12:59, Wei Yang wrote:
>[...]
>> Hi, Michal
>> 
>> Not sure you missed this one or you think this is fine.
>> 
>> Hmm... this will not happen since we must offline a whole memory_block?
>
>yes
>
>> So the memory_hotplug/unplug unit is memory_block instead of mem_section?
>
>yes.

If this is true, the check_hotplug_memory_range() should be fixed too.

This function just makes sure the range is section aligned, instead of
memory_block aligned.

>-- 
>Michal Hocko
>SUSE Labs

-- 
Wei Yang
Help you, Help me

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

^ permalink raw reply	[flat|nested] 74+ messages in thread

* Re: [PATCH 04/14] mm, memory_hotplug: get rid of is_zone_device_section
  2017-06-14  9:12         ` Wei Yang
@ 2017-06-14  9:24             ` Michal Hocko
  0 siblings, 0 replies; 74+ messages in thread
From: Michal Hocko @ 2017-06-14  9:24 UTC (permalink / raw)
  To: Wei Yang
  Cc: Andrew Morton, Linux-MM, Mel Gorman, Vlastimil Babka,
	Andrea Arcangeli, Jerome Glisse, Reza Arbab, Yasuaki Ishimatsu,
	qiuxishi, Kani Toshimitsu, slaoub, Joonsoo Kim, Andi Kleen,
	David Rientjes, Daniel Kiper, Igor Mammedov, Vitaly Kuznetsov,
	LKML, Dan Williams

On Wed 14-06-17 17:12:06, Wei Yang wrote:
> On Wed, Jun 14, 2017 at 08:32:06AM +0200, Michal Hocko wrote:
> >On Wed 14-06-17 14:12:59, Wei Yang wrote:
> >[...]
> >> Hi, Michal
> >> 
> >> Not sure you missed this one or you think this is fine.
> >> 
> >> Hmm... this will not happen since we must offline a whole memory_block?
> >
> >yes
> >
> >> So the memory_hotplug/unplug unit is memory_block instead of mem_section?
> >
> >yes.
> 
> If this is true, the check_hotplug_memory_range() should be fixed too.

as I've said earlier. There are many code paths which are quite
confusing and they expect sub-section granularity while they in fact
won't work with sub memblock granularity. This is a larger project
I am afraid and it would be great if you are willing to try to
consolidate that code. I have that on my todo list but there are more
pressing things to address first for me now.

-- 
Michal Hocko
SUSE Labs

^ permalink raw reply	[flat|nested] 74+ messages in thread

* Re: [PATCH 04/14] mm, memory_hotplug: get rid of is_zone_device_section
@ 2017-06-14  9:24             ` Michal Hocko
  0 siblings, 0 replies; 74+ messages in thread
From: Michal Hocko @ 2017-06-14  9:24 UTC (permalink / raw)
  To: Wei Yang
  Cc: Andrew Morton, Linux-MM, Mel Gorman, Vlastimil Babka,
	Andrea Arcangeli, Jerome Glisse, Reza Arbab, Yasuaki Ishimatsu,
	qiuxishi, Kani Toshimitsu, slaoub, Joonsoo Kim, Andi Kleen,
	David Rientjes, Daniel Kiper, Igor Mammedov, Vitaly Kuznetsov,
	LKML, Dan Williams

On Wed 14-06-17 17:12:06, Wei Yang wrote:
> On Wed, Jun 14, 2017 at 08:32:06AM +0200, Michal Hocko wrote:
> >On Wed 14-06-17 14:12:59, Wei Yang wrote:
> >[...]
> >> Hi, Michal
> >> 
> >> Not sure you missed this one or you think this is fine.
> >> 
> >> Hmm... this will not happen since we must offline a whole memory_block?
> >
> >yes
> >
> >> So the memory_hotplug/unplug unit is memory_block instead of mem_section?
> >
> >yes.
> 
> If this is true, the check_hotplug_memory_range() should be fixed too.

as I've said earlier. There are many code paths which are quite
confusing and they expect sub-section granularity while they in fact
won't work with sub memblock granularity. This is a larger project
I am afraid and it would be great if you are willing to try to
consolidate that code. I have that on my todo list but there are more
pressing things to address first for me now.

-- 
Michal Hocko
SUSE Labs

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

^ permalink raw reply	[flat|nested] 74+ messages in thread

* Re: [PATCH 04/14] mm, memory_hotplug: get rid of is_zone_device_section
  2017-06-14  9:24             ` Michal Hocko
  (?)
@ 2017-06-15  1:02             ` Wei Yang
  -1 siblings, 0 replies; 74+ messages in thread
From: Wei Yang @ 2017-06-15  1:02 UTC (permalink / raw)
  To: Michal Hocko
  Cc: Wei Yang, Andrew Morton, Linux-MM, Mel Gorman, Vlastimil Babka,
	Andrea Arcangeli, Jerome Glisse, Reza Arbab, Yasuaki Ishimatsu,
	qiuxishi, Kani Toshimitsu, slaoub, Joonsoo Kim, Andi Kleen,
	David Rientjes, Daniel Kiper, Igor Mammedov, Vitaly Kuznetsov,
	LKML, Dan Williams

[-- Attachment #1: Type: text/plain, Size: 1112 bytes --]

On Wed, Jun 14, 2017 at 11:24:38AM +0200, Michal Hocko wrote:
>On Wed 14-06-17 17:12:06, Wei Yang wrote:
>> On Wed, Jun 14, 2017 at 08:32:06AM +0200, Michal Hocko wrote:
>> >On Wed 14-06-17 14:12:59, Wei Yang wrote:
>> >[...]
>> >> Hi, Michal
>> >> 
>> >> Not sure you missed this one or you think this is fine.
>> >> 
>> >> Hmm... this will not happen since we must offline a whole memory_block?
>> >
>> >yes
>> >
>> >> So the memory_hotplug/unplug unit is memory_block instead of mem_section?
>> >
>> >yes.
>> 
>> If this is true, the check_hotplug_memory_range() should be fixed too.
>
>as I've said earlier. There are many code paths which are quite
>confusing and they expect sub-section granularity while they in fact
>won't work with sub memblock granularity. This is a larger project
>I am afraid and it would be great if you are willing to try to
>consolidate that code. I have that on my todo list but there are more
>pressing things to address first for me now.
>

Yep, I am willing to have a try. :-)

>-- 
>Michal Hocko
>SUSE Labs

-- 
Wei Yang
Help you, Help me

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

^ permalink raw reply	[flat|nested] 74+ messages in thread

* Re: [PATCH 11/14] mm, memory_hotplug: do not associate hotadded memory to zones until online
  2017-05-15  8:58   ` Michal Hocko
  (?)
  (?)
@ 2017-06-16  4:20   ` Wei Yang
  2017-06-16  8:05       ` Michal Hocko
  -1 siblings, 1 reply; 74+ messages in thread
From: Wei Yang @ 2017-06-16  4:20 UTC (permalink / raw)
  To: Michal Hocko
  Cc: Andrew Morton, linux-mm, Mel Gorman, Vlastimil Babka,
	Andrea Arcangeli, Jerome Glisse, Reza Arbab, Yasuaki Ishimatsu,
	qiuxishi, Kani Toshimitsu, slaoub, Joonsoo Kim, Andi Kleen,
	David Rientjes, Daniel Kiper, Igor Mammedov, Vitaly Kuznetsov,
	LKML, Michal Hocko, Dan Williams, Heiko Carstens,
	Martin Schwidefsky

[-- Attachment #1: Type: text/plain, Size: 30164 bytes --]

On Mon, May 15, 2017 at 10:58:24AM +0200, Michal Hocko wrote:
>From: Michal Hocko <mhocko@suse.com>
>
>The current memory hotplug implementation relies on having all the
>struct pages associate with a zone/node during the physical hotplug phase
>(arch_add_memory->__add_pages->__add_section->__add_zone). In the vast
>majority of cases this means that they are added to ZONE_NORMAL. This
>has been so since 9d99aaa31f59 ("[PATCH] x86_64: Support memory hotadd
>without sparsemem") and it wasn't a big deal back then because movable
>onlining didn't exist yet.
>
>Much later memory hotplug wanted to (ab)use ZONE_MOVABLE for movable
>onlining 511c2aba8f07 ("mm, memory-hotplug: dynamic configure movable
>memory and portion memory") and then things got more complicated. Rather
>than reconsidering the zone association which was no longer needed
>(because the memory hotplug already depended on SPARSEMEM) a convoluted
>semantic of zone shifting has been developed. Only the currently last
>memblock or the one adjacent to the zone_movable can be onlined movable.
>This essentially means that the online type changes as the new memblocks
>are added.
>
>Let's simulate memory hot online manually
>$ echo 0x100000000 > /sys/devices/system/memory/probe
>$ grep . /sys/devices/system/memory/memory32/valid_zones
>Normal Movable
>
>$ echo $((0x100000000+(128<<20))) > /sys/devices/system/memory/probe
>$ grep . /sys/devices/system/memory/memory3?/valid_zones
>/sys/devices/system/memory/memory32/valid_zones:Normal
>/sys/devices/system/memory/memory33/valid_zones:Normal Movable
>
>$ echo $((0x100000000+2*(128<<20))) > /sys/devices/system/memory/probe
>$ grep . /sys/devices/system/memory/memory3?/valid_zones
>/sys/devices/system/memory/memory32/valid_zones:Normal
>/sys/devices/system/memory/memory33/valid_zones:Normal
>/sys/devices/system/memory/memory34/valid_zones:Normal Movable
>
>$ echo online_movable > /sys/devices/system/memory/memory34/state
>$ grep . /sys/devices/system/memory/memory3?/valid_zones
>/sys/devices/system/memory/memory32/valid_zones:Normal
>/sys/devices/system/memory/memory33/valid_zones:Normal Movable
>/sys/devices/system/memory/memory34/valid_zones:Movable Normal
>
>This is an awkward semantic because an udev event is sent as soon as the
>block is onlined and an udev handler might want to online it based on
>some policy (e.g. association with a node) but it will inherently race
>with new blocks showing up.
>
>This patch changes the physical online phase to not associate pages
>with any zone at all. All the pages are just marked reserved and wait
>for the onlining phase to be associated with the zone as per the online
>request. There are only two requirements
>	- existing ZONE_NORMAL and ZONE_MOVABLE cannot overlap
>	- ZONE_NORMAL precedes ZONE_MOVABLE in physical addresses
>the later on is not an inherent requirement and can be changed in the
>future. It preserves the current behavior and made the code slightly
>simpler. This is subject to change in future.
>
>This means that the same physical online steps as above will lead to the
>following state:
>Normal Movable
>
>/sys/devices/system/memory/memory32/valid_zones:Normal Movable
>/sys/devices/system/memory/memory33/valid_zones:Normal Movable
>
>/sys/devices/system/memory/memory32/valid_zones:Normal Movable
>/sys/devices/system/memory/memory33/valid_zones:Normal Movable
>/sys/devices/system/memory/memory34/valid_zones:Normal Movable
>
>/sys/devices/system/memory/memory32/valid_zones:Normal Movable
>/sys/devices/system/memory/memory33/valid_zones:Normal Movable
>/sys/devices/system/memory/memory34/valid_zones:Movable
>
>Implementation:
>The current move_pfn_range is reimplemented to check the above
>requirements (allow_online_pfn_range) and then updates the respective
>zone (move_pfn_range_to_zone), the pgdat and links all the pages in the
>pfn range with the zone/node. __add_pages is updated to not require the
>zone and only initializes sections in the range. This allowed to
>simplify the arch_add_memory code (s390 could get rid of quite some
>of code).
>
>devm_memremap_pages is the only user of arch_add_memory which relies
>on the zone association because it only hooks into the memory hotplug
>only half way. It uses it to associate the new memory with ZONE_DEVICE
>but doesn't allow it to be {on,off}lined via sysfs. This means that this
>particular code path has to call move_pfn_range_to_zone explicitly.
>
>The original zone shifting code is kept in place and will be removed in
>the follow up patch for an easier review.
>
>Please note that this patch also changes the original behavior when
>offlining a memory block adjacent to another zone (Normal vs. Movable)
>used to allow to change its movable type. This will be handled later.
>
>Changes since v1
>- we have to associate the page with the node early (in __add_section),
>  because pfn_to_node depends on struct page containing this
>  information - based on testing by Reza Arbab
>- resize_{zone,pgdat}_range has to check whether they are popoulated -
>  Reza Arbab
>- fix devm_memremap_pages to use pfn rather than physical address -
>  Jérôme Glisse
>- move_pfn_range has to check for intersection with zone_movable rather
>  than to rely on allow_online_pfn_range(MMOP_ONLINE_MOVABLE) for
>  MMOP_ONLINE_KEEP
>
>Changes since v2
>- fix show_valid_zones nr_pages calculation
>- allow_online_pfn_range has to check managed pages rather than present
>- zone_intersects fix bogus check
>- fix zone_intersects + few nits as per Vlastimil
>
>Cc: Martin Schwidefsky <schwidefsky@de.ibm.com>
>Cc: linux-arch@vger.kernel.org
>Tested-by: Dan Williams <dan.j.williams@intel.com>
>Acked-by: Heiko Carstens <heiko.carstens@de.ibm.com> # For s390 bits
>Tested-by: Reza Arbab <arbab@linux.vnet.ibm.com>
>Signed-off-by: Michal Hocko <mhocko@suse.com>
>---
> arch/ia64/mm/init.c            |   9 +-
> arch/powerpc/mm/mem.c          |  10 +-
> arch/s390/mm/init.c            |  30 +-----
> arch/sh/mm/init.c              |   8 +-
> arch/x86/mm/init_32.c          |   5 +-
> arch/x86/mm/init_64.c          |   9 +-
> drivers/base/memory.c          |  52 ++++++-----
> include/linux/memory_hotplug.h |  13 +--
> include/linux/mmzone.h         |  20 ++++
> kernel/memremap.c              |   4 +
> mm/memory_hotplug.c            | 204 +++++++++++++++++++++++++----------------
> mm/sparse.c                    |   3 +-
> 12 files changed, 193 insertions(+), 174 deletions(-)
>
>diff --git a/arch/ia64/mm/init.c b/arch/ia64/mm/init.c
>index 39e2aeb4669d..80db57d063d0 100644
>--- a/arch/ia64/mm/init.c
>+++ b/arch/ia64/mm/init.c
>@@ -648,18 +648,11 @@ mem_init (void)
> #ifdef CONFIG_MEMORY_HOTPLUG
> int arch_add_memory(int nid, u64 start, u64 size, bool for_device)
> {
>-	pg_data_t *pgdat;
>-	struct zone *zone;
> 	unsigned long start_pfn = start >> PAGE_SHIFT;
> 	unsigned long nr_pages = size >> PAGE_SHIFT;
> 	int ret;
> 
>-	pgdat = NODE_DATA(nid);
>-
>-	zone = pgdat->node_zones +
>-		zone_for_memory(nid, start, size, ZONE_NORMAL, for_device);
>-	ret = __add_pages(nid, zone, start_pfn, nr_pages, !for_device);
>-
>+	ret = __add_pages(nid, start_pfn, nr_pages, !for_device);
> 	if (ret)
> 		printk("%s: Problem encountered in __add_pages() as ret=%d\n",
> 		       __func__,  ret);
>diff --git a/arch/powerpc/mm/mem.c b/arch/powerpc/mm/mem.c
>index e6b2e6618b6c..72c46eb53215 100644
>--- a/arch/powerpc/mm/mem.c
>+++ b/arch/powerpc/mm/mem.c
>@@ -128,16 +128,12 @@ int __weak remove_section_mapping(unsigned long start, unsigned long end)
> 
> int arch_add_memory(int nid, u64 start, u64 size, bool for_device)
> {
>-	struct pglist_data *pgdata;
>-	struct zone *zone;
> 	unsigned long start_pfn = start >> PAGE_SHIFT;
> 	unsigned long nr_pages = size >> PAGE_SHIFT;
> 	int rc;
> 
> 	resize_hpt_for_hotplug(memblock_phys_mem_size());
> 
>-	pgdata = NODE_DATA(nid);
>-
> 	start = (unsigned long)__va(start);
> 	rc = create_section_mapping(start, start + size);
> 	if (rc) {
>@@ -147,11 +143,7 @@ int arch_add_memory(int nid, u64 start, u64 size, bool for_device)
> 		return -EFAULT;
> 	}
> 
>-	/* this should work for most non-highmem platforms */
>-	zone = pgdata->node_zones +
>-		zone_for_memory(nid, start, size, 0, for_device);
>-
>-	return __add_pages(nid, zone, start_pfn, nr_pages, !for_device);
>+	return __add_pages(nid, start_pfn, nr_pages, !for_device);
> }
> 
> #ifdef CONFIG_MEMORY_HOTREMOVE
>diff --git a/arch/s390/mm/init.c b/arch/s390/mm/init.c
>index 893cf88cf02d..862824924ba6 100644
>--- a/arch/s390/mm/init.c
>+++ b/arch/s390/mm/init.c
>@@ -164,41 +164,15 @@ unsigned long memory_block_size_bytes(void)
> #ifdef CONFIG_MEMORY_HOTPLUG
> int arch_add_memory(int nid, u64 start, u64 size, bool for_device)
> {
>-	unsigned long zone_start_pfn, zone_end_pfn, nr_pages;
> 	unsigned long start_pfn = PFN_DOWN(start);
> 	unsigned long size_pages = PFN_DOWN(size);
>-	pg_data_t *pgdat = NODE_DATA(nid);
>-	struct zone *zone;
>-	int rc, i;
>+	int rc;
> 
> 	rc = vmem_add_mapping(start, size);
> 	if (rc)
> 		return rc;
> 
>-	for (i = 0; i < MAX_NR_ZONES; i++) {
>-		zone = pgdat->node_zones + i;
>-		if (zone_idx(zone) != ZONE_MOVABLE) {
>-			/* Add range within existing zone limits, if possible */
>-			zone_start_pfn = zone->zone_start_pfn;
>-			zone_end_pfn = zone->zone_start_pfn +
>-				       zone->spanned_pages;
>-		} else {
>-			/* Add remaining range to ZONE_MOVABLE */
>-			zone_start_pfn = start_pfn;
>-			zone_end_pfn = start_pfn + size_pages;
>-		}
>-		if (start_pfn < zone_start_pfn || start_pfn >= zone_end_pfn)
>-			continue;
>-		nr_pages = (start_pfn + size_pages > zone_end_pfn) ?
>-			   zone_end_pfn - start_pfn : size_pages;
>-		rc = __add_pages(nid, zone, start_pfn, nr_pages, !for_device);
>-		if (rc)
>-			break;
>-		start_pfn += nr_pages;
>-		size_pages -= nr_pages;
>-		if (!size_pages)
>-			break;
>-	}
>+	rc = __add_pages(nid, start_pfn, size_pages, !for_device);
> 	if (rc)
> 		vmem_remove_mapping(start, size);
> 	return rc;
>diff --git a/arch/sh/mm/init.c b/arch/sh/mm/init.c
>index a9d57f75ae8c..3813a610a2bb 100644
>--- a/arch/sh/mm/init.c
>+++ b/arch/sh/mm/init.c
>@@ -487,18 +487,12 @@ void free_initrd_mem(unsigned long start, unsigned long end)
> #ifdef CONFIG_MEMORY_HOTPLUG
> int arch_add_memory(int nid, u64 start, u64 size, bool for_device)
> {
>-	pg_data_t *pgdat;
> 	unsigned long start_pfn = PFN_DOWN(start);
> 	unsigned long nr_pages = size >> PAGE_SHIFT;
> 	int ret;
> 
>-	pgdat = NODE_DATA(nid);
>-
> 	/* We only have ZONE_NORMAL, so this is easy.. */
>-	ret = __add_pages(nid, pgdat->node_zones +
>-			zone_for_memory(nid, start, size, ZONE_NORMAL,
>-			for_device),
>-			start_pfn, nr_pages, !for_device);
>+	ret = __add_pages(nid, start_pfn, nr_pages, !for_device);
> 	if (unlikely(ret))
> 		printk("%s: Failed, __add_pages() == %d\n", __func__, ret);
> 
>diff --git a/arch/x86/mm/init_32.c b/arch/x86/mm/init_32.c
>index 94594b889144..a424066d0552 100644
>--- a/arch/x86/mm/init_32.c
>+++ b/arch/x86/mm/init_32.c
>@@ -825,13 +825,10 @@ void __init mem_init(void)
> #ifdef CONFIG_MEMORY_HOTPLUG
> int arch_add_memory(int nid, u64 start, u64 size, bool for_device)
> {
>-	struct pglist_data *pgdata = NODE_DATA(nid);
>-	struct zone *zone = pgdata->node_zones +
>-		zone_for_memory(nid, start, size, ZONE_HIGHMEM, for_device);
> 	unsigned long start_pfn = start >> PAGE_SHIFT;
> 	unsigned long nr_pages = size >> PAGE_SHIFT;
> 
>-	return __add_pages(nid, zone, start_pfn, nr_pages, !for_device);
>+	return __add_pages(nid, start_pfn, nr_pages, !for_device);
> }
> 
> #ifdef CONFIG_MEMORY_HOTREMOVE
>diff --git a/arch/x86/mm/init_64.c b/arch/x86/mm/init_64.c
>index 2e004364a373..884c1d0a57b3 100644
>--- a/arch/x86/mm/init_64.c
>+++ b/arch/x86/mm/init_64.c
>@@ -682,22 +682,15 @@ static void  update_end_of_memory_vars(u64 start, u64 size)
> 	}
> }
> 
>-/*
>- * Memory is added always to NORMAL zone. This means you will never get
>- * additional DMA/DMA32 memory.
>- */
> int arch_add_memory(int nid, u64 start, u64 size, bool for_device)
> {
>-	struct pglist_data *pgdat = NODE_DATA(nid);
>-	struct zone *zone = pgdat->node_zones +
>-		zone_for_memory(nid, start, size, ZONE_NORMAL, for_device);
> 	unsigned long start_pfn = start >> PAGE_SHIFT;
> 	unsigned long nr_pages = size >> PAGE_SHIFT;
> 	int ret;
> 
> 	init_memory_mapping(start, start + size);
> 
>-	ret = __add_pages(nid, zone, start_pfn, nr_pages, !for_device);
>+	ret = __add_pages(nid, start_pfn, nr_pages, !for_device);
> 	WARN_ON_ONCE(ret);
> 
> 	/* update max_pfn, max_low_pfn and high_memory */
>diff --git a/drivers/base/memory.c b/drivers/base/memory.c
>index 1e884d82af6f..b86fda30ce62 100644
>--- a/drivers/base/memory.c
>+++ b/drivers/base/memory.c
>@@ -392,39 +392,43 @@ static ssize_t show_valid_zones(struct device *dev,
> 				struct device_attribute *attr, char *buf)
> {
> 	struct memory_block *mem = to_memory_block(dev);
>-	unsigned long start_pfn, end_pfn;
>-	unsigned long valid_start, valid_end, valid_pages;
>+	unsigned long start_pfn = section_nr_to_pfn(mem->start_section_nr);
> 	unsigned long nr_pages = PAGES_PER_SECTION * sections_per_block;
>-	struct zone *zone;
>-	int zone_shift = 0;
>+	unsigned long valid_start_pfn, valid_end_pfn;
>+	bool append = false;
>+	int nid;
> 
>-	start_pfn = section_nr_to_pfn(mem->start_section_nr);
>-	end_pfn = start_pfn + nr_pages;
>-
>-	/* The block contains more than one zone can not be offlined. */
>-	if (!test_pages_in_a_zone(start_pfn, end_pfn, &valid_start, &valid_end))
>+	/*
>+	 * The block contains more than one zone can not be offlined.
>+	 * This can happen e.g. for ZONE_DMA and ZONE_DMA32
>+	 */
>+	if (!test_pages_in_a_zone(start_pfn, start_pfn + nr_pages, &valid_start_pfn, &valid_end_pfn))
> 		return sprintf(buf, "none\n");
> 
>-	zone = page_zone(pfn_to_page(valid_start));
>-	valid_pages = valid_end - valid_start;
>-
>-	/* MMOP_ONLINE_KEEP */
>-	sprintf(buf, "%s", zone->name);
>+	start_pfn = valid_start_pfn;
>+	nr_pages = valid_end_pfn - start_pfn;
> 
>-	/* MMOP_ONLINE_KERNEL */
>-	zone_can_shift(valid_start, valid_pages, ZONE_NORMAL, &zone_shift);
>-	if (zone_shift) {
>-		strcat(buf, " ");
>-		strcat(buf, (zone + zone_shift)->name);
>+	/*
>+	 * Check the existing zone. Make sure that we do that only on the
>+	 * online nodes otherwise the page_zone is not reliable
>+	 */
>+	if (mem->state == MEM_ONLINE) {
>+		strcat(buf, page_zone(pfn_to_page(start_pfn))->name);
>+		goto out;
> 	}
> 
>-	/* MMOP_ONLINE_MOVABLE */
>-	zone_can_shift(valid_start, valid_pages, ZONE_MOVABLE, &zone_shift);
>-	if (zone_shift) {
>-		strcat(buf, " ");
>-		strcat(buf, (zone + zone_shift)->name);
>+	nid = pfn_to_nid(start_pfn);
>+	if (allow_online_pfn_range(nid, start_pfn, nr_pages, MMOP_ONLINE_KERNEL)) {
>+		strcat(buf, NODE_DATA(nid)->node_zones[ZONE_NORMAL].name);
>+		append = true;
> 	}
> 
>+	if (allow_online_pfn_range(nid, start_pfn, nr_pages, MMOP_ONLINE_MOVABLE)) {
>+		if (append)
>+			strcat(buf, " ");
>+		strcat(buf, NODE_DATA(nid)->node_zones[ZONE_MOVABLE].name);
>+	}
>+out:
> 	strcat(buf, "\n");
> 
> 	return strlen(buf);
>diff --git a/include/linux/memory_hotplug.h b/include/linux/memory_hotplug.h
>index fc1c873504eb..9cd76ff7b0c5 100644
>--- a/include/linux/memory_hotplug.h
>+++ b/include/linux/memory_hotplug.h
>@@ -122,8 +122,8 @@ extern int __remove_pages(struct zone *zone, unsigned long start_pfn,
> 	unsigned long nr_pages);
> #endif /* CONFIG_MEMORY_HOTREMOVE */
> 
>-/* reasonably generic interface to expand the physical pages in a zone  */
>-extern int __add_pages(int nid, struct zone *zone, unsigned long start_pfn,
>+/* reasonably generic interface to expand the physical pages */
>+extern int __add_pages(int nid, unsigned long start_pfn,
> 	unsigned long nr_pages, bool want_memblock);
> 
> #ifdef CONFIG_NUMA
>@@ -298,15 +298,16 @@ extern int add_memory_resource(int nid, struct resource *resource, bool online);
> extern int zone_for_memory(int nid, u64 start, u64 size, int zone_default,
> 		bool for_device);
> extern int arch_add_memory(int nid, u64 start, u64 size, bool for_device);
>+extern void move_pfn_range_to_zone(struct zone *zone, unsigned long start_pfn,
>+		unsigned long nr_pages);
> extern int offline_pages(unsigned long start_pfn, unsigned long nr_pages);
> extern bool is_memblock_offlined(struct memory_block *mem);
> extern void remove_memory(int nid, u64 start, u64 size);
>-extern int sparse_add_one_section(struct zone *zone, unsigned long start_pfn);
>+extern int sparse_add_one_section(struct pglist_data *pgdat, unsigned long start_pfn);
> extern void sparse_remove_one_section(struct zone *zone, struct mem_section *ms,
> 		unsigned long map_offset);
> extern struct page *sparse_decode_mem_map(unsigned long coded_mem_map,
> 					  unsigned long pnum);
>-extern bool zone_can_shift(unsigned long pfn, unsigned long nr_pages,
>-			  enum zone_type target, int *zone_shift);
>-
>+extern bool allow_online_pfn_range(int nid, unsigned long pfn, unsigned long nr_pages,
>+		int online_type);
> #endif /* __LINUX_MEMORY_HOTPLUG_H */
>diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h
>index 927ad95a4552..f887fccb6ef0 100644
>--- a/include/linux/mmzone.h
>+++ b/include/linux/mmzone.h
>@@ -533,6 +533,26 @@ static inline bool zone_is_empty(struct zone *zone)
> }
> 
> /*
>+ * Return true if [start_pfn, start_pfn + nr_pages) range has a non-empty
>+ * intersection with the given zone
>+ */
>+static inline bool zone_intersects(struct zone *zone,
>+		unsigned long start_pfn, unsigned long nr_pages)
>+{
>+	if (zone_is_empty(zone))
>+		return false;
>+	if (start_pfn >= zone_end_pfn(zone))
>+		return false;
>+
>+	if (zone->zone_start_pfn <= start_pfn)
>+		return true;
>+	if (start_pfn + nr_pages > zone->zone_start_pfn)
>+		return true;
>+
>+	return false;
>+}

I think this could be simplified as:

static inline bool zone_intersects(struct zone *zone,
		unsigned long start_pfn, unsigned long nr_pages)
{
	if (zone_is_empty(zone))
		return false;

	if (start_pfn >= zone_end_pfn(zone) ||
	    start_pfn + nr_pages <= zone->zone_start_pfn)
		return false;

	return true;
}

>+
>+/*
>  * The "priority" of VM scanning is how much of the queues we will scan in one
>  * go. A value of 12 for DEF_PRIORITY implies that we will scan 1/4096th of the
>  * queues ("queue_length >> 12") during an aging round.
>diff --git a/kernel/memremap.c b/kernel/memremap.c
>index 23a6483c3666..281eb478856a 100644
>--- a/kernel/memremap.c
>+++ b/kernel/memremap.c
>@@ -359,6 +359,10 @@ void *devm_memremap_pages(struct device *dev, struct resource *res,
> 
> 	mem_hotplug_begin();
> 	error = arch_add_memory(nid, align_start, align_size, true);
>+	if (!error)
>+		move_pfn_range_to_zone(&NODE_DATA(nid)->node_zones[ZONE_DEVICE],
>+					align_start >> PAGE_SHIFT,
>+					align_size >> PAGE_SHIFT);
> 	mem_hotplug_done();
> 	if (error)
> 		goto err_add_memory;
>diff --git a/mm/memory_hotplug.c b/mm/memory_hotplug.c
>index c3a146028ba6..dc363370e9d8 100644
>--- a/mm/memory_hotplug.c
>+++ b/mm/memory_hotplug.c
>@@ -433,25 +433,6 @@ static int __meminit move_pfn_range_right(struct zone *z1, struct zone *z2,
> 	return -1;
> }
> 
>-static struct zone * __meminit move_pfn_range(int zone_shift,
>-		unsigned long start_pfn, unsigned long end_pfn)
>-{
>-	struct zone *zone = page_zone(pfn_to_page(start_pfn));
>-	int ret = 0;
>-
>-	if (zone_shift < 0)
>-		ret = move_pfn_range_left(zone + zone_shift, zone,
>-					  start_pfn, end_pfn);
>-	else if (zone_shift)
>-		ret = move_pfn_range_right(zone, zone + zone_shift,
>-					   start_pfn, end_pfn);
>-
>-	if (ret)
>-		return NULL;
>-
>-	return zone + zone_shift;
>-}
>-
> static void __meminit grow_pgdat_span(struct pglist_data *pgdat, unsigned long start_pfn,
> 				      unsigned long end_pfn)
> {
>@@ -493,23 +474,35 @@ static int __meminit __add_zone(struct zone *zone, unsigned long phys_start_pfn)
> 	return 0;
> }
> 
>-static int __meminit __add_section(int nid, struct zone *zone,
>-		unsigned long phys_start_pfn, bool want_memblock)
>+static int __meminit __add_section(int nid, unsigned long phys_start_pfn,
>+		bool want_memblock)
> {
> 	int ret;
>+	int i;
> 
> 	if (pfn_valid(phys_start_pfn))
> 		return -EEXIST;
> 
>-	ret = sparse_add_one_section(zone, phys_start_pfn);
>-
>+	ret = sparse_add_one_section(NODE_DATA(nid), phys_start_pfn);
> 	if (ret < 0)
> 		return ret;
> 
>-	ret = __add_zone(zone, phys_start_pfn);
>+	/*
>+	 * Make all the pages reserved so that nobody will stumble over half
>+	 * initialized state.
>+	 * FIXME: We also have to associate it with a node because pfn_to_node
>+	 * relies on having page with the proper node.
>+	 */
>+	for (i = 0; i < PAGES_PER_SECTION; i++) {
>+		unsigned long pfn = phys_start_pfn + i;
>+		struct page *page;
>+		if (!pfn_valid(pfn))
>+			continue;
> 
>-	if (ret < 0)
>-		return ret;
>+		page = pfn_to_page(pfn);
>+		set_page_node(page, nid);
>+		SetPageReserved(page);
>+	}
> 
> 	if (!want_memblock)
> 		return 0;
>@@ -523,7 +516,7 @@ static int __meminit __add_section(int nid, struct zone *zone,
>  * call this function after deciding the zone to which to
>  * add the new pages.
>  */
>-int __ref __add_pages(int nid, struct zone *zone, unsigned long phys_start_pfn,
>+int __ref __add_pages(int nid, unsigned long phys_start_pfn,
> 			unsigned long nr_pages, bool want_memblock)
> {
> 	unsigned long i;
>@@ -531,8 +524,6 @@ int __ref __add_pages(int nid, struct zone *zone, unsigned long phys_start_pfn,
> 	int start_sec, end_sec;
> 	struct vmem_altmap *altmap;
> 
>-	clear_zone_contiguous(zone);
>-
> 	/* during initialize mem_map, align hot-added range to section */
> 	start_sec = pfn_to_section_nr(phys_start_pfn);
> 	end_sec = pfn_to_section_nr(phys_start_pfn + nr_pages - 1);
>@@ -552,7 +543,7 @@ int __ref __add_pages(int nid, struct zone *zone, unsigned long phys_start_pfn,
> 	}
> 
> 	for (i = start_sec; i <= end_sec; i++) {
>-		err = __add_section(nid, zone, section_nr_to_pfn(i), want_memblock);
>+		err = __add_section(nid, section_nr_to_pfn(i), want_memblock);
> 
> 		/*
> 		 * EEXIST is finally dealt with by ioresource collision
>@@ -565,7 +556,6 @@ int __ref __add_pages(int nid, struct zone *zone, unsigned long phys_start_pfn,
> 	}
> 	vmemmap_populate_print_last();
> out:
>-	set_zone_contiguous(zone);
> 	return err;
> }
> EXPORT_SYMBOL_GPL(__add_pages);
>@@ -1037,39 +1027,114 @@ static void node_states_set_node(int node, struct memory_notify *arg)
> 	node_set_state(node, N_MEMORY);
> }
> 
>-bool zone_can_shift(unsigned long pfn, unsigned long nr_pages,
>-		   enum zone_type target, int *zone_shift)
>+bool allow_online_pfn_range(int nid, unsigned long pfn, unsigned long nr_pages, int online_type)
> {
>-	struct zone *zone = page_zone(pfn_to_page(pfn));
>-	enum zone_type idx = zone_idx(zone);
>-	int i;
>+	struct pglist_data *pgdat = NODE_DATA(nid);
>+	struct zone *movable_zone = &pgdat->node_zones[ZONE_MOVABLE];
>+	struct zone *normal_zone =  &pgdat->node_zones[ZONE_NORMAL];
> 
>-	*zone_shift = 0;
>+	/*
>+	 * TODO there shouldn't be any inherent reason to have ZONE_NORMAL
>+	 * physically before ZONE_MOVABLE. All we need is they do not
>+	 * overlap. Historically we didn't allow ZONE_NORMAL after ZONE_MOVABLE
>+	 * though so let's stick with it for simplicity for now.
>+	 * TODO make sure we do not overlap with ZONE_DEVICE
>+	 */
>+	if (online_type == MMOP_ONLINE_KERNEL) {
>+		if (zone_is_empty(movable_zone))
>+			return true;
>+		return movable_zone->zone_start_pfn >= pfn + nr_pages;
>+	} else if (online_type == MMOP_ONLINE_MOVABLE) {
>+		return zone_end_pfn(normal_zone) <= pfn;
>+	}
> 
>-	if (idx < target) {
>-		/* pages must be at end of current zone */
>-		if (pfn + nr_pages != zone_end_pfn(zone))
>-			return false;
>+	/* MMOP_ONLINE_KEEP will always succeed and inherits the current zone */
>+	return online_type == MMOP_ONLINE_KEEP;
>+}
>+
>+static void __meminit resize_zone_range(struct zone *zone, unsigned long start_pfn,
>+		unsigned long nr_pages)
>+{
>+	unsigned long old_end_pfn = zone_end_pfn(zone);
>+
>+	if (zone_is_empty(zone) || start_pfn < zone->zone_start_pfn)
>+		zone->zone_start_pfn = start_pfn;
>+
>+	zone->spanned_pages = max(start_pfn + nr_pages, old_end_pfn) - zone->zone_start_pfn;
>+}
>+
>+static void __meminit resize_pgdat_range(struct pglist_data *pgdat, unsigned long start_pfn,
>+                                     unsigned long nr_pages)
>+{
>+	unsigned long old_end_pfn = pgdat_end_pfn(pgdat);
> 
>-		/* no zones in use between current zone and target */
>-		for (i = idx + 1; i < target; i++)
>-			if (zone_is_initialized(zone - idx + i))
>-				return false;
>+	if (!pgdat->node_spanned_pages || start_pfn < pgdat->node_start_pfn)
>+		pgdat->node_start_pfn = start_pfn;
>+
>+	pgdat->node_spanned_pages = max(start_pfn + nr_pages, old_end_pfn) - pgdat->node_start_pfn;
>+}
>+
>+void move_pfn_range_to_zone(struct zone *zone,
>+		unsigned long start_pfn, unsigned long nr_pages)
>+{
>+	struct pglist_data *pgdat = zone->zone_pgdat;
>+	int nid = pgdat->node_id;
>+	unsigned long flags;
>+	unsigned long i;
>+
>+	if (zone_is_empty(zone))
>+		init_currently_empty_zone(zone, start_pfn, nr_pages);
>+
>+	clear_zone_contiguous(zone);
>+
>+	/* TODO Huh pgdat is irqsave while zone is not. It used to be like that before */
>+	pgdat_resize_lock(pgdat, &flags);
>+	zone_span_writelock(zone);
>+	resize_zone_range(zone, start_pfn, nr_pages);
>+	zone_span_writeunlock(zone);
>+	resize_pgdat_range(pgdat, start_pfn, nr_pages);
>+	pgdat_resize_unlock(pgdat, &flags);
>+
>+	/*
>+	 * TODO now we have a visible range of pages which are not associated
>+	 * with their zone properly. Not nice but set_pfnblock_flags_mask
>+	 * expects the zone spans the pfn range. All the pages in the range
>+	 * are reserved so nobody should be touching them so we should be safe
>+	 */
>+	memmap_init_zone(nr_pages, nid, zone_idx(zone), start_pfn, MEMMAP_HOTPLUG);
>+	for (i = 0; i < nr_pages; i++) {
>+		unsigned long pfn = start_pfn + i;
>+		set_page_links(pfn_to_page(pfn), zone_idx(zone), nid, pfn);
> 	}
> 
>-	if (target < idx) {
>-		/* pages must be at beginning of current zone */
>-		if (pfn != zone->zone_start_pfn)
>-			return false;
>+	set_zone_contiguous(zone);
>+}
>+
>+/*
>+ * Associates the given pfn range with the given node and the zone appropriate
>+ * for the given online type.
>+ */
>+static struct zone * __meminit move_pfn_range(int online_type, int nid,
>+		unsigned long start_pfn, unsigned long nr_pages)
>+{
>+	struct pglist_data *pgdat = NODE_DATA(nid);
>+	struct zone *zone = &pgdat->node_zones[ZONE_NORMAL];
> 
>-		/* no zones in use between current zone and target */
>-		for (i = target + 1; i < idx; i++)
>-			if (zone_is_initialized(zone - idx + i))
>-				return false;
>+	if (online_type == MMOP_ONLINE_KEEP) {
>+		struct zone *movable_zone = &pgdat->node_zones[ZONE_MOVABLE];
>+		/*
>+		 * MMOP_ONLINE_KEEP inherits the current zone which is
>+		 * ZONE_NORMAL by default but we might be within ZONE_MOVABLE
>+		 * already.
>+		 */
>+		if (zone_intersects(movable_zone, start_pfn, nr_pages))
>+			zone = movable_zone;
>+	} else if (online_type == MMOP_ONLINE_MOVABLE) {
>+		zone = &pgdat->node_zones[ZONE_MOVABLE];
> 	}
> 
>-	*zone_shift = target - idx;
>-	return true;
>+	move_pfn_range_to_zone(zone, start_pfn, nr_pages);
>+	return zone;
> }
> 
> /* Must be protected by mem_hotplug_begin() */
>@@ -1082,38 +1147,21 @@ int __ref online_pages(unsigned long pfn, unsigned long nr_pages, int online_typ
> 	int nid;
> 	int ret;
> 	struct memory_notify arg;
>-	int zone_shift = 0;
> 
>-	/*
>-	 * This doesn't need a lock to do pfn_to_page().
>-	 * The section can't be removed here because of the
>-	 * memory_block->state_mutex.
>-	 */
>-	zone = page_zone(pfn_to_page(pfn));
>-
>-	if ((zone_idx(zone) > ZONE_NORMAL ||
>-	    online_type == MMOP_ONLINE_MOVABLE) &&
>-	    !can_online_high_movable(pfn_to_nid(pfn)))
>+	nid = pfn_to_nid(pfn);
>+	if (!allow_online_pfn_range(nid, pfn, nr_pages, online_type))
> 		return -EINVAL;
> 
>-	if (online_type == MMOP_ONLINE_KERNEL) {
>-		if (!zone_can_shift(pfn, nr_pages, ZONE_NORMAL, &zone_shift))
>-			return -EINVAL;
>-	} else if (online_type == MMOP_ONLINE_MOVABLE) {
>-		if (!zone_can_shift(pfn, nr_pages, ZONE_MOVABLE, &zone_shift))
>-			return -EINVAL;
>-	}
>-
>-	zone = move_pfn_range(zone_shift, pfn, pfn + nr_pages);
>-	if (!zone)
>+	if (online_type == MMOP_ONLINE_MOVABLE && !can_online_high_movable(nid))
> 		return -EINVAL;
> 
>+	/* associate pfn range with the zone */
>+	zone = move_pfn_range(online_type, nid, pfn, nr_pages);
>+
> 	arg.start_pfn = pfn;
> 	arg.nr_pages = nr_pages;
> 	node_states_check_changes_online(nr_pages, zone, &arg);
> 
>-	nid = zone_to_nid(zone);
>-
> 	ret = memory_notify(MEM_GOING_ONLINE, &arg);
> 	ret = notifier_to_errno(ret);
> 	if (ret)
>diff --git a/mm/sparse.c b/mm/sparse.c
>index 9d7fd666015e..7b4be3fd5cac 100644
>--- a/mm/sparse.c
>+++ b/mm/sparse.c
>@@ -761,10 +761,9 @@ static void free_map_bootmem(struct page *memmap)
>  * set.  If this is <=0, then that means that the passed-in
>  * map was not consumed and must be freed.
>  */
>-int __meminit sparse_add_one_section(struct zone *zone, unsigned long start_pfn)
>+int __meminit sparse_add_one_section(struct pglist_data *pgdat, unsigned long start_pfn)
> {
> 	unsigned long section_nr = pfn_to_section_nr(start_pfn);
>-	struct pglist_data *pgdat = zone->zone_pgdat;
> 	struct mem_section *ms;
> 	struct page *memmap;
> 	unsigned long *usemap;
>-- 
>2.11.0

-- 
Wei Yang
Help you, Help me

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

^ permalink raw reply	[flat|nested] 74+ messages in thread

* Re: [PATCH 11/14] mm, memory_hotplug: do not associate hotadded memory to zones until online
  2017-06-16  4:20   ` Wei Yang
@ 2017-06-16  8:05       ` Michal Hocko
  0 siblings, 0 replies; 74+ messages in thread
From: Michal Hocko @ 2017-06-16  8:05 UTC (permalink / raw)
  To: Wei Yang
  Cc: Andrew Morton, linux-mm, Mel Gorman, Vlastimil Babka,
	Andrea Arcangeli, Jerome Glisse, Reza Arbab, Yasuaki Ishimatsu,
	qiuxishi, Kani Toshimitsu, slaoub, Joonsoo Kim, Andi Kleen,
	David Rientjes, Daniel Kiper, Igor Mammedov, Vitaly Kuznetsov,
	LKML, Dan Williams, Heiko Carstens, Martin Schwidefsky

[Please try to trim the context you are replying to]

On Fri 16-06-17 12:20:58, Wei Yang wrote:
> On Mon, May 15, 2017 at 10:58:24AM +0200, Michal Hocko wrote:
[...]
> > /*
> >+ * Return true if [start_pfn, start_pfn + nr_pages) range has a non-empty
> >+ * intersection with the given zone
> >+ */
> >+static inline bool zone_intersects(struct zone *zone,
> >+		unsigned long start_pfn, unsigned long nr_pages)
> >+{
> >+	if (zone_is_empty(zone))
> >+		return false;
> >+	if (start_pfn >= zone_end_pfn(zone))
> >+		return false;
> >+
> >+	if (zone->zone_start_pfn <= start_pfn)
> >+		return true;
> >+	if (start_pfn + nr_pages > zone->zone_start_pfn)
> >+		return true;
> >+
> >+	return false;
> >+}
> 
> I think this could be simplified as:
> 
> static inline bool zone_intersects(struct zone *zone,
> 		unsigned long start_pfn, unsigned long nr_pages)
> {
> 	if (zone_is_empty(zone))
> 		return false;
> 
> 	if (start_pfn >= zone_end_pfn(zone) ||
> 	    start_pfn + nr_pages <= zone->zone_start_pfn)
> 		return false;
> 
> 	return true;
> }

Feel free to send a patch.
-- 
Michal Hocko
SUSE Labs

^ permalink raw reply	[flat|nested] 74+ messages in thread

* Re: [PATCH 11/14] mm, memory_hotplug: do not associate hotadded memory to zones until online
@ 2017-06-16  8:05       ` Michal Hocko
  0 siblings, 0 replies; 74+ messages in thread
From: Michal Hocko @ 2017-06-16  8:05 UTC (permalink / raw)
  To: Wei Yang
  Cc: Andrew Morton, linux-mm, Mel Gorman, Vlastimil Babka,
	Andrea Arcangeli, Jerome Glisse, Reza Arbab, Yasuaki Ishimatsu,
	qiuxishi, Kani Toshimitsu, slaoub, Joonsoo Kim, Andi Kleen,
	David Rientjes, Daniel Kiper, Igor Mammedov, Vitaly Kuznetsov,
	LKML, Dan Williams, Heiko Carstens, Martin Schwidefsky

[Please try to trim the context you are replying to]

On Fri 16-06-17 12:20:58, Wei Yang wrote:
> On Mon, May 15, 2017 at 10:58:24AM +0200, Michal Hocko wrote:
[...]
> > /*
> >+ * Return true if [start_pfn, start_pfn + nr_pages) range has a non-empty
> >+ * intersection with the given zone
> >+ */
> >+static inline bool zone_intersects(struct zone *zone,
> >+		unsigned long start_pfn, unsigned long nr_pages)
> >+{
> >+	if (zone_is_empty(zone))
> >+		return false;
> >+	if (start_pfn >= zone_end_pfn(zone))
> >+		return false;
> >+
> >+	if (zone->zone_start_pfn <= start_pfn)
> >+		return true;
> >+	if (start_pfn + nr_pages > zone->zone_start_pfn)
> >+		return true;
> >+
> >+	return false;
> >+}
> 
> I think this could be simplified as:
> 
> static inline bool zone_intersects(struct zone *zone,
> 		unsigned long start_pfn, unsigned long nr_pages)
> {
> 	if (zone_is_empty(zone))
> 		return false;
> 
> 	if (start_pfn >= zone_end_pfn(zone) ||
> 	    start_pfn + nr_pages <= zone->zone_start_pfn)
> 		return false;
> 
> 	return true;
> }

Feel free to send a patch.
-- 
Michal Hocko
SUSE Labs

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

^ permalink raw reply	[flat|nested] 74+ messages in thread

* Re: [PATCH 11/14] mm, memory_hotplug: do not associate hotadded memory to zones until online
  2017-05-15  8:58   ` Michal Hocko
                     ` (2 preceding siblings ...)
  (?)
@ 2017-06-16  8:11   ` Wei Yang
  2017-06-16  8:45       ` Michal Hocko
  -1 siblings, 1 reply; 74+ messages in thread
From: Wei Yang @ 2017-06-16  8:11 UTC (permalink / raw)
  To: Michal Hocko
  Cc: Andrew Morton, linux-mm, Mel Gorman, Vlastimil Babka,
	Andrea Arcangeli, Jerome Glisse, Reza Arbab, Yasuaki Ishimatsu,
	qiuxishi, Kani Toshimitsu, slaoub, Joonsoo Kim, Andi Kleen,
	David Rientjes, Daniel Kiper, Igor Mammedov, Vitaly Kuznetsov,
	LKML, Michal Hocko, Dan Williams, Heiko Carstens,
	Martin Schwidefsky

[-- Attachment #1: Type: text/plain, Size: 1648 bytes --]

Well, I love this patch a lot. We don't need to put the hotadd memory in one
zone and move it to another. This looks great!

On Mon, May 15, 2017 at 10:58:24AM +0200, Michal Hocko wrote:
>From: Michal Hocko <mhocko@suse.com>
>
[...]
+
>+void move_pfn_range_to_zone(struct zone *zone,
>+		unsigned long start_pfn, unsigned long nr_pages)
>+{
>+	struct pglist_data *pgdat = zone->zone_pgdat;
>+	int nid = pgdat->node_id;
>+	unsigned long flags;
>+	unsigned long i;
>+
>+	if (zone_is_empty(zone))
>+		init_currently_empty_zone(zone, start_pfn, nr_pages);
>+
>+	clear_zone_contiguous(zone);
>+
>+	/* TODO Huh pgdat is irqsave while zone is not. It used to be like that before */
>+	pgdat_resize_lock(pgdat, &flags);
>+	zone_span_writelock(zone);
>+	resize_zone_range(zone, start_pfn, nr_pages);
>+	zone_span_writeunlock(zone);
>+	resize_pgdat_range(pgdat, start_pfn, nr_pages);
>+	pgdat_resize_unlock(pgdat, &flags);
>+
>+	/*
>+	 * TODO now we have a visible range of pages which are not associated
>+	 * with their zone properly. Not nice but set_pfnblock_flags_mask
>+	 * expects the zone spans the pfn range. All the pages in the range
>+	 * are reserved so nobody should be touching them so we should be safe
>+	 */
>+	memmap_init_zone(nr_pages, nid, zone_idx(zone), start_pfn, MEMMAP_HOTPLUG);
>+	for (i = 0; i < nr_pages; i++) {
>+		unsigned long pfn = start_pfn + i;
>+		set_page_links(pfn_to_page(pfn), zone_idx(zone), nid, pfn);
> 	}

memmap_init_zone()->__init_single_page()->set_page_links()

Do I miss something that you call set_page_links() explicitly here?

-- 
Wei Yang
Help you, Help me

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

^ permalink raw reply	[flat|nested] 74+ messages in thread

* Re: [PATCH 11/14] mm, memory_hotplug: do not associate hotadded memory to zones until online
  2017-06-16  8:11   ` Wei Yang
@ 2017-06-16  8:45       ` Michal Hocko
  0 siblings, 0 replies; 74+ messages in thread
From: Michal Hocko @ 2017-06-16  8:45 UTC (permalink / raw)
  To: Wei Yang
  Cc: Andrew Morton, linux-mm, Mel Gorman, Vlastimil Babka,
	Andrea Arcangeli, Jerome Glisse, Reza Arbab, Yasuaki Ishimatsu,
	qiuxishi, Kani Toshimitsu, slaoub, Joonsoo Kim, Andi Kleen,
	David Rientjes, Daniel Kiper, Igor Mammedov, Vitaly Kuznetsov,
	LKML, Dan Williams, Heiko Carstens, Martin Schwidefsky

On Fri 16-06-17 16:11:42, Wei Yang wrote:
> Well, I love this patch a lot. We don't need to put the hotadd memory in one
> zone and move it to another. This looks great!
> 
> On Mon, May 15, 2017 at 10:58:24AM +0200, Michal Hocko wrote:
> >From: Michal Hocko <mhocko@suse.com>
> >
> [...]
> +
> >+void move_pfn_range_to_zone(struct zone *zone,
> >+		unsigned long start_pfn, unsigned long nr_pages)
> >+{
> >+	struct pglist_data *pgdat = zone->zone_pgdat;
> >+	int nid = pgdat->node_id;
> >+	unsigned long flags;
> >+	unsigned long i;
> >+
> >+	if (zone_is_empty(zone))
> >+		init_currently_empty_zone(zone, start_pfn, nr_pages);
> >+
> >+	clear_zone_contiguous(zone);
> >+
> >+	/* TODO Huh pgdat is irqsave while zone is not. It used to be like that before */
> >+	pgdat_resize_lock(pgdat, &flags);
> >+	zone_span_writelock(zone);
> >+	resize_zone_range(zone, start_pfn, nr_pages);
> >+	zone_span_writeunlock(zone);
> >+	resize_pgdat_range(pgdat, start_pfn, nr_pages);
> >+	pgdat_resize_unlock(pgdat, &flags);
> >+
> >+	/*
> >+	 * TODO now we have a visible range of pages which are not associated
> >+	 * with their zone properly. Not nice but set_pfnblock_flags_mask
> >+	 * expects the zone spans the pfn range. All the pages in the range
> >+	 * are reserved so nobody should be touching them so we should be safe
> >+	 */
> >+	memmap_init_zone(nr_pages, nid, zone_idx(zone), start_pfn, MEMMAP_HOTPLUG);
> >+	for (i = 0; i < nr_pages; i++) {
> >+		unsigned long pfn = start_pfn + i;
> >+		set_page_links(pfn_to_page(pfn), zone_idx(zone), nid, pfn);
> > 	}
> 
> memmap_init_zone()->__init_single_page()->set_page_links()
> 
> Do I miss something that you call set_page_links() explicitly here?

I guess you are right. Not sure why I've done this explicitly. I've most
probably just missed that. Could you post a patch that removes the for
loop.

Thanks!
-- 
Michal Hocko
SUSE Labs

^ permalink raw reply	[flat|nested] 74+ messages in thread

* Re: [PATCH 11/14] mm, memory_hotplug: do not associate hotadded memory to zones until online
@ 2017-06-16  8:45       ` Michal Hocko
  0 siblings, 0 replies; 74+ messages in thread
From: Michal Hocko @ 2017-06-16  8:45 UTC (permalink / raw)
  To: Wei Yang
  Cc: Andrew Morton, linux-mm, Mel Gorman, Vlastimil Babka,
	Andrea Arcangeli, Jerome Glisse, Reza Arbab, Yasuaki Ishimatsu,
	qiuxishi, Kani Toshimitsu, slaoub, Joonsoo Kim, Andi Kleen,
	David Rientjes, Daniel Kiper, Igor Mammedov, Vitaly Kuznetsov,
	LKML, Dan Williams, Heiko Carstens, Martin Schwidefsky

On Fri 16-06-17 16:11:42, Wei Yang wrote:
> Well, I love this patch a lot. We don't need to put the hotadd memory in one
> zone and move it to another. This looks great!
> 
> On Mon, May 15, 2017 at 10:58:24AM +0200, Michal Hocko wrote:
> >From: Michal Hocko <mhocko@suse.com>
> >
> [...]
> +
> >+void move_pfn_range_to_zone(struct zone *zone,
> >+		unsigned long start_pfn, unsigned long nr_pages)
> >+{
> >+	struct pglist_data *pgdat = zone->zone_pgdat;
> >+	int nid = pgdat->node_id;
> >+	unsigned long flags;
> >+	unsigned long i;
> >+
> >+	if (zone_is_empty(zone))
> >+		init_currently_empty_zone(zone, start_pfn, nr_pages);
> >+
> >+	clear_zone_contiguous(zone);
> >+
> >+	/* TODO Huh pgdat is irqsave while zone is not. It used to be like that before */
> >+	pgdat_resize_lock(pgdat, &flags);
> >+	zone_span_writelock(zone);
> >+	resize_zone_range(zone, start_pfn, nr_pages);
> >+	zone_span_writeunlock(zone);
> >+	resize_pgdat_range(pgdat, start_pfn, nr_pages);
> >+	pgdat_resize_unlock(pgdat, &flags);
> >+
> >+	/*
> >+	 * TODO now we have a visible range of pages which are not associated
> >+	 * with their zone properly. Not nice but set_pfnblock_flags_mask
> >+	 * expects the zone spans the pfn range. All the pages in the range
> >+	 * are reserved so nobody should be touching them so we should be safe
> >+	 */
> >+	memmap_init_zone(nr_pages, nid, zone_idx(zone), start_pfn, MEMMAP_HOTPLUG);
> >+	for (i = 0; i < nr_pages; i++) {
> >+		unsigned long pfn = start_pfn + i;
> >+		set_page_links(pfn_to_page(pfn), zone_idx(zone), nid, pfn);
> > 	}
> 
> memmap_init_zone()->__init_single_page()->set_page_links()
> 
> Do I miss something that you call set_page_links() explicitly here?

I guess you are right. Not sure why I've done this explicitly. I've most
probably just missed that. Could you post a patch that removes the for
loop.

Thanks!
-- 
Michal Hocko
SUSE Labs

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

^ permalink raw reply	[flat|nested] 74+ messages in thread

* Re: [PATCH 11/14] mm, memory_hotplug: do not associate hotadded memory to zones until online
  2017-06-16  8:45       ` Michal Hocko
  (?)
@ 2017-06-16  9:11       ` Wei Yang
  -1 siblings, 0 replies; 74+ messages in thread
From: Wei Yang @ 2017-06-16  9:11 UTC (permalink / raw)
  To: Michal Hocko
  Cc: Wei Yang, Andrew Morton, linux-mm, Mel Gorman, Vlastimil Babka,
	Andrea Arcangeli, Jerome Glisse, Reza Arbab, Yasuaki Ishimatsu,
	qiuxishi, Kani Toshimitsu, slaoub, Joonsoo Kim, Andi Kleen,
	David Rientjes, Daniel Kiper, Igor Mammedov, Vitaly Kuznetsov,
	LKML, Dan Williams, Heiko Carstens, Martin Schwidefsky

[-- Attachment #1: Type: text/plain, Size: 2168 bytes --]

On Fri, Jun 16, 2017 at 10:45:55AM +0200, Michal Hocko wrote:
>On Fri 16-06-17 16:11:42, Wei Yang wrote:
>> Well, I love this patch a lot. We don't need to put the hotadd memory in one
>> zone and move it to another. This looks great!
>> 
>> On Mon, May 15, 2017 at 10:58:24AM +0200, Michal Hocko wrote:
>> >From: Michal Hocko <mhocko@suse.com>
>> >
>> [...]
>> +
>> >+void move_pfn_range_to_zone(struct zone *zone,
>> >+		unsigned long start_pfn, unsigned long nr_pages)
>> >+{
>> >+	struct pglist_data *pgdat = zone->zone_pgdat;
>> >+	int nid = pgdat->node_id;
>> >+	unsigned long flags;
>> >+	unsigned long i;
>> >+
>> >+	if (zone_is_empty(zone))
>> >+		init_currently_empty_zone(zone, start_pfn, nr_pages);
>> >+
>> >+	clear_zone_contiguous(zone);
>> >+
>> >+	/* TODO Huh pgdat is irqsave while zone is not. It used to be like that before */
>> >+	pgdat_resize_lock(pgdat, &flags);
>> >+	zone_span_writelock(zone);
>> >+	resize_zone_range(zone, start_pfn, nr_pages);
>> >+	zone_span_writeunlock(zone);
>> >+	resize_pgdat_range(pgdat, start_pfn, nr_pages);
>> >+	pgdat_resize_unlock(pgdat, &flags);
>> >+
>> >+	/*
>> >+	 * TODO now we have a visible range of pages which are not associated
>> >+	 * with their zone properly. Not nice but set_pfnblock_flags_mask
>> >+	 * expects the zone spans the pfn range. All the pages in the range
>> >+	 * are reserved so nobody should be touching them so we should be safe
>> >+	 */
>> >+	memmap_init_zone(nr_pages, nid, zone_idx(zone), start_pfn, MEMMAP_HOTPLUG);
>> >+	for (i = 0; i < nr_pages; i++) {
>> >+		unsigned long pfn = start_pfn + i;
>> >+		set_page_links(pfn_to_page(pfn), zone_idx(zone), nid, pfn);
>> > 	}
>> 
>> memmap_init_zone()->__init_single_page()->set_page_links()
>> 
>> Do I miss something that you call set_page_links() explicitly here?
>
>I guess you are right. Not sure why I've done this explicitly. I've most
>probably just missed that. Could you post a patch that removes the for
>loop.
>

Sure, I will come up with two patches based on you auto-latest branch.

>Thanks!
>-- 
>Michal Hocko
>SUSE Labs

-- 
Wei Yang
Help you, Help me

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

^ permalink raw reply	[flat|nested] 74+ messages in thread

* Re: [PATCH 11/14] mm, memory_hotplug: do not associate hotadded memory to zones until online
  2017-05-15  8:58   ` Michal Hocko
                     ` (3 preceding siblings ...)
  (?)
@ 2017-06-25  0:14   ` Wei Yang
  2017-06-26  5:38       ` Michal Hocko
  -1 siblings, 1 reply; 74+ messages in thread
From: Wei Yang @ 2017-06-25  0:14 UTC (permalink / raw)
  To: Michal Hocko
  Cc: Andrew Morton, linux-mm, Mel Gorman, Vlastimil Babka,
	Andrea Arcangeli, Jerome Glisse, Reza Arbab, Yasuaki Ishimatsu,
	qiuxishi, Kani Toshimitsu, slaoub, Joonsoo Kim, Andi Kleen,
	David Rientjes, Daniel Kiper, Igor Mammedov, Vitaly Kuznetsov,
	LKML, Michal Hocko, Dan Williams, Heiko Carstens,
	Martin Schwidefsky

[-- Attachment #1: Type: text/plain, Size: 1662 bytes --]

On Mon, May 15, 2017 at 10:58:24AM +0200, Michal Hocko wrote:
>From: Michal Hocko <mhocko@suse.com>
>
[...]
>+void move_pfn_range_to_zone(struct zone *zone,
>+		unsigned long start_pfn, unsigned long nr_pages)
>+{
>+	struct pglist_data *pgdat = zone->zone_pgdat;
>+	int nid = pgdat->node_id;
>+	unsigned long flags;
>+	unsigned long i;

This is an unused variable:

  mm/memory_hotplug.c: In function ‘move_pfn_range_to_zone’:
  mm/memory_hotplug.c:895:16: warning: unused variable ‘i’ [-Wunused-variable]

Do you suggest me to write a patch or you would fix it in your later rework?

>+
>+	if (zone_is_empty(zone))
>+		init_currently_empty_zone(zone, start_pfn, nr_pages);
>+
>+	clear_zone_contiguous(zone);
>+
>+	/* TODO Huh pgdat is irqsave while zone is not. It used to be like that before */
>+	pgdat_resize_lock(pgdat, &flags);
>+	zone_span_writelock(zone);
>+	resize_zone_range(zone, start_pfn, nr_pages);
>+	zone_span_writeunlock(zone);
>+	resize_pgdat_range(pgdat, start_pfn, nr_pages);
>+	pgdat_resize_unlock(pgdat, &flags);
>+
>+	/*
>+	 * TODO now we have a visible range of pages which are not associated
>+	 * with their zone properly. Not nice but set_pfnblock_flags_mask
>+	 * expects the zone spans the pfn range. All the pages in the range
>+	 * are reserved so nobody should be touching them so we should be safe
>+	 */
>+	memmap_init_zone(nr_pages, nid, zone_idx(zone), start_pfn, MEMMAP_HOTPLUG);
>+	for (i = 0; i < nr_pages; i++) {
>+		unsigned long pfn = start_pfn + i;
>+		set_page_links(pfn_to_page(pfn), zone_idx(zone), nid, pfn);
> 	}
> 
>2.11.0

-- 
Wei Yang
Help you, Help me

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

^ permalink raw reply	[flat|nested] 74+ messages in thread

* Re: [PATCH 11/14] mm, memory_hotplug: do not associate hotadded memory to zones until online
  2017-06-25  0:14   ` Wei Yang
@ 2017-06-26  5:38       ` Michal Hocko
  0 siblings, 0 replies; 74+ messages in thread
From: Michal Hocko @ 2017-06-26  5:38 UTC (permalink / raw)
  To: Wei Yang
  Cc: Andrew Morton, linux-mm, Mel Gorman, Vlastimil Babka,
	Andrea Arcangeli, Jerome Glisse, Reza Arbab, Yasuaki Ishimatsu,
	qiuxishi, Kani Toshimitsu, slaoub, Joonsoo Kim, Andi Kleen,
	David Rientjes, Daniel Kiper, Igor Mammedov, Vitaly Kuznetsov,
	LKML, Dan Williams, Heiko Carstens, Martin Schwidefsky

On Sun 25-06-17 08:14:13, Wei Yang wrote:
> On Mon, May 15, 2017 at 10:58:24AM +0200, Michal Hocko wrote:
> >From: Michal Hocko <mhocko@suse.com>
> >
> [...]
> >+void move_pfn_range_to_zone(struct zone *zone,
> >+		unsigned long start_pfn, unsigned long nr_pages)
> >+{
> >+	struct pglist_data *pgdat = zone->zone_pgdat;
> >+	int nid = pgdat->node_id;
> >+	unsigned long flags;
> >+	unsigned long i;
> 
> This is an unused variable:
> 
>   mm/memory_hotplug.c: In function ‘move_pfn_range_to_zone’:
>   mm/memory_hotplug.c:895:16: warning: unused variable ‘i’ [-Wunused-variable]
> 
> Do you suggest me to write a patch or you would fix it in your later rework?

Please send a fix for your
http://lkml.kernel.org/r/20170616092335.5177-2-richard.weiyang@gmail.com
Andrew will fold it into that patch.

> 
> >+
> >+	if (zone_is_empty(zone))
> >+		init_currently_empty_zone(zone, start_pfn, nr_pages);
> >+
> >+	clear_zone_contiguous(zone);
> >+
> >+	/* TODO Huh pgdat is irqsave while zone is not. It used to be like that before */
> >+	pgdat_resize_lock(pgdat, &flags);
> >+	zone_span_writelock(zone);
> >+	resize_zone_range(zone, start_pfn, nr_pages);
> >+	zone_span_writeunlock(zone);
> >+	resize_pgdat_range(pgdat, start_pfn, nr_pages);
> >+	pgdat_resize_unlock(pgdat, &flags);
> >+
> >+	/*
> >+	 * TODO now we have a visible range of pages which are not associated
> >+	 * with their zone properly. Not nice but set_pfnblock_flags_mask
> >+	 * expects the zone spans the pfn range. All the pages in the range
> >+	 * are reserved so nobody should be touching them so we should be safe
> >+	 */
> >+	memmap_init_zone(nr_pages, nid, zone_idx(zone), start_pfn, MEMMAP_HOTPLUG);
> >+	for (i = 0; i < nr_pages; i++) {
> >+		unsigned long pfn = start_pfn + i;
> >+		set_page_links(pfn_to_page(pfn), zone_idx(zone), nid, pfn);
> > 	}
> > 
> >2.11.0
> 
> -- 
> Wei Yang
> Help you, Help me



-- 
Michal Hocko
SUSE Labs

^ permalink raw reply	[flat|nested] 74+ messages in thread

* Re: [PATCH 11/14] mm, memory_hotplug: do not associate hotadded memory to zones until online
@ 2017-06-26  5:38       ` Michal Hocko
  0 siblings, 0 replies; 74+ messages in thread
From: Michal Hocko @ 2017-06-26  5:38 UTC (permalink / raw)
  To: Wei Yang
  Cc: Andrew Morton, linux-mm, Mel Gorman, Vlastimil Babka,
	Andrea Arcangeli, Jerome Glisse, Reza Arbab, Yasuaki Ishimatsu,
	qiuxishi, Kani Toshimitsu, slaoub, Joonsoo Kim, Andi Kleen,
	David Rientjes, Daniel Kiper, Igor Mammedov, Vitaly Kuznetsov,
	LKML, Dan Williams, Heiko Carstens, Martin Schwidefsky

On Sun 25-06-17 08:14:13, Wei Yang wrote:
> On Mon, May 15, 2017 at 10:58:24AM +0200, Michal Hocko wrote:
> >From: Michal Hocko <mhocko@suse.com>
> >
> [...]
> >+void move_pfn_range_to_zone(struct zone *zone,
> >+		unsigned long start_pfn, unsigned long nr_pages)
> >+{
> >+	struct pglist_data *pgdat = zone->zone_pgdat;
> >+	int nid = pgdat->node_id;
> >+	unsigned long flags;
> >+	unsigned long i;
> 
> This is an unused variable:
> 
>   mm/memory_hotplug.c: In function a??move_pfn_range_to_zonea??:
>   mm/memory_hotplug.c:895:16: warning: unused variable a??ia?? [-Wunused-variable]
> 
> Do you suggest me to write a patch or you would fix it in your later rework?

Please send a fix for your
http://lkml.kernel.org/r/20170616092335.5177-2-richard.weiyang@gmail.com
Andrew will fold it into that patch.

> 
> >+
> >+	if (zone_is_empty(zone))
> >+		init_currently_empty_zone(zone, start_pfn, nr_pages);
> >+
> >+	clear_zone_contiguous(zone);
> >+
> >+	/* TODO Huh pgdat is irqsave while zone is not. It used to be like that before */
> >+	pgdat_resize_lock(pgdat, &flags);
> >+	zone_span_writelock(zone);
> >+	resize_zone_range(zone, start_pfn, nr_pages);
> >+	zone_span_writeunlock(zone);
> >+	resize_pgdat_range(pgdat, start_pfn, nr_pages);
> >+	pgdat_resize_unlock(pgdat, &flags);
> >+
> >+	/*
> >+	 * TODO now we have a visible range of pages which are not associated
> >+	 * with their zone properly. Not nice but set_pfnblock_flags_mask
> >+	 * expects the zone spans the pfn range. All the pages in the range
> >+	 * are reserved so nobody should be touching them so we should be safe
> >+	 */
> >+	memmap_init_zone(nr_pages, nid, zone_idx(zone), start_pfn, MEMMAP_HOTPLUG);
> >+	for (i = 0; i < nr_pages; i++) {
> >+		unsigned long pfn = start_pfn + i;
> >+		set_page_links(pfn_to_page(pfn), zone_idx(zone), nid, pfn);
> > 	}
> > 
> >2.11.0
> 
> -- 
> Wei Yang
> Help you, Help me



-- 
Michal Hocko
SUSE Labs

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

^ permalink raw reply	[flat|nested] 74+ messages in thread

end of thread, other threads:[~2017-06-26  5:38 UTC | newest]

Thread overview: 74+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2017-05-15  8:58 [PATCH -v4 0/14] mm: make movable onlining suck less Michal Hocko
2017-05-15  8:58 ` Michal Hocko
2017-05-15  8:58 ` [PATCH 01/14] mm: remove return value from init_currently_empty_zone Michal Hocko
2017-05-15  8:58   ` Michal Hocko
2017-05-15  8:58 ` [PATCH 02/14] mm, memory_hotplug: use node instead of zone in can_online_high_movable Michal Hocko
2017-05-15  8:58   ` Michal Hocko
2017-05-15  8:58 ` [PATCH 03/14] mm: drop page_initialized check from get_nid_for_pfn Michal Hocko
2017-05-15  8:58   ` Michal Hocko
2017-05-15  8:58 ` [PATCH 04/14] mm, memory_hotplug: get rid of is_zone_device_section Michal Hocko
2017-05-15  8:58   ` Michal Hocko
2017-06-10  9:56   ` Wei Yang
2017-06-10  9:56     ` Wei Yang
2017-06-10 14:58     ` Wei Yang
2017-06-10 14:58       ` Wei Yang
2017-06-12  6:49       ` Michal Hocko
2017-06-12  6:49         ` Michal Hocko
2017-06-14  6:17         ` Wei Yang
2017-06-14  6:12     ` Wei Yang
2017-06-14  6:32       ` Michal Hocko
2017-06-14  6:32         ` Michal Hocko
2017-06-14  9:12         ` Wei Yang
2017-06-14  9:24           ` Michal Hocko
2017-06-14  9:24             ` Michal Hocko
2017-06-15  1:02             ` Wei Yang
2017-05-15  8:58 ` [PATCH 05/14] mm, memory_hotplug: split up register_one_node Michal Hocko
2017-05-15  8:58   ` Michal Hocko
2017-05-15  8:58 ` [PATCH 06/14] mm, memory_hotplug: consider offline memblocks removable Michal Hocko
2017-05-15  8:58   ` Michal Hocko
2017-05-15  8:58 ` [PATCH 07/14] mm: consider zone which is not fully populated to have holes Michal Hocko
2017-05-15  8:58   ` Michal Hocko
2017-05-18 16:14   ` Vlastimil Babka
2017-05-18 16:14     ` Vlastimil Babka
2017-05-18 16:42     ` Michal Hocko
2017-05-18 16:42       ` Michal Hocko
2017-05-19  7:21       ` Vlastimil Babka
2017-05-19  7:21         ` Vlastimil Babka
2017-05-15  8:58 ` [PATCH 08/14] mm, compaction: skip over holes in __reset_isolation_suitable Michal Hocko
2017-05-15  8:58   ` Michal Hocko
2017-05-19  7:23   ` Vlastimil Babka
2017-05-19  7:23     ` Vlastimil Babka
2017-05-15  8:58 ` [PATCH 09/14] mm: __first_valid_page skip over offline pages Michal Hocko
2017-05-15  8:58   ` Michal Hocko
2017-05-19  8:05   ` Vlastimil Babka
2017-05-19  8:05     ` Vlastimil Babka
2017-05-15  8:58 ` [PATCH 10/14] mm, vmstat: skip reporting offline pages in pagetypeinfo Michal Hocko
2017-05-15  8:58   ` Michal Hocko
2017-05-19  8:17   ` Vlastimil Babka
2017-05-19  8:17     ` Vlastimil Babka
2017-05-15  8:58 ` [PATCH 11/14] mm, memory_hotplug: do not associate hotadded memory to zones until online Michal Hocko
2017-05-15  8:58   ` Michal Hocko
2017-05-19  8:56   ` Vlastimil Babka
2017-05-19  8:56     ` Vlastimil Babka
2017-06-16  4:20   ` Wei Yang
2017-06-16  8:05     ` Michal Hocko
2017-06-16  8:05       ` Michal Hocko
2017-06-16  8:11   ` Wei Yang
2017-06-16  8:45     ` Michal Hocko
2017-06-16  8:45       ` Michal Hocko
2017-06-16  9:11       ` Wei Yang
2017-06-25  0:14   ` Wei Yang
2017-06-26  5:38     ` Michal Hocko
2017-06-26  5:38       ` Michal Hocko
2017-05-15  8:58 ` [PATCH 12/14] mm, memory_hotplug: replace for_device by want_memblock in arch_add_memory Michal Hocko
2017-05-15  8:58   ` Michal Hocko
2017-05-15  8:58 ` [PATCH 13/14] mm, memory_hotplug: fix the section mismatch warning Michal Hocko
2017-05-15  8:58   ` Michal Hocko
2017-05-15  8:58 ` [PATCH 14/14] mm, memory_hotplug: remove unused cruft after memory hotplug rework Michal Hocko
2017-05-15  8:58   ` Michal Hocko
2017-06-09  9:51 ` [PATCH -v4 0/14] mm: make movable onlining suck less Wei Yang
2017-06-09  9:51   ` Wei Yang
2017-06-09 10:49   ` Michal Hocko
2017-06-09 10:49     ` Michal Hocko
2017-06-10  2:20     ` Wei Yang
2017-06-10  2:20       ` Wei Yang

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.