* [PATCH 0/2] docs: memory-hotplug: add details about locking internals @ 2018-10-11 4:58 Mike Rapoport 2018-10-11 4:58 ` [PATCH 1/2] docs/core-api: rename memory-hotplug-notifier to memory-hotplug Mike Rapoport ` (2 more replies) 0 siblings, 3 replies; 7+ messages in thread From: Mike Rapoport @ 2018-10-11 4:58 UTC (permalink / raw) To: Jonathan Corbet Cc: David Hildenbrand, Andrew Morton, Stephen Rothwell, linux-doc, linux-kernel, Mike Rapoport Hi, As discussed at [1], the latest updates to memory hotplug documentation are causing a conflict between docs and mmotm trees. These patches resolve the conflict. [1] https://lkml.org/lkml/2018/10/8/227 David Hildenbrand (1): docs/core-api: memory-hotplug: add some details about locking internals Mike Rapoport (1): docs/core-api: rename memory-hotplug-notifier to memory-hotplug Documentation/core-api/index.rst | 2 +- Documentation/core-api/memory-hotplug-notifier.rst | 84 -------------- Documentation/core-api/memory-hotplug.rst | 125 +++++++++++++++++++++ 3 files changed, 126 insertions(+), 85 deletions(-) delete mode 100644 Documentation/core-api/memory-hotplug-notifier.rst create mode 100644 Documentation/core-api/memory-hotplug.rst -- 2.7.4 ^ permalink raw reply [flat|nested] 7+ messages in thread
* [PATCH 1/2] docs/core-api: rename memory-hotplug-notifier to memory-hotplug 2018-10-11 4:58 [PATCH 0/2] docs: memory-hotplug: add details about locking internals Mike Rapoport @ 2018-10-11 4:58 ` Mike Rapoport 2018-10-11 7:43 ` David Hildenbrand 2018-10-11 4:58 ` [PATCH 2/2] docs/core-api: memory-hotplug: add some details about locking internals Mike Rapoport 2018-10-12 17:21 ` [PATCH 0/2] docs: memory-hotplug: add " Jonathan Corbet 2 siblings, 1 reply; 7+ messages in thread From: Mike Rapoport @ 2018-10-11 4:58 UTC (permalink / raw) To: Jonathan Corbet Cc: David Hildenbrand, Andrew Morton, Stephen Rothwell, linux-doc, linux-kernel, Mike Rapoport From: Mike Rapoport <rppt@linux.vnet.ibm.com> to allow additions of new documentation about memory hotplug under the same roof. Signed-off-by: Mike Rapoport <rppt@linux.vnet.ibm.com> --- Documentation/core-api/index.rst | 2 +- Documentation/core-api/memory-hotplug-notifier.rst | 84 --------------------- Documentation/core-api/memory-hotplug.rst | 87 ++++++++++++++++++++++ 3 files changed, 88 insertions(+), 85 deletions(-) delete mode 100644 Documentation/core-api/memory-hotplug-notifier.rst create mode 100644 Documentation/core-api/memory-hotplug.rst diff --git a/Documentation/core-api/index.rst b/Documentation/core-api/index.rst index 4f8a426..29c790f 100644 --- a/Documentation/core-api/index.rst +++ b/Documentation/core-api/index.rst @@ -32,7 +32,7 @@ Core utilities gfp_mask-from-fs-io timekeeping boot-time-mm - memory-hotplug-notifier + memory-hotplug Interfaces for kernel debugging diff --git a/Documentation/core-api/memory-hotplug-notifier.rst b/Documentation/core-api/memory-hotplug-notifier.rst deleted file mode 100644 index 35347cc..0000000 --- a/Documentation/core-api/memory-hotplug-notifier.rst +++ /dev/null @@ -1,84 +0,0 @@ -.. _memory_hotplug_notifier: - -============================= -Memory hotplug event notifier -============================= - -Hotplugging events are sent to a notification queue. - -There are six types of notification defined in ``include/linux/memory.h``: - -MEM_GOING_ONLINE - Generated before new memory becomes available in order to be able to - prepare subsystems to handle memory. The page allocator is still unable - to allocate from the new memory. - -MEM_CANCEL_ONLINE - Generated if MEM_GOING_ONLINE fails. - -MEM_ONLINE - Generated when memory has successfully brought online. The callback may - allocate pages from the new memory. - -MEM_GOING_OFFLINE - Generated to begin the process of offlining memory. Allocations are no - longer possible from the memory but some of the memory to be offlined - is still in use. The callback can be used to free memory known to a - subsystem from the indicated memory block. - -MEM_CANCEL_OFFLINE - Generated if MEM_GOING_OFFLINE fails. Memory is available again from - the memory block that we attempted to offline. - -MEM_OFFLINE - Generated after offlining memory is complete. - -A callback routine can be registered by calling:: - - hotplug_memory_notifier(callback_func, priority) - -Callback functions with higher values of priority are called before callback -functions with lower values. - -A callback function must have the following prototype:: - - int callback_func( - struct notifier_block *self, unsigned long action, void *arg); - -The first argument of the callback function (self) is a pointer to the block -of the notifier chain that points to the callback function itself. -The second argument (action) is one of the event types described above. -The third argument (arg) passes a pointer of struct memory_notify:: - - struct memory_notify { - unsigned long start_pfn; - unsigned long nr_pages; - int status_change_nid_normal; - int status_change_nid_high; - int status_change_nid; - } - -- start_pfn is start_pfn of online/offline memory. -- nr_pages is # of pages of online/offline memory. -- status_change_nid_normal is set node id when N_NORMAL_MEMORY of nodemask - is (will be) set/clear, if this is -1, then nodemask status is not changed. -- status_change_nid_high is set node id when N_HIGH_MEMORY of nodemask - is (will be) set/clear, if this is -1, then nodemask status is not changed. -- status_change_nid is set node id when N_MEMORY of nodemask is (will be) - set/clear. It means a new(memoryless) node gets new memory by online and a - node loses all memory. If this is -1, then nodemask status is not changed. - - If status_changed_nid* >= 0, callback should create/discard structures for the - node if necessary. - -The callback routine shall return one of the values -NOTIFY_DONE, NOTIFY_OK, NOTIFY_BAD, NOTIFY_STOP -defined in ``include/linux/notifier.h`` - -NOTIFY_DONE and NOTIFY_OK have no effect on the further processing. - -NOTIFY_BAD is used as response to the MEM_GOING_ONLINE, MEM_GOING_OFFLINE, -MEM_ONLINE, or MEM_OFFLINE action to cancel hotplugging. It stops -further processing of the notification queue. - -NOTIFY_STOP stops further processing of the notification queue. diff --git a/Documentation/core-api/memory-hotplug.rst b/Documentation/core-api/memory-hotplug.rst new file mode 100644 index 0000000..a99f2f2 --- /dev/null +++ b/Documentation/core-api/memory-hotplug.rst @@ -0,0 +1,87 @@ +.. _memory_hotplug: + +============== +Memory hotplug +============== + +Memory hotplug event notifier +============================= + +Hotplugging events are sent to a notification queue. + +There are six types of notification defined in ``include/linux/memory.h``: + +MEM_GOING_ONLINE + Generated before new memory becomes available in order to be able to + prepare subsystems to handle memory. The page allocator is still unable + to allocate from the new memory. + +MEM_CANCEL_ONLINE + Generated if MEM_GOING_ONLINE fails. + +MEM_ONLINE + Generated when memory has successfully brought online. The callback may + allocate pages from the new memory. + +MEM_GOING_OFFLINE + Generated to begin the process of offlining memory. Allocations are no + longer possible from the memory but some of the memory to be offlined + is still in use. The callback can be used to free memory known to a + subsystem from the indicated memory block. + +MEM_CANCEL_OFFLINE + Generated if MEM_GOING_OFFLINE fails. Memory is available again from + the memory block that we attempted to offline. + +MEM_OFFLINE + Generated after offlining memory is complete. + +A callback routine can be registered by calling:: + + hotplug_memory_notifier(callback_func, priority) + +Callback functions with higher values of priority are called before callback +functions with lower values. + +A callback function must have the following prototype:: + + int callback_func( + struct notifier_block *self, unsigned long action, void *arg); + +The first argument of the callback function (self) is a pointer to the block +of the notifier chain that points to the callback function itself. +The second argument (action) is one of the event types described above. +The third argument (arg) passes a pointer of struct memory_notify:: + + struct memory_notify { + unsigned long start_pfn; + unsigned long nr_pages; + int status_change_nid_normal; + int status_change_nid_high; + int status_change_nid; + } + +- start_pfn is start_pfn of online/offline memory. +- nr_pages is # of pages of online/offline memory. +- status_change_nid_normal is set node id when N_NORMAL_MEMORY of nodemask + is (will be) set/clear, if this is -1, then nodemask status is not changed. +- status_change_nid_high is set node id when N_HIGH_MEMORY of nodemask + is (will be) set/clear, if this is -1, then nodemask status is not changed. +- status_change_nid is set node id when N_MEMORY of nodemask is (will be) + set/clear. It means a new(memoryless) node gets new memory by online and a + node loses all memory. If this is -1, then nodemask status is not changed. + + If status_changed_nid* >= 0, callback should create/discard structures for the + node if necessary. + +The callback routine shall return one of the values +NOTIFY_DONE, NOTIFY_OK, NOTIFY_BAD, NOTIFY_STOP +defined in ``include/linux/notifier.h`` + +NOTIFY_DONE and NOTIFY_OK have no effect on the further processing. + +NOTIFY_BAD is used as response to the MEM_GOING_ONLINE, MEM_GOING_OFFLINE, +MEM_ONLINE, or MEM_OFFLINE action to cancel hotplugging. It stops +further processing of the notification queue. + +NOTIFY_STOP stops further processing of the notification queue. -- 2.7.4 ^ permalink raw reply related [flat|nested] 7+ messages in thread
* Re: [PATCH 1/2] docs/core-api: rename memory-hotplug-notifier to memory-hotplug 2018-10-11 4:58 ` [PATCH 1/2] docs/core-api: rename memory-hotplug-notifier to memory-hotplug Mike Rapoport @ 2018-10-11 7:43 ` David Hildenbrand 0 siblings, 0 replies; 7+ messages in thread From: David Hildenbrand @ 2018-10-11 7:43 UTC (permalink / raw) To: Mike Rapoport, Jonathan Corbet Cc: Andrew Morton, Stephen Rothwell, linux-doc, linux-kernel, Mike Rapoport On 11/10/2018 06:58, Mike Rapoport wrote: > From: Mike Rapoport <rppt@linux.vnet.ibm.com> > > to allow additions of new documentation about memory hotplug under the same > roof. > > Signed-off-by: Mike Rapoport <rppt@linux.vnet.ibm.com> > --- > Documentation/core-api/index.rst | 2 +- > Documentation/core-api/memory-hotplug-notifier.rst | 84 --------------------- > Documentation/core-api/memory-hotplug.rst | 87 ++++++++++++++++++++++ > 3 files changed, 88 insertions(+), 85 deletions(-) > delete mode 100644 Documentation/core-api/memory-hotplug-notifier.rst > create mode 100644 Documentation/core-api/memory-hotplug.rst > > diff --git a/Documentation/core-api/index.rst b/Documentation/core-api/index.rst > index 4f8a426..29c790f 100644 > --- a/Documentation/core-api/index.rst > +++ b/Documentation/core-api/index.rst > @@ -32,7 +32,7 @@ Core utilities > gfp_mask-from-fs-io > timekeeping > boot-time-mm > - memory-hotplug-notifier > + memory-hotplug > > > Interfaces for kernel debugging > diff --git a/Documentation/core-api/memory-hotplug-notifier.rst b/Documentation/core-api/memory-hotplug-notifier.rst > deleted file mode 100644 > index 35347cc..0000000 > --- a/Documentation/core-api/memory-hotplug-notifier.rst > +++ /dev/null > @@ -1,84 +0,0 @@ > -.. _memory_hotplug_notifier: > - > -============================= > -Memory hotplug event notifier > -============================= > - > -Hotplugging events are sent to a notification queue. > - > -There are six types of notification defined in ``include/linux/memory.h``: > - > -MEM_GOING_ONLINE > - Generated before new memory becomes available in order to be able to > - prepare subsystems to handle memory. The page allocator is still unable > - to allocate from the new memory. > - > -MEM_CANCEL_ONLINE > - Generated if MEM_GOING_ONLINE fails. > - > -MEM_ONLINE > - Generated when memory has successfully brought online. The callback may > - allocate pages from the new memory. > - > -MEM_GOING_OFFLINE > - Generated to begin the process of offlining memory. Allocations are no > - longer possible from the memory but some of the memory to be offlined > - is still in use. The callback can be used to free memory known to a > - subsystem from the indicated memory block. > - > -MEM_CANCEL_OFFLINE > - Generated if MEM_GOING_OFFLINE fails. Memory is available again from > - the memory block that we attempted to offline. > - > -MEM_OFFLINE > - Generated after offlining memory is complete. > - > -A callback routine can be registered by calling:: > - > - hotplug_memory_notifier(callback_func, priority) > - > -Callback functions with higher values of priority are called before callback > -functions with lower values. > - > -A callback function must have the following prototype:: > - > - int callback_func( > - struct notifier_block *self, unsigned long action, void *arg); > - > -The first argument of the callback function (self) is a pointer to the block > -of the notifier chain that points to the callback function itself. > -The second argument (action) is one of the event types described above. > -The third argument (arg) passes a pointer of struct memory_notify:: > - > - struct memory_notify { > - unsigned long start_pfn; > - unsigned long nr_pages; > - int status_change_nid_normal; > - int status_change_nid_high; > - int status_change_nid; > - } > - > -- start_pfn is start_pfn of online/offline memory. > -- nr_pages is # of pages of online/offline memory. > -- status_change_nid_normal is set node id when N_NORMAL_MEMORY of nodemask > - is (will be) set/clear, if this is -1, then nodemask status is not changed. > -- status_change_nid_high is set node id when N_HIGH_MEMORY of nodemask > - is (will be) set/clear, if this is -1, then nodemask status is not changed. > -- status_change_nid is set node id when N_MEMORY of nodemask is (will be) > - set/clear. It means a new(memoryless) node gets new memory by online and a > - node loses all memory. If this is -1, then nodemask status is not changed. > - > - If status_changed_nid* >= 0, callback should create/discard structures for the > - node if necessary. > - > -The callback routine shall return one of the values > -NOTIFY_DONE, NOTIFY_OK, NOTIFY_BAD, NOTIFY_STOP > -defined in ``include/linux/notifier.h`` > - > -NOTIFY_DONE and NOTIFY_OK have no effect on the further processing. > - > -NOTIFY_BAD is used as response to the MEM_GOING_ONLINE, MEM_GOING_OFFLINE, > -MEM_ONLINE, or MEM_OFFLINE action to cancel hotplugging. It stops > -further processing of the notification queue. > - > -NOTIFY_STOP stops further processing of the notification queue. > diff --git a/Documentation/core-api/memory-hotplug.rst b/Documentation/core-api/memory-hotplug.rst > new file mode 100644 > index 0000000..a99f2f2 > --- /dev/null > +++ b/Documentation/core-api/memory-hotplug.rst > @@ -0,0 +1,87 @@ > +.. _memory_hotplug: > + > +============== > +Memory hotplug > +============== > + > +Memory hotplug event notifier > +============================= > + > +Hotplugging events are sent to a notification queue. > + > +There are six types of notification defined in ``include/linux/memory.h``: > + > +MEM_GOING_ONLINE > + Generated before new memory becomes available in order to be able to > + prepare subsystems to handle memory. The page allocator is still unable > + to allocate from the new memory. > + > +MEM_CANCEL_ONLINE > + Generated if MEM_GOING_ONLINE fails. > + > +MEM_ONLINE > + Generated when memory has successfully brought online. The callback may > + allocate pages from the new memory. > + > +MEM_GOING_OFFLINE > + Generated to begin the process of offlining memory. Allocations are no > + longer possible from the memory but some of the memory to be offlined > + is still in use. The callback can be used to free memory known to a > + subsystem from the indicated memory block. > + > +MEM_CANCEL_OFFLINE > + Generated if MEM_GOING_OFFLINE fails. Memory is available again from > + the memory block that we attempted to offline. > + > +MEM_OFFLINE > + Generated after offlining memory is complete. > + > +A callback routine can be registered by calling:: > + > + hotplug_memory_notifier(callback_func, priority) > + > +Callback functions with higher values of priority are called before callback > +functions with lower values. > + > +A callback function must have the following prototype:: > + > + int callback_func( > + struct notifier_block *self, unsigned long action, void *arg); > + > +The first argument of the callback function (self) is a pointer to the block > +of the notifier chain that points to the callback function itself. > +The second argument (action) is one of the event types described above. > +The third argument (arg) passes a pointer of struct memory_notify:: > + > + struct memory_notify { > + unsigned long start_pfn; > + unsigned long nr_pages; > + int status_change_nid_normal; > + int status_change_nid_high; > + int status_change_nid; > + } > + > +- start_pfn is start_pfn of online/offline memory. > +- nr_pages is # of pages of online/offline memory. > +- status_change_nid_normal is set node id when N_NORMAL_MEMORY of nodemask > + is (will be) set/clear, if this is -1, then nodemask status is not changed. > +- status_change_nid_high is set node id when N_HIGH_MEMORY of nodemask > + is (will be) set/clear, if this is -1, then nodemask status is not changed. > +- status_change_nid is set node id when N_MEMORY of nodemask is (will be) > + set/clear. It means a new(memoryless) node gets new memory by online and a > + node loses all memory. If this is -1, then nodemask status is not changed. > + > + If status_changed_nid* >= 0, callback should create/discard structures for the > + node if necessary. > + > +The callback routine shall return one of the values > +NOTIFY_DONE, NOTIFY_OK, NOTIFY_BAD, NOTIFY_STOP > +defined in ``include/linux/notifier.h`` > + > +NOTIFY_DONE and NOTIFY_OK have no effect on the further processing. > + > +NOTIFY_BAD is used as response to the MEM_GOING_ONLINE, MEM_GOING_OFFLINE, > +MEM_ONLINE, or MEM_OFFLINE action to cancel hotplugging. It stops > +further processing of the notification queue. > + > +NOTIFY_STOP stops further processing of the notification queue. > Reviewed-by: David Hildenbrand <david@redhat.com> -- Thanks, David / dhildenb ^ permalink raw reply [flat|nested] 7+ messages in thread
* [PATCH 2/2] docs/core-api: memory-hotplug: add some details about locking internals 2018-10-11 4:58 [PATCH 0/2] docs: memory-hotplug: add details about locking internals Mike Rapoport 2018-10-11 4:58 ` [PATCH 1/2] docs/core-api: rename memory-hotplug-notifier to memory-hotplug Mike Rapoport @ 2018-10-11 4:58 ` Mike Rapoport 2018-10-11 7:43 ` David Hildenbrand 2018-10-12 17:21 ` [PATCH 0/2] docs: memory-hotplug: add " Jonathan Corbet 2 siblings, 1 reply; 7+ messages in thread From: Mike Rapoport @ 2018-10-11 4:58 UTC (permalink / raw) To: Jonathan Corbet Cc: David Hildenbrand, Andrew Morton, Stephen Rothwell, linux-doc, linux-kernel, Mike Rapoport From: David Hildenbrand <david@redhat.com> Let's document the magic a bit, especially why device_hotplug_lock is required when adding/removing memory and how it all play together with requests to online/offline memory from user space. [ rppt: moved the text to Documentation/core-api/memory-hotplug.rst ] Link: http://lkml.kernel.org/r/20180925091457.28651-7-david@redhat.com Signed-off-by: David Hildenbrand <david@redhat.com> Reviewed-by: Pavel Tatashin <pavel.tatashin@microsoft.com> Reviewed-by: Rashmica Gupta <rashmica.g@gmail.com> Cc: Jonathan Corbet <corbet@lwn.net> Cc: Michal Hocko <mhocko@suse.com> Cc: Balbir Singh <bsingharora@gmail.com> Cc: Benjamin Herrenschmidt <benh@kernel.crashing.org> Cc: Boris Ostrovsky <boris.ostrovsky@oracle.com> Cc: Dan Williams <dan.j.williams@intel.com> Cc: Greg Kroah-Hartman <gregkh@linuxfoundation.org> Cc: Haiyang Zhang <haiyangz@microsoft.com> Cc: Heiko Carstens <heiko.carstens@de.ibm.com> Cc: John Allen <jallen@linux.vnet.ibm.com> Cc: Joonsoo Kim <iamjoonsoo.kim@lge.com> Cc: Juergen Gross <jgross@suse.com> Cc: Kate Stewart <kstewart@linuxfoundation.org> Cc: "K. Y. Srinivasan" <kys@microsoft.com> Cc: Len Brown <lenb@kernel.org> Cc: Martin Schwidefsky <schwidefsky@de.ibm.com> Cc: Mathieu Malaterre <malat@debian.org> Cc: Michael Ellerman <mpe@ellerman.id.au> Cc: Michael Neuling <mikey@neuling.org> Cc: Nathan Fontenot <nfont@linux.vnet.ibm.com> Cc: Oscar Salvador <osalvador@suse.de> Cc: Paul Mackerras <paulus@samba.org> Cc: Philippe Ombredanne <pombredanne@nexb.com> Cc: Rafael J. Wysocki <rafael.j.wysocki@intel.com> Cc: "Rafael J. Wysocki" <rjw@rjwysocki.net> Cc: Stephen Hemminger <sthemmin@microsoft.com> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Vlastimil Babka <vbabka@suse.cz> Cc: YASUAKI ISHIMATSU <yasu.isimatu@gmail.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> Signed-off-by: Mike Rapoport <rppt@linux.ibm.com> --- Documentation/core-api/memory-hotplug.rst | 38 +++++++++++++++++++++++++++++++ 1 file changed, 38 insertions(+) diff --git a/Documentation/core-api/memory-hotplug.rst b/Documentation/core-api/memory-hotplug.rst index a99f2f2..de7467e 100644 --- a/Documentation/core-api/memory-hotplug.rst +++ b/Documentation/core-api/memory-hotplug.rst @@ -85,3 +85,41 @@ MEM_ONLINE, or MEM_OFFLINE action to cancel hotplugging. It stops further processing of the notification queue. NOTIFY_STOP stops further processing of the notification queue. + +Locking Internals +================= + +When adding/removing memory that uses memory block devices (i.e. ordinary RAM), +the device_hotplug_lock should be held to: + +- synchronize against online/offline requests (e.g. via sysfs). This way, memory + block devices can only be accessed (.online/.state attributes) by user + space once memory has been fully added. And when removing memory, we + know nobody is in critical sections. +- synchronize against CPU hotplug and similar (e.g. relevant for ACPI and PPC) + +Especially, there is a possible lock inversion that is avoided using +device_hotplug_lock when adding memory and user space tries to online that +memory faster than expected: + +- device_online() will first take the device_lock(), followed by + mem_hotplug_lock +- add_memory_resource() will first take the mem_hotplug_lock, followed by + the device_lock() (while creating the devices, during bus_add_device()). + +As the device is visible to user space before taking the device_lock(), this +can result in a lock inversion. + +onlining/offlining of memory should be done via device_online()/ +device_offline() - to make sure it is properly synchronized to actions +via sysfs. Holding device_hotplug_lock is advised (to e.g. protect online_type) + +When adding/removing/onlining/offlining memory or adding/removing +heterogeneous/device memory, we should always hold the mem_hotplug_lock in +write mode to serialise memory hotplug (e.g. access to global/zone +variables). + +In addition, mem_hotplug_lock (in contrast to device_hotplug_lock) in read +mode allows for a quite efficient get_online_mems/put_online_mems +implementation, so code accessing memory can protect from that memory +vanishing. -- 2.7.4 ^ permalink raw reply related [flat|nested] 7+ messages in thread
* Re: [PATCH 2/2] docs/core-api: memory-hotplug: add some details about locking internals 2018-10-11 4:58 ` [PATCH 2/2] docs/core-api: memory-hotplug: add some details about locking internals Mike Rapoport @ 2018-10-11 7:43 ` David Hildenbrand 0 siblings, 0 replies; 7+ messages in thread From: David Hildenbrand @ 2018-10-11 7:43 UTC (permalink / raw) To: Mike Rapoport, Jonathan Corbet Cc: Andrew Morton, Stephen Rothwell, linux-doc, linux-kernel On 11/10/2018 06:58, Mike Rapoport wrote: > From: David Hildenbrand <david@redhat.com> > > Let's document the magic a bit, especially why device_hotplug_lock is > required when adding/removing memory and how it all play together with > requests to online/offline memory from user space. > > [ rppt: moved the text to Documentation/core-api/memory-hotplug.rst ] > > Link: http://lkml.kernel.org/r/20180925091457.28651-7-david@redhat.com > Signed-off-by: David Hildenbrand <david@redhat.com> > Reviewed-by: Pavel Tatashin <pavel.tatashin@microsoft.com> > Reviewed-by: Rashmica Gupta <rashmica.g@gmail.com> > Cc: Jonathan Corbet <corbet@lwn.net> > Cc: Michal Hocko <mhocko@suse.com> > Cc: Balbir Singh <bsingharora@gmail.com> > Cc: Benjamin Herrenschmidt <benh@kernel.crashing.org> > Cc: Boris Ostrovsky <boris.ostrovsky@oracle.com> > Cc: Dan Williams <dan.j.williams@intel.com> > Cc: Greg Kroah-Hartman <gregkh@linuxfoundation.org> > Cc: Haiyang Zhang <haiyangz@microsoft.com> > Cc: Heiko Carstens <heiko.carstens@de.ibm.com> > Cc: John Allen <jallen@linux.vnet.ibm.com> > Cc: Joonsoo Kim <iamjoonsoo.kim@lge.com> > Cc: Juergen Gross <jgross@suse.com> > Cc: Kate Stewart <kstewart@linuxfoundation.org> > Cc: "K. Y. Srinivasan" <kys@microsoft.com> > Cc: Len Brown <lenb@kernel.org> > Cc: Martin Schwidefsky <schwidefsky@de.ibm.com> > Cc: Mathieu Malaterre <malat@debian.org> > Cc: Michael Ellerman <mpe@ellerman.id.au> > Cc: Michael Neuling <mikey@neuling.org> > Cc: Nathan Fontenot <nfont@linux.vnet.ibm.com> > Cc: Oscar Salvador <osalvador@suse.de> > Cc: Paul Mackerras <paulus@samba.org> > Cc: Philippe Ombredanne <pombredanne@nexb.com> > Cc: Rafael J. Wysocki <rafael.j.wysocki@intel.com> > Cc: "Rafael J. Wysocki" <rjw@rjwysocki.net> > Cc: Stephen Hemminger <sthemmin@microsoft.com> > Cc: Thomas Gleixner <tglx@linutronix.de> > Cc: Vlastimil Babka <vbabka@suse.cz> > Cc: YASUAKI ISHIMATSU <yasu.isimatu@gmail.com> > Signed-off-by: Andrew Morton <akpm@linux-foundation.org> > Signed-off-by: Mike Rapoport <rppt@linux.ibm.com> > --- > Documentation/core-api/memory-hotplug.rst | 38 +++++++++++++++++++++++++++++++ > 1 file changed, 38 insertions(+) > > diff --git a/Documentation/core-api/memory-hotplug.rst b/Documentation/core-api/memory-hotplug.rst > index a99f2f2..de7467e 100644 > --- a/Documentation/core-api/memory-hotplug.rst > +++ b/Documentation/core-api/memory-hotplug.rst > @@ -85,3 +85,41 @@ MEM_ONLINE, or MEM_OFFLINE action to cancel hotplugging. It stops > further processing of the notification queue. > > NOTIFY_STOP stops further processing of the notification queue. > + > +Locking Internals > +================= > + > +When adding/removing memory that uses memory block devices (i.e. ordinary RAM), > +the device_hotplug_lock should be held to: > + > +- synchronize against online/offline requests (e.g. via sysfs). This way, memory > + block devices can only be accessed (.online/.state attributes) by user > + space once memory has been fully added. And when removing memory, we > + know nobody is in critical sections. > +- synchronize against CPU hotplug and similar (e.g. relevant for ACPI and PPC) > + > +Especially, there is a possible lock inversion that is avoided using > +device_hotplug_lock when adding memory and user space tries to online that > +memory faster than expected: > + > +- device_online() will first take the device_lock(), followed by > + mem_hotplug_lock > +- add_memory_resource() will first take the mem_hotplug_lock, followed by > + the device_lock() (while creating the devices, during bus_add_device()). > + > +As the device is visible to user space before taking the device_lock(), this > +can result in a lock inversion. > + > +onlining/offlining of memory should be done via device_online()/ > +device_offline() - to make sure it is properly synchronized to actions > +via sysfs. Holding device_hotplug_lock is advised (to e.g. protect online_type) > + > +When adding/removing/onlining/offlining memory or adding/removing > +heterogeneous/device memory, we should always hold the mem_hotplug_lock in > +write mode to serialise memory hotplug (e.g. access to global/zone > +variables). > + > +In addition, mem_hotplug_lock (in contrast to device_hotplug_lock) in read > +mode allows for a quite efficient get_online_mems/put_online_mems > +implementation, so code accessing memory can protect from that memory > +vanishing. > Looks good to me. -- Thanks, David / dhildenb ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [PATCH 0/2] docs: memory-hotplug: add details about locking internals 2018-10-11 4:58 [PATCH 0/2] docs: memory-hotplug: add details about locking internals Mike Rapoport 2018-10-11 4:58 ` [PATCH 1/2] docs/core-api: rename memory-hotplug-notifier to memory-hotplug Mike Rapoport 2018-10-11 4:58 ` [PATCH 2/2] docs/core-api: memory-hotplug: add some details about locking internals Mike Rapoport @ 2018-10-12 17:21 ` Jonathan Corbet 2018-12-03 11:23 ` David Hildenbrand 2 siblings, 1 reply; 7+ messages in thread From: Jonathan Corbet @ 2018-10-12 17:21 UTC (permalink / raw) To: Mike Rapoport Cc: David Hildenbrand, Andrew Morton, Stephen Rothwell, linux-doc, linux-kernel On Thu, 11 Oct 2018 07:58:15 +0300 Mike Rapoport <rppt@linux.ibm.com> wrote: > As discussed at [1], the latest updates to memory hotplug documentation are > causing a conflict between docs and mmotm trees. > These patches resolve the conflict. > > [1] https://lkml.org/lkml/2018/10/8/227 > > David Hildenbrand (1): > docs/core-api: memory-hotplug: add some details about locking internals > > Mike Rapoport (1): > docs/core-api: rename memory-hotplug-notifier to memory-hotplug I've applied the pair, thanks. jon ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [PATCH 0/2] docs: memory-hotplug: add details about locking internals 2018-10-12 17:21 ` [PATCH 0/2] docs: memory-hotplug: add " Jonathan Corbet @ 2018-12-03 11:23 ` David Hildenbrand 0 siblings, 0 replies; 7+ messages in thread From: David Hildenbrand @ 2018-12-03 11:23 UTC (permalink / raw) To: Jonathan Corbet, Mike Rapoport Cc: Andrew Morton, Stephen Rothwell, linux-doc, linux-kernel On 12.10.18 19:21, Jonathan Corbet wrote: > On Thu, 11 Oct 2018 07:58:15 +0300 > Mike Rapoport <rppt@linux.ibm.com> wrote: > >> As discussed at [1], the latest updates to memory hotplug documentation are >> causing a conflict between docs and mmotm trees. >> These patches resolve the conflict. >> >> [1] https://lkml.org/lkml/2018/10/8/227 >> >> David Hildenbrand (1): >> docs/core-api: memory-hotplug: add some details about locking internals >> >> Mike Rapoport (1): >> docs/core-api: rename memory-hotplug-notifier to memory-hotplug > > I've applied the pair, thanks. > > jon > Looking at linux-next, we now have duplicate documentation: $ git grep mem_hotplug_lock Documentation/admin-guide/mm/memory-hotplug.rst: mem_hotplug_lock Documentation/admin-guide/mm/memory-hotplug.rst:- add_memory_resource() will first take the mem_hotplug_lock, followed by Documentation/admin-guide/mm/memory-hotplug.rst:heterogeneous/device memory, we should always hold the mem_hotplug_lock in Documentation/admin-guide/mm/memory-hotplug.rst:In addition, mem_hotplug_lock (in contrast to device_hotplug_lock) in read Documentation/core-api/memory-hotplug.rst: mem_hotplug_lock Documentation/core-api/memory-hotplug.rst:- add_memory_resource() will first take the mem_hotplug_lock, followed by Documentation/core-api/memory-hotplug.rst:heterogeneous/device memory, we should always hold the mem_hotplug_lock in Documentation/core-api/memory-hotplug.rst:In addition, mem_hotplug_lock (in contrast to device_hotplug_lock) in read It really only should go to Documentation/core-api/memory-hotplug.rst Should I send a patch or who can fix that up? -- Thanks, David / dhildenb ^ permalink raw reply [flat|nested] 7+ messages in thread
end of thread, other threads:[~2018-12-03 11:23 UTC | newest] Thread overview: 7+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2018-10-11 4:58 [PATCH 0/2] docs: memory-hotplug: add details about locking internals Mike Rapoport 2018-10-11 4:58 ` [PATCH 1/2] docs/core-api: rename memory-hotplug-notifier to memory-hotplug Mike Rapoport 2018-10-11 7:43 ` David Hildenbrand 2018-10-11 4:58 ` [PATCH 2/2] docs/core-api: memory-hotplug: add some details about locking internals Mike Rapoport 2018-10-11 7:43 ` David Hildenbrand 2018-10-12 17:21 ` [PATCH 0/2] docs: memory-hotplug: add " Jonathan Corbet 2018-12-03 11:23 ` David Hildenbrand
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.