* [PATCH v5] mm/memory_hotplug: Fix remove_memory() lockdep splat
@ 2020-01-25 1:07 Dan Williams
2020-01-27 7:44 ` David Hildenbrand
2020-01-27 13:41 ` Michal Hocko
0 siblings, 2 replies; 3+ messages in thread
From: Dan Williams @ 2020-01-25 1:07 UTC (permalink / raw)
To: akpm
Cc: Vishal Verma, David Hildenbrand, Pavel Tatashin, Michal Hocko,
Dave Hansen, linux-mm, linux-kernel
The daxctl unit test for the dax_kmem driver currently triggers the
(false positive) lockdep splat below. It results from the fact that
remove_memory_block_devices() is invoked under the mem_hotplug_lock()
causing lockdep entanglements with cpu_hotplug_lock() and sysfs (kernfs
active state tracking). It is a false positive because the sysfs
attribute path triggering the memory remove is not the same attribute
path associated with memory-block device.
sysfs_break_active_protection() is not applicable since there is no real
deadlock conflict, instead move memory-block device removal outside the
lock. The mem_hotplug_lock() is not needed to synchronize the
memory-block device removal vs the page online state, that is already
handled by lock_device_hotplug(). Specifically, lock_device_hotplug() is
sufficient to allow try_remove_memory() to check the offline state of
the memblocks and be assured that any in progress online attempts are
flushed / blocked by kernfs_drain() / attribute removal.
The add_memory() path safely creates memblock devices under the
mem_hotplug_lock(). There is no kernfs active state synchronization in
the memblock device_register() path, so nothing to fix there.
This change is only possible thanks to the recent change that refactored
memory block device removal out of arch_remove_memory() (commit
4c4b7f9ba948 mm/memory_hotplug: remove memory block devices before
arch_remove_memory()), and David's due diligence tracking down the
guarantees afforded by kernfs_drain(). Not flagged for -stable since
this only impacts ongoing development and lockdep validation, not a
runtime issue.
======================================================
WARNING: possible circular locking dependency detected
5.5.0-rc3+ #230 Tainted: G OE
------------------------------------------------------
lt-daxctl/6459 is trying to acquire lock:
ffff99c7f0003510 (kn->count#241){++++}, at: kernfs_remove_by_name_ns+0x41/0x80
but task is already holding lock:
ffffffffa76a5450 (mem_hotplug_lock.rw_sem){++++}, at: percpu_down_write+0x20/0xe0
which lock already depends on the new lock.
the existing dependency chain (in reverse order) is:
-> #2 (mem_hotplug_lock.rw_sem){++++}:
__lock_acquire+0x39c/0x790
lock_acquire+0xa2/0x1b0
get_online_mems+0x3e/0xb0
kmem_cache_create_usercopy+0x2e/0x260
kmem_cache_create+0x12/0x20
ptlock_cache_init+0x20/0x28
start_kernel+0x243/0x547
secondary_startup_64+0xb6/0xc0
-> #1 (cpu_hotplug_lock.rw_sem){++++}:
__lock_acquire+0x39c/0x790
lock_acquire+0xa2/0x1b0
cpus_read_lock+0x3e/0xb0
online_pages+0x37/0x300
memory_subsys_online+0x17d/0x1c0
device_online+0x60/0x80
state_store+0x65/0xd0
kernfs_fop_write+0xcf/0x1c0
vfs_write+0xdb/0x1d0
ksys_write+0x65/0xe0
do_syscall_64+0x5c/0xa0
entry_SYSCALL_64_after_hwframe+0x49/0xbe
-> #0 (kn->count#241){++++}:
check_prev_add+0x98/0xa40
validate_chain+0x576/0x860
__lock_acquire+0x39c/0x790
lock_acquire+0xa2/0x1b0
__kernfs_remove+0x25f/0x2e0
kernfs_remove_by_name_ns+0x41/0x80
remove_files.isra.0+0x30/0x70
sysfs_remove_group+0x3d/0x80
sysfs_remove_groups+0x29/0x40
device_remove_attrs+0x39/0x70
device_del+0x16a/0x3f0
device_unregister+0x16/0x60
remove_memory_block_devices+0x82/0xb0
try_remove_memory+0xb5/0x130
remove_memory+0x26/0x40
dev_dax_kmem_remove+0x44/0x6a [kmem]
device_release_driver_internal+0xe4/0x1c0
unbind_store+0xef/0x120
kernfs_fop_write+0xcf/0x1c0
vfs_write+0xdb/0x1d0
ksys_write+0x65/0xe0
do_syscall_64+0x5c/0xa0
entry_SYSCALL_64_after_hwframe+0x49/0xbe
other info that might help us debug this:
Chain exists of:
kn->count#241 --> cpu_hotplug_lock.rw_sem --> mem_hotplug_lock.rw_sem
Possible unsafe locking scenario:
CPU0 CPU1
---- ----
lock(mem_hotplug_lock.rw_sem);
lock(cpu_hotplug_lock.rw_sem);
lock(mem_hotplug_lock.rw_sem);
lock(kn->count#241);
*** DEADLOCK ***
No fixes tag as this has been a long standing issue that predated the
addition of kernfs lockdep annotations.
Cc: Vishal Verma <vishal.l.verma@intel.com>
Cc: David Hildenbrand <david@redhat.com>
Cc: Pavel Tatashin <pasha.tatashin@soleen.com>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Dave Hansen <dave.hansen@linux.intel.com>
Signed-off-by: Dan Williams <dan.j.williams@intel.com>
---
Changes since v4 [1]:
- Drop the unnecessary consideration of mem->section_count.
kernfs_drain() + lock_device_hotplug() is sufficient protection
(David)
[1]: http://lore.kernel.org/r/157869128062.2451572.4093315441083744888.stgit@dwillia2-desk3.amr.corp.intel.com
mm/memory_hotplug.c | 9 ++++++---
1 file changed, 6 insertions(+), 3 deletions(-)
diff --git a/mm/memory_hotplug.c b/mm/memory_hotplug.c
index 55ac23ef11c1..65ddaf3a2a12 100644
--- a/mm/memory_hotplug.c
+++ b/mm/memory_hotplug.c
@@ -1763,8 +1763,6 @@ static int __ref try_remove_memory(int nid, u64 start, u64 size)
BUG_ON(check_hotplug_memory_range(start, size));
- mem_hotplug_begin();
-
/*
* All memory blocks must be offlined before removing memory. Check
* whether all memory blocks in question are offline and return error
@@ -1777,9 +1775,14 @@ static int __ref try_remove_memory(int nid, u64 start, u64 size)
/* remove memmap entry */
firmware_map_remove(start, start + size, "System RAM");
- /* remove memory block devices before removing memory */
+ /*
+ * Memory block device removal under the device_hotplug_lock is
+ * a barrier against racing online attempts.
+ */
remove_memory_block_devices(start, size);
+ mem_hotplug_begin();
+
arch_remove_memory(nid, start, size, NULL);
memblock_free(start, size);
memblock_remove(start, size);
^ permalink raw reply related [flat|nested] 3+ messages in thread
* Re: [PATCH v5] mm/memory_hotplug: Fix remove_memory() lockdep splat
2020-01-25 1:07 [PATCH v5] mm/memory_hotplug: Fix remove_memory() lockdep splat Dan Williams
@ 2020-01-27 7:44 ` David Hildenbrand
2020-01-27 13:41 ` Michal Hocko
1 sibling, 0 replies; 3+ messages in thread
From: David Hildenbrand @ 2020-01-27 7:44 UTC (permalink / raw)
To: Dan Williams
Cc: akpm, Vishal Verma, David Hildenbrand, Pavel Tatashin,
Michal Hocko, Dave Hansen, linux-mm, linux-kernel
> Am 25.01.2020 um 02:23 schrieb Dan Williams <dan.j.williams@intel.com>:
>
> The daxctl unit test for the dax_kmem driver currently triggers the
> (false positive) lockdep splat below. It results from the fact that
> remove_memory_block_devices() is invoked under the mem_hotplug_lock()
> causing lockdep entanglements with cpu_hotplug_lock() and sysfs (kernfs
> active state tracking). It is a false positive because the sysfs
> attribute path triggering the memory remove is not the same attribute
> path associated with memory-block device.
>
> sysfs_break_active_protection() is not applicable since there is no real
> deadlock conflict, instead move memory-block device removal outside the
> lock. The mem_hotplug_lock() is not needed to synchronize the
> memory-block device removal vs the page online state, that is already
> handled by lock_device_hotplug(). Specifically, lock_device_hotplug() is
> sufficient to allow try_remove_memory() to check the offline state of
> the memblocks and be assured that any in progress online attempts are
> flushed / blocked by kernfs_drain() / attribute removal.
>
> The add_memory() path safely creates memblock devices under the
> mem_hotplug_lock(). There is no kernfs active state synchronization in
> the memblock device_register() path, so nothing to fix there.
>
> This change is only possible thanks to the recent change that refactored
> memory block device removal out of arch_remove_memory() (commit
> 4c4b7f9ba948 mm/memory_hotplug: remove memory block devices before
> arch_remove_memory()), and David's due diligence tracking down the
> guarantees afforded by kernfs_drain(). Not flagged for -stable since
> this only impacts ongoing development and lockdep validation, not a
> runtime issue.
>
> ======================================================
> WARNING: possible circular locking dependency detected
> 5.5.0-rc3+ #230 Tainted: G OE
> ------------------------------------------------------
> lt-daxctl/6459 is trying to acquire lock:
> ffff99c7f0003510 (kn->count#241){++++}, at: kernfs_remove_by_name_ns+0x41/0x80
>
> but task is already holding lock:
> ffffffffa76a5450 (mem_hotplug_lock.rw_sem){++++}, at: percpu_down_write+0x20/0xe0
>
> which lock already depends on the new lock.
>
>
> the existing dependency chain (in reverse order) is:
>
> -> #2 (mem_hotplug_lock.rw_sem){++++}:
> __lock_acquire+0x39c/0x790
> lock_acquire+0xa2/0x1b0
> get_online_mems+0x3e/0xb0
> kmem_cache_create_usercopy+0x2e/0x260
> kmem_cache_create+0x12/0x20
> ptlock_cache_init+0x20/0x28
> start_kernel+0x243/0x547
> secondary_startup_64+0xb6/0xc0
>
> -> #1 (cpu_hotplug_lock.rw_sem){++++}:
> __lock_acquire+0x39c/0x790
> lock_acquire+0xa2/0x1b0
> cpus_read_lock+0x3e/0xb0
> online_pages+0x37/0x300
> memory_subsys_online+0x17d/0x1c0
> device_online+0x60/0x80
> state_store+0x65/0xd0
> kernfs_fop_write+0xcf/0x1c0
> vfs_write+0xdb/0x1d0
> ksys_write+0x65/0xe0
> do_syscall_64+0x5c/0xa0
> entry_SYSCALL_64_after_hwframe+0x49/0xbe
>
> -> #0 (kn->count#241){++++}:
> check_prev_add+0x98/0xa40
> validate_chain+0x576/0x860
> __lock_acquire+0x39c/0x790
> lock_acquire+0xa2/0x1b0
> __kernfs_remove+0x25f/0x2e0
> kernfs_remove_by_name_ns+0x41/0x80
> remove_files.isra.0+0x30/0x70
> sysfs_remove_group+0x3d/0x80
> sysfs_remove_groups+0x29/0x40
> device_remove_attrs+0x39/0x70
> device_del+0x16a/0x3f0
> device_unregister+0x16/0x60
> remove_memory_block_devices+0x82/0xb0
> try_remove_memory+0xb5/0x130
> remove_memory+0x26/0x40
> dev_dax_kmem_remove+0x44/0x6a [kmem]
> device_release_driver_internal+0xe4/0x1c0
> unbind_store+0xef/0x120
> kernfs_fop_write+0xcf/0x1c0
> vfs_write+0xdb/0x1d0
> ksys_write+0x65/0xe0
> do_syscall_64+0x5c/0xa0
> entry_SYSCALL_64_after_hwframe+0x49/0xbe
>
> other info that might help us debug this:
>
> Chain exists of:
> kn->count#241 --> cpu_hotplug_lock.rw_sem --> mem_hotplug_lock.rw_sem
>
> Possible unsafe locking scenario:
>
> CPU0 CPU1
> ---- ----
> lock(mem_hotplug_lock.rw_sem);
> lock(cpu_hotplug_lock.rw_sem);
> lock(mem_hotplug_lock.rw_sem);
> lock(kn->count#241);
>
> *** DEADLOCK ***
>
> No fixes tag as this has been a long standing issue that predated the
> addition of kernfs lockdep annotations.
>
> Cc: Vishal Verma <vishal.l.verma@intel.com>
> Cc: David Hildenbrand <david@redhat.com>
> Cc: Pavel Tatashin <pasha.tatashin@soleen.com>
> Cc: Michal Hocko <mhocko@suse.com>
> Cc: Dave Hansen <dave.hansen@linux.intel.com>
> Signed-off-by: Dan Williams <dan.j.williams@intel.com>
> ---
> Changes since v4 [1]:
> - Drop the unnecessary consideration of mem->section_count.
> kernfs_drain() + lock_device_hotplug() is sufficient protection
> (David)
>
> [1]: http://lore.kernel.org/r/157869128062.2451572.4093315441083744888.stgit@dwillia2-desk3.amr.corp.intel.com
>
> mm/memory_hotplug.c | 9 ++++++---
> 1 file changed, 6 insertions(+), 3 deletions(-)
>
> diff --git a/mm/memory_hotplug.c b/mm/memory_hotplug.c
> index 55ac23ef11c1..65ddaf3a2a12 100644
> --- a/mm/memory_hotplug.c
> +++ b/mm/memory_hotplug.c
> @@ -1763,8 +1763,6 @@ static int __ref try_remove_memory(int nid, u64 start, u64 size)
>
> BUG_ON(check_hotplug_memory_range(start, size));
>
> - mem_hotplug_begin();
> -
> /*
> * All memory blocks must be offlined before removing memory. Check
> * whether all memory blocks in question are offline and return error
> @@ -1777,9 +1775,14 @@ static int __ref try_remove_memory(int nid, u64 start, u64 size)
> /* remove memmap entry */
> firmware_map_remove(start, start + size, "System RAM");
>
> - /* remove memory block devices before removing memory */
> + /*
> + * Memory block device removal under the device_hotplug_lock is
> + * a barrier against racing online attempts.
> + */
> remove_memory_block_devices(start, size);
>
> + mem_hotplug_begin();
> +
> arch_remove_memory(nid, start, size, NULL);
> memblock_free(start, size);
> memblock_remove(start, size);
>
>
Thanks!
Reviewed-by: David Hildenbrand <david@redhat.com>
^ permalink raw reply [flat|nested] 3+ messages in thread
* Re: [PATCH v5] mm/memory_hotplug: Fix remove_memory() lockdep splat
2020-01-25 1:07 [PATCH v5] mm/memory_hotplug: Fix remove_memory() lockdep splat Dan Williams
2020-01-27 7:44 ` David Hildenbrand
@ 2020-01-27 13:41 ` Michal Hocko
1 sibling, 0 replies; 3+ messages in thread
From: Michal Hocko @ 2020-01-27 13:41 UTC (permalink / raw)
To: Dan Williams
Cc: akpm, Vishal Verma, David Hildenbrand, Pavel Tatashin,
Dave Hansen, linux-mm, linux-kernel
On Fri 24-01-20 17:07:21, Dan Williams wrote:
> The daxctl unit test for the dax_kmem driver currently triggers the
> (false positive) lockdep splat below. It results from the fact that
> remove_memory_block_devices() is invoked under the mem_hotplug_lock()
> causing lockdep entanglements with cpu_hotplug_lock() and sysfs (kernfs
> active state tracking). It is a false positive because the sysfs
> attribute path triggering the memory remove is not the same attribute
> path associated with memory-block device.
>
> sysfs_break_active_protection() is not applicable since there is no real
> deadlock conflict, instead move memory-block device removal outside the
> lock. The mem_hotplug_lock() is not needed to synchronize the
> memory-block device removal vs the page online state, that is already
> handled by lock_device_hotplug(). Specifically, lock_device_hotplug() is
> sufficient to allow try_remove_memory() to check the offline state of
> the memblocks and be assured that any in progress online attempts are
> flushed / blocked by kernfs_drain() / attribute removal.
>
> The add_memory() path safely creates memblock devices under the
> mem_hotplug_lock(). There is no kernfs active state synchronization in
> the memblock device_register() path, so nothing to fix there.
>
> This change is only possible thanks to the recent change that refactored
> memory block device removal out of arch_remove_memory() (commit
> 4c4b7f9ba948 mm/memory_hotplug: remove memory block devices before
> arch_remove_memory()), and David's due diligence tracking down the
> guarantees afforded by kernfs_drain(). Not flagged for -stable since
> this only impacts ongoing development and lockdep validation, not a
> runtime issue.
>
> ======================================================
> WARNING: possible circular locking dependency detected
> 5.5.0-rc3+ #230 Tainted: G OE
> ------------------------------------------------------
> lt-daxctl/6459 is trying to acquire lock:
> ffff99c7f0003510 (kn->count#241){++++}, at: kernfs_remove_by_name_ns+0x41/0x80
>
> but task is already holding lock:
> ffffffffa76a5450 (mem_hotplug_lock.rw_sem){++++}, at: percpu_down_write+0x20/0xe0
>
> which lock already depends on the new lock.
>
>
> the existing dependency chain (in reverse order) is:
>
> -> #2 (mem_hotplug_lock.rw_sem){++++}:
> __lock_acquire+0x39c/0x790
> lock_acquire+0xa2/0x1b0
> get_online_mems+0x3e/0xb0
> kmem_cache_create_usercopy+0x2e/0x260
> kmem_cache_create+0x12/0x20
> ptlock_cache_init+0x20/0x28
> start_kernel+0x243/0x547
> secondary_startup_64+0xb6/0xc0
>
> -> #1 (cpu_hotplug_lock.rw_sem){++++}:
> __lock_acquire+0x39c/0x790
> lock_acquire+0xa2/0x1b0
> cpus_read_lock+0x3e/0xb0
> online_pages+0x37/0x300
> memory_subsys_online+0x17d/0x1c0
> device_online+0x60/0x80
> state_store+0x65/0xd0
> kernfs_fop_write+0xcf/0x1c0
> vfs_write+0xdb/0x1d0
> ksys_write+0x65/0xe0
> do_syscall_64+0x5c/0xa0
> entry_SYSCALL_64_after_hwframe+0x49/0xbe
>
> -> #0 (kn->count#241){++++}:
> check_prev_add+0x98/0xa40
> validate_chain+0x576/0x860
> __lock_acquire+0x39c/0x790
> lock_acquire+0xa2/0x1b0
> __kernfs_remove+0x25f/0x2e0
> kernfs_remove_by_name_ns+0x41/0x80
> remove_files.isra.0+0x30/0x70
> sysfs_remove_group+0x3d/0x80
> sysfs_remove_groups+0x29/0x40
> device_remove_attrs+0x39/0x70
> device_del+0x16a/0x3f0
> device_unregister+0x16/0x60
> remove_memory_block_devices+0x82/0xb0
> try_remove_memory+0xb5/0x130
> remove_memory+0x26/0x40
> dev_dax_kmem_remove+0x44/0x6a [kmem]
> device_release_driver_internal+0xe4/0x1c0
> unbind_store+0xef/0x120
> kernfs_fop_write+0xcf/0x1c0
> vfs_write+0xdb/0x1d0
> ksys_write+0x65/0xe0
> do_syscall_64+0x5c/0xa0
> entry_SYSCALL_64_after_hwframe+0x49/0xbe
>
> other info that might help us debug this:
>
> Chain exists of:
> kn->count#241 --> cpu_hotplug_lock.rw_sem --> mem_hotplug_lock.rw_sem
>
> Possible unsafe locking scenario:
>
> CPU0 CPU1
> ---- ----
> lock(mem_hotplug_lock.rw_sem);
> lock(cpu_hotplug_lock.rw_sem);
> lock(mem_hotplug_lock.rw_sem);
> lock(kn->count#241);
>
> *** DEADLOCK ***
>
> No fixes tag as this has been a long standing issue that predated the
> addition of kernfs lockdep annotations.
>
> Cc: Vishal Verma <vishal.l.verma@intel.com>
> Cc: David Hildenbrand <david@redhat.com>
> Cc: Pavel Tatashin <pasha.tatashin@soleen.com>
> Cc: Michal Hocko <mhocko@suse.com>
> Cc: Dave Hansen <dave.hansen@linux.intel.com>
> Signed-off-by: Dan Williams <dan.j.williams@intel.com>
Acked-by: Michal Hocko <mhocko@suse.com>
Thanks!
> ---
> Changes since v4 [1]:
> - Drop the unnecessary consideration of mem->section_count.
> kernfs_drain() + lock_device_hotplug() is sufficient protection
> (David)
>
> [1]: http://lore.kernel.org/r/157869128062.2451572.4093315441083744888.stgit@dwillia2-desk3.amr.corp.intel.com
>
> mm/memory_hotplug.c | 9 ++++++---
> 1 file changed, 6 insertions(+), 3 deletions(-)
>
> diff --git a/mm/memory_hotplug.c b/mm/memory_hotplug.c
> index 55ac23ef11c1..65ddaf3a2a12 100644
> --- a/mm/memory_hotplug.c
> +++ b/mm/memory_hotplug.c
> @@ -1763,8 +1763,6 @@ static int __ref try_remove_memory(int nid, u64 start, u64 size)
>
> BUG_ON(check_hotplug_memory_range(start, size));
>
> - mem_hotplug_begin();
> -
> /*
> * All memory blocks must be offlined before removing memory. Check
> * whether all memory blocks in question are offline and return error
> @@ -1777,9 +1775,14 @@ static int __ref try_remove_memory(int nid, u64 start, u64 size)
> /* remove memmap entry */
> firmware_map_remove(start, start + size, "System RAM");
>
> - /* remove memory block devices before removing memory */
> + /*
> + * Memory block device removal under the device_hotplug_lock is
> + * a barrier against racing online attempts.
> + */
> remove_memory_block_devices(start, size);
>
> + mem_hotplug_begin();
> +
> arch_remove_memory(nid, start, size, NULL);
> memblock_free(start, size);
> memblock_remove(start, size);
--
Michal Hocko
SUSE Labs
^ permalink raw reply [flat|nested] 3+ messages in thread
end of thread, other threads:[~2020-01-27 13:41 UTC | newest]
Thread overview: 3+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2020-01-25 1:07 [PATCH v5] mm/memory_hotplug: Fix remove_memory() lockdep splat Dan Williams
2020-01-27 7:44 ` David Hildenbrand
2020-01-27 13:41 ` Michal Hocko
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.