* [PATCH] mm, add_memory_resource: hold device_hotplug lock over mem_hotplug_{begin, done}
@ 2017-02-26 11:42 ` Sebastian Ott
0 siblings, 0 replies; 18+ messages in thread
From: Sebastian Ott @ 2017-02-26 11:42 UTC (permalink / raw)
To: Dan Williams; +Cc: linux-mm, linux-kernel, Andrew Morton, Heiko Carstens
With 4.10.0-10265-gc4f3f22 the following warning is triggered on s390:
WARNING: CPU: 6 PID: 1 at drivers/base/core.c:643 assert_held_device_hotplug+0x4a/0x58
[ 5.731214] Call Trace:
[ 5.731219] ([<000000000067b8b0>] assert_held_device_hotplug+0x40/0x58)
[ 5.731225] [<0000000000337914>] mem_hotplug_begin+0x34/0xc8
[ 5.731231] [<00000000008b897e>] add_memory_resource+0x7e/0x1f8
[ 5.731236] [<00000000008b8bd2>] add_memory+0xda/0x130
[ 5.731243] [<0000000000d7f0dc>] add_memory_merged+0x15c/0x178
[ 5.731247] [<0000000000d7f3a6>] sclp_detect_standby_memory+0x2ae/0x2f8
[ 5.731252] [<00000000001002ba>] do_one_initcall+0xa2/0x150
[ 5.731258] [<0000000000d3adc0>] kernel_init_freeable+0x228/0x2d8
[ 5.731263] [<00000000008b6572>] kernel_init+0x2a/0x140
[ 5.731267] [<00000000008c3972>] kernel_thread_starter+0x6/0xc
[ 5.731272] [<00000000008c396c>] kernel_thread_starter+0x0/0xc
[ 5.731276] no locks held by swapper/0/1.
[ 5.731280] Last Breaking-Event-Address:
[ 5.731285] [<000000000067b8b6>] assert_held_device_hotplug+0x46/0x58
[ 5.731292] ---[ end trace 46480df21194c96a ]---
The following patch fixes that for me:
----->8
mm, add_memory_resource: hold device_hotplug lock over mem_hotplug_{begin, done}
With commit 3fc219241 ("mm: validate device_hotplug is held for memory hotplug")
a lock assertion was added to mem_hotplug_begin() which led to a warning
when add_memory() is called. Fix this by acquiring device_hotplug_lock in
add_memory_resource().
Signed-off-by: Sebastian Ott <sebott@linux.vnet.ibm.com>
---
mm/memory_hotplug.c | 2 ++
1 file changed, 2 insertions(+)
diff --git a/mm/memory_hotplug.c b/mm/memory_hotplug.c
index 1d3ed58..c633bbc 100644
--- a/mm/memory_hotplug.c
+++ b/mm/memory_hotplug.c
@@ -1361,6 +1361,7 @@ int __ref add_memory_resource(int nid, struct resource *res, bool online)
new_pgdat = !p;
}
+ lock_device_hotplug();
mem_hotplug_begin();
/*
@@ -1416,6 +1417,7 @@ int __ref add_memory_resource(int nid, struct resource *res, bool online)
out:
mem_hotplug_done();
+ unlock_device_hotplug();
return ret;
}
EXPORT_SYMBOL_GPL(add_memory_resource);
--
2.3.0
^ permalink raw reply related [flat|nested] 18+ messages in thread
* [PATCH] mm, add_memory_resource: hold device_hotplug lock over mem_hotplug_{begin, done}
@ 2017-02-26 11:42 ` Sebastian Ott
0 siblings, 0 replies; 18+ messages in thread
From: Sebastian Ott @ 2017-02-26 11:42 UTC (permalink / raw)
To: Dan Williams; +Cc: linux-mm, linux-kernel, Andrew Morton, Heiko Carstens
With 4.10.0-10265-gc4f3f22 the following warning is triggered on s390:
WARNING: CPU: 6 PID: 1 at drivers/base/core.c:643 assert_held_device_hotplug+0x4a/0x58
[ 5.731214] Call Trace:
[ 5.731219] ([<000000000067b8b0>] assert_held_device_hotplug+0x40/0x58)
[ 5.731225] [<0000000000337914>] mem_hotplug_begin+0x34/0xc8
[ 5.731231] [<00000000008b897e>] add_memory_resource+0x7e/0x1f8
[ 5.731236] [<00000000008b8bd2>] add_memory+0xda/0x130
[ 5.731243] [<0000000000d7f0dc>] add_memory_merged+0x15c/0x178
[ 5.731247] [<0000000000d7f3a6>] sclp_detect_standby_memory+0x2ae/0x2f8
[ 5.731252] [<00000000001002ba>] do_one_initcall+0xa2/0x150
[ 5.731258] [<0000000000d3adc0>] kernel_init_freeable+0x228/0x2d8
[ 5.731263] [<00000000008b6572>] kernel_init+0x2a/0x140
[ 5.731267] [<00000000008c3972>] kernel_thread_starter+0x6/0xc
[ 5.731272] [<00000000008c396c>] kernel_thread_starter+0x0/0xc
[ 5.731276] no locks held by swapper/0/1.
[ 5.731280] Last Breaking-Event-Address:
[ 5.731285] [<000000000067b8b6>] assert_held_device_hotplug+0x46/0x58
[ 5.731292] ---[ end trace 46480df21194c96a ]---
The following patch fixes that for me:
----->8
mm, add_memory_resource: hold device_hotplug lock over mem_hotplug_{begin, done}
With commit 3fc219241 ("mm: validate device_hotplug is held for memory hotplug")
a lock assertion was added to mem_hotplug_begin() which led to a warning
when add_memory() is called. Fix this by acquiring device_hotplug_lock in
add_memory_resource().
Signed-off-by: Sebastian Ott <sebott@linux.vnet.ibm.com>
---
mm/memory_hotplug.c | 2 ++
1 file changed, 2 insertions(+)
diff --git a/mm/memory_hotplug.c b/mm/memory_hotplug.c
index 1d3ed58..c633bbc 100644
--- a/mm/memory_hotplug.c
+++ b/mm/memory_hotplug.c
@@ -1361,6 +1361,7 @@ int __ref add_memory_resource(int nid, struct resource *res, bool online)
new_pgdat = !p;
}
+ lock_device_hotplug();
mem_hotplug_begin();
/*
@@ -1416,6 +1417,7 @@ int __ref add_memory_resource(int nid, struct resource *res, bool online)
out:
mem_hotplug_done();
+ unlock_device_hotplug();
return ret;
}
EXPORT_SYMBOL_GPL(add_memory_resource);
--
2.3.0
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply related [flat|nested] 18+ messages in thread
* Re: [PATCH] mm, add_memory_resource: hold device_hotplug lock over mem_hotplug_{begin, done}
2017-02-26 11:42 ` Sebastian Ott
@ 2017-02-27 16:20 ` Michal Hocko
-1 siblings, 0 replies; 18+ messages in thread
From: Michal Hocko @ 2017-02-27 16:20 UTC (permalink / raw)
To: Sebastian Ott
Cc: Dan Williams, linux-mm, linux-kernel, Andrew Morton,
Heiko Carstens, Rafael J. Wysocki
[CC Rafael]
I've got lost in the acpi indirection (again). I can see
acpi_device_hotplug calling lock_device_hotplug() but i cannot find a
path down to add_memory() which might call add_memory_resource. But the
patch below sounds suspicious to me. Is it possible that this could lead
to a deadlock. I would suspect that it is the s390 code which needs to
do the locking. But I would have to double check - it is really easy to
get lost there.
On Sun 26-02-17 12:42:44, Sebastian Ott wrote:
> With 4.10.0-10265-gc4f3f22 the following warning is triggered on s390:
>
> WARNING: CPU: 6 PID: 1 at drivers/base/core.c:643 assert_held_device_hotplug+0x4a/0x58
> [ 5.731214] Call Trace:
> [ 5.731219] ([<000000000067b8b0>] assert_held_device_hotplug+0x40/0x58)
> [ 5.731225] [<0000000000337914>] mem_hotplug_begin+0x34/0xc8
> [ 5.731231] [<00000000008b897e>] add_memory_resource+0x7e/0x1f8
> [ 5.731236] [<00000000008b8bd2>] add_memory+0xda/0x130
> [ 5.731243] [<0000000000d7f0dc>] add_memory_merged+0x15c/0x178
> [ 5.731247] [<0000000000d7f3a6>] sclp_detect_standby_memory+0x2ae/0x2f8
> [ 5.731252] [<00000000001002ba>] do_one_initcall+0xa2/0x150
> [ 5.731258] [<0000000000d3adc0>] kernel_init_freeable+0x228/0x2d8
> [ 5.731263] [<00000000008b6572>] kernel_init+0x2a/0x140
> [ 5.731267] [<00000000008c3972>] kernel_thread_starter+0x6/0xc
> [ 5.731272] [<00000000008c396c>] kernel_thread_starter+0x0/0xc
> [ 5.731276] no locks held by swapper/0/1.
> [ 5.731280] Last Breaking-Event-Address:
> [ 5.731285] [<000000000067b8b6>] assert_held_device_hotplug+0x46/0x58
> [ 5.731292] ---[ end trace 46480df21194c96a ]---
such an informtion belongs to the changelog
> ----->8
> mm, add_memory_resource: hold device_hotplug lock over mem_hotplug_{begin, done}
>
> With commit 3fc219241 ("mm: validate device_hotplug is held for memory hotplug")
> a lock assertion was added to mem_hotplug_begin() which led to a warning
> when add_memory() is called. Fix this by acquiring device_hotplug_lock in
> add_memory_resource().
>
> Signed-off-by: Sebastian Ott <sebott@linux.vnet.ibm.com>
> ---
> mm/memory_hotplug.c | 2 ++
> 1 file changed, 2 insertions(+)
>
> diff --git a/mm/memory_hotplug.c b/mm/memory_hotplug.c
> index 1d3ed58..c633bbc 100644
> --- a/mm/memory_hotplug.c
> +++ b/mm/memory_hotplug.c
> @@ -1361,6 +1361,7 @@ int __ref add_memory_resource(int nid, struct resource *res, bool online)
> new_pgdat = !p;
> }
>
> + lock_device_hotplug();
> mem_hotplug_begin();
>
> /*
> @@ -1416,6 +1417,7 @@ int __ref add_memory_resource(int nid, struct resource *res, bool online)
>
> out:
> mem_hotplug_done();
> + unlock_device_hotplug();
> return ret;
> }
> EXPORT_SYMBOL_GPL(add_memory_resource);
> --
> 2.3.0
>
> --
> To unsubscribe, send a message with 'unsubscribe linux-mm' in
> the body to majordomo@kvack.org. For more info on Linux MM,
> see: http://www.linux-mm.org/ .
> Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
--
Michal Hocko
SUSE Labs
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [PATCH] mm, add_memory_resource: hold device_hotplug lock over mem_hotplug_{begin, done}
@ 2017-02-27 16:20 ` Michal Hocko
0 siblings, 0 replies; 18+ messages in thread
From: Michal Hocko @ 2017-02-27 16:20 UTC (permalink / raw)
To: Sebastian Ott
Cc: Dan Williams, linux-mm, linux-kernel, Andrew Morton,
Heiko Carstens, Rafael J. Wysocki
[CC Rafael]
I've got lost in the acpi indirection (again). I can see
acpi_device_hotplug calling lock_device_hotplug() but i cannot find a
path down to add_memory() which might call add_memory_resource. But the
patch below sounds suspicious to me. Is it possible that this could lead
to a deadlock. I would suspect that it is the s390 code which needs to
do the locking. But I would have to double check - it is really easy to
get lost there.
On Sun 26-02-17 12:42:44, Sebastian Ott wrote:
> With 4.10.0-10265-gc4f3f22 the following warning is triggered on s390:
>
> WARNING: CPU: 6 PID: 1 at drivers/base/core.c:643 assert_held_device_hotplug+0x4a/0x58
> [ 5.731214] Call Trace:
> [ 5.731219] ([<000000000067b8b0>] assert_held_device_hotplug+0x40/0x58)
> [ 5.731225] [<0000000000337914>] mem_hotplug_begin+0x34/0xc8
> [ 5.731231] [<00000000008b897e>] add_memory_resource+0x7e/0x1f8
> [ 5.731236] [<00000000008b8bd2>] add_memory+0xda/0x130
> [ 5.731243] [<0000000000d7f0dc>] add_memory_merged+0x15c/0x178
> [ 5.731247] [<0000000000d7f3a6>] sclp_detect_standby_memory+0x2ae/0x2f8
> [ 5.731252] [<00000000001002ba>] do_one_initcall+0xa2/0x150
> [ 5.731258] [<0000000000d3adc0>] kernel_init_freeable+0x228/0x2d8
> [ 5.731263] [<00000000008b6572>] kernel_init+0x2a/0x140
> [ 5.731267] [<00000000008c3972>] kernel_thread_starter+0x6/0xc
> [ 5.731272] [<00000000008c396c>] kernel_thread_starter+0x0/0xc
> [ 5.731276] no locks held by swapper/0/1.
> [ 5.731280] Last Breaking-Event-Address:
> [ 5.731285] [<000000000067b8b6>] assert_held_device_hotplug+0x46/0x58
> [ 5.731292] ---[ end trace 46480df21194c96a ]---
such an informtion belongs to the changelog
> ----->8
> mm, add_memory_resource: hold device_hotplug lock over mem_hotplug_{begin, done}
>
> With commit 3fc219241 ("mm: validate device_hotplug is held for memory hotplug")
> a lock assertion was added to mem_hotplug_begin() which led to a warning
> when add_memory() is called. Fix this by acquiring device_hotplug_lock in
> add_memory_resource().
>
> Signed-off-by: Sebastian Ott <sebott@linux.vnet.ibm.com>
> ---
> mm/memory_hotplug.c | 2 ++
> 1 file changed, 2 insertions(+)
>
> diff --git a/mm/memory_hotplug.c b/mm/memory_hotplug.c
> index 1d3ed58..c633bbc 100644
> --- a/mm/memory_hotplug.c
> +++ b/mm/memory_hotplug.c
> @@ -1361,6 +1361,7 @@ int __ref add_memory_resource(int nid, struct resource *res, bool online)
> new_pgdat = !p;
> }
>
> + lock_device_hotplug();
> mem_hotplug_begin();
>
> /*
> @@ -1416,6 +1417,7 @@ int __ref add_memory_resource(int nid, struct resource *res, bool online)
>
> out:
> mem_hotplug_done();
> + unlock_device_hotplug();
> return ret;
> }
> EXPORT_SYMBOL_GPL(add_memory_resource);
> --
> 2.3.0
>
> --
> To unsubscribe, send a message with 'unsubscribe linux-mm' in
> the body to majordomo@kvack.org. For more info on Linux MM,
> see: http://www.linux-mm.org/ .
> Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
--
Michal Hocko
SUSE Labs
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [PATCH] mm, add_memory_resource: hold device_hotplug lock over mem_hotplug_{begin, done}
2017-02-27 16:20 ` Michal Hocko
@ 2017-02-28 11:57 ` Heiko Carstens
-1 siblings, 0 replies; 18+ messages in thread
From: Heiko Carstens @ 2017-02-28 11:57 UTC (permalink / raw)
To: Michal Hocko, Vladimir Davydov
Cc: Sebastian Ott, Dan Williams, linux-mm, linux-kernel,
Andrew Morton, Rafael J. Wysocki
On Mon, Feb 27, 2017 at 05:20:31PM +0100, Michal Hocko wrote:
> [CC Rafael]
>
> I've got lost in the acpi indirection (again). I can see
> acpi_device_hotplug calling lock_device_hotplug() but i cannot find a
> path down to add_memory() which might call add_memory_resource. But the
> patch below sounds suspicious to me. Is it possible that this could lead
> to a deadlock. I would suspect that it is the s390 code which needs to
> do the locking. But I would have to double check - it is really easy to
> get lost there.
To me it rather looks like bfc8c90139eb ("mem-hotplug: implement
get/put_online_mems") introduced quite subtle and probably wrong locking
rules.
The patch introduced mem_hotplug_begin() in order to have something like
cpu_hotplug_begin() for memory. Note that for cpu hotplug all
cpu_hotplug_begin() calls are serialized by cpu_maps_update_begin().
Especially this makes sure that active_writer can only be changed by one
process. (See also Dan's commit which introduced the lock_device_hotplug()
calls: https://marc.info/?l=linux-kernel&m=148693912419972&w=2 )
If you look at the above commit bfc8c90139eb: there is nothing like
cpu_maps_update_begin() for memory. And therefore it's possible to have
concurrent writers to active_writer.
It looks like now lock_device_hotplug() is supposed to be the new
cpu_maps_update_begin() for memory. But.. this looks like a mess, unless I
read the code completely wrong ;)
> On Sun 26-02-17 12:42:44, Sebastian Ott wrote:
> > With 4.10.0-10265-gc4f3f22 the following warning is triggered on s390:
> >
> > WARNING: CPU: 6 PID: 1 at drivers/base/core.c:643 assert_held_device_hotplug+0x4a/0x58
> > [ 5.731214] Call Trace:
> > [ 5.731219] ([<000000000067b8b0>] assert_held_device_hotplug+0x40/0x58)
> > [ 5.731225] [<0000000000337914>] mem_hotplug_begin+0x34/0xc8
> > [ 5.731231] [<00000000008b897e>] add_memory_resource+0x7e/0x1f8
> > [ 5.731236] [<00000000008b8bd2>] add_memory+0xda/0x130
> > [ 5.731243] [<0000000000d7f0dc>] add_memory_merged+0x15c/0x178
> > [ 5.731247] [<0000000000d7f3a6>] sclp_detect_standby_memory+0x2ae/0x2f8
> > [ 5.731252] [<00000000001002ba>] do_one_initcall+0xa2/0x150
> > [ 5.731258] [<0000000000d3adc0>] kernel_init_freeable+0x228/0x2d8
> > [ 5.731263] [<00000000008b6572>] kernel_init+0x2a/0x140
> > [ 5.731267] [<00000000008c3972>] kernel_thread_starter+0x6/0xc
> > [ 5.731272] [<00000000008c396c>] kernel_thread_starter+0x0/0xc
> > [ 5.731276] no locks held by swapper/0/1.
> > [ 5.731280] Last Breaking-Event-Address:
> > [ 5.731285] [<000000000067b8b6>] assert_held_device_hotplug+0x46/0x58
> > [ 5.731292] ---[ end trace 46480df21194c96a ]---
>
> such an informtion belongs to the changelog
>
> > ----->8
> > mm, add_memory_resource: hold device_hotplug lock over mem_hotplug_{begin, done}
> >
> > With commit 3fc219241 ("mm: validate device_hotplug is held for memory hotplug")
> > a lock assertion was added to mem_hotplug_begin() which led to a warning
> > when add_memory() is called. Fix this by acquiring device_hotplug_lock in
> > add_memory_resource().
> >
> > Signed-off-by: Sebastian Ott <sebott@linux.vnet.ibm.com>
> > ---
> > mm/memory_hotplug.c | 2 ++
> > 1 file changed, 2 insertions(+)
> >
> > diff --git a/mm/memory_hotplug.c b/mm/memory_hotplug.c
> > index 1d3ed58..c633bbc 100644
> > --- a/mm/memory_hotplug.c
> > +++ b/mm/memory_hotplug.c
> > @@ -1361,6 +1361,7 @@ int __ref add_memory_resource(int nid, struct resource *res, bool online)
> > new_pgdat = !p;
> > }
> >
> > + lock_device_hotplug();
> > mem_hotplug_begin();
> >
> > /*
> > @@ -1416,6 +1417,7 @@ int __ref add_memory_resource(int nid, struct resource *res, bool online)
> >
> > out:
> > mem_hotplug_done();
> > + unlock_device_hotplug();
> > return ret;
> > }
> > EXPORT_SYMBOL_GPL(add_memory_resource);
> > --
> > 2.3.0
> >
> > --
> > To unsubscribe, send a message with 'unsubscribe linux-mm' in
> > the body to majordomo@kvack.org. For more info on Linux MM,
> > see: http://www.linux-mm.org/ .
> > Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
>
> --
> Michal Hocko
> SUSE Labs
>
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [PATCH] mm, add_memory_resource: hold device_hotplug lock over mem_hotplug_{begin, done}
@ 2017-02-28 11:57 ` Heiko Carstens
0 siblings, 0 replies; 18+ messages in thread
From: Heiko Carstens @ 2017-02-28 11:57 UTC (permalink / raw)
To: Michal Hocko, Vladimir Davydov
Cc: Sebastian Ott, Dan Williams, linux-mm, linux-kernel,
Andrew Morton, Rafael J. Wysocki
On Mon, Feb 27, 2017 at 05:20:31PM +0100, Michal Hocko wrote:
> [CC Rafael]
>
> I've got lost in the acpi indirection (again). I can see
> acpi_device_hotplug calling lock_device_hotplug() but i cannot find a
> path down to add_memory() which might call add_memory_resource. But the
> patch below sounds suspicious to me. Is it possible that this could lead
> to a deadlock. I would suspect that it is the s390 code which needs to
> do the locking. But I would have to double check - it is really easy to
> get lost there.
To me it rather looks like bfc8c90139eb ("mem-hotplug: implement
get/put_online_mems") introduced quite subtle and probably wrong locking
rules.
The patch introduced mem_hotplug_begin() in order to have something like
cpu_hotplug_begin() for memory. Note that for cpu hotplug all
cpu_hotplug_begin() calls are serialized by cpu_maps_update_begin().
Especially this makes sure that active_writer can only be changed by one
process. (See also Dan's commit which introduced the lock_device_hotplug()
calls: https://marc.info/?l=linux-kernel&m=148693912419972&w=2 )
If you look at the above commit bfc8c90139eb: there is nothing like
cpu_maps_update_begin() for memory. And therefore it's possible to have
concurrent writers to active_writer.
It looks like now lock_device_hotplug() is supposed to be the new
cpu_maps_update_begin() for memory. But.. this looks like a mess, unless I
read the code completely wrong ;)
> On Sun 26-02-17 12:42:44, Sebastian Ott wrote:
> > With 4.10.0-10265-gc4f3f22 the following warning is triggered on s390:
> >
> > WARNING: CPU: 6 PID: 1 at drivers/base/core.c:643 assert_held_device_hotplug+0x4a/0x58
> > [ 5.731214] Call Trace:
> > [ 5.731219] ([<000000000067b8b0>] assert_held_device_hotplug+0x40/0x58)
> > [ 5.731225] [<0000000000337914>] mem_hotplug_begin+0x34/0xc8
> > [ 5.731231] [<00000000008b897e>] add_memory_resource+0x7e/0x1f8
> > [ 5.731236] [<00000000008b8bd2>] add_memory+0xda/0x130
> > [ 5.731243] [<0000000000d7f0dc>] add_memory_merged+0x15c/0x178
> > [ 5.731247] [<0000000000d7f3a6>] sclp_detect_standby_memory+0x2ae/0x2f8
> > [ 5.731252] [<00000000001002ba>] do_one_initcall+0xa2/0x150
> > [ 5.731258] [<0000000000d3adc0>] kernel_init_freeable+0x228/0x2d8
> > [ 5.731263] [<00000000008b6572>] kernel_init+0x2a/0x140
> > [ 5.731267] [<00000000008c3972>] kernel_thread_starter+0x6/0xc
> > [ 5.731272] [<00000000008c396c>] kernel_thread_starter+0x0/0xc
> > [ 5.731276] no locks held by swapper/0/1.
> > [ 5.731280] Last Breaking-Event-Address:
> > [ 5.731285] [<000000000067b8b6>] assert_held_device_hotplug+0x46/0x58
> > [ 5.731292] ---[ end trace 46480df21194c96a ]---
>
> such an informtion belongs to the changelog
>
> > ----->8
> > mm, add_memory_resource: hold device_hotplug lock over mem_hotplug_{begin, done}
> >
> > With commit 3fc219241 ("mm: validate device_hotplug is held for memory hotplug")
> > a lock assertion was added to mem_hotplug_begin() which led to a warning
> > when add_memory() is called. Fix this by acquiring device_hotplug_lock in
> > add_memory_resource().
> >
> > Signed-off-by: Sebastian Ott <sebott@linux.vnet.ibm.com>
> > ---
> > mm/memory_hotplug.c | 2 ++
> > 1 file changed, 2 insertions(+)
> >
> > diff --git a/mm/memory_hotplug.c b/mm/memory_hotplug.c
> > index 1d3ed58..c633bbc 100644
> > --- a/mm/memory_hotplug.c
> > +++ b/mm/memory_hotplug.c
> > @@ -1361,6 +1361,7 @@ int __ref add_memory_resource(int nid, struct resource *res, bool online)
> > new_pgdat = !p;
> > }
> >
> > + lock_device_hotplug();
> > mem_hotplug_begin();
> >
> > /*
> > @@ -1416,6 +1417,7 @@ int __ref add_memory_resource(int nid, struct resource *res, bool online)
> >
> > out:
> > mem_hotplug_done();
> > + unlock_device_hotplug();
> > return ret;
> > }
> > EXPORT_SYMBOL_GPL(add_memory_resource);
> > --
> > 2.3.0
> >
> > --
> > To unsubscribe, send a message with 'unsubscribe linux-mm' in
> > the body to majordomo@kvack.org. For more info on Linux MM,
> > see: http://www.linux-mm.org/ .
> > Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
>
> --
> Michal Hocko
> SUSE Labs
>
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [PATCH] mm, add_memory_resource: hold device_hotplug lock over mem_hotplug_{begin, done}
2017-02-28 11:57 ` Heiko Carstens
@ 2017-03-01 12:51 ` Heiko Carstens
-1 siblings, 0 replies; 18+ messages in thread
From: Heiko Carstens @ 2017-03-01 12:51 UTC (permalink / raw)
To: Michal Hocko, Sebastian Ott, Dan Williams, linux-mm,
linux-kernel, Andrew Morton, Rafael J. Wysocki, Vladimir Davydov,
Ben Hutchings
On Tue, Feb 28, 2017 at 12:57:29PM +0100, Heiko Carstens wrote:
> On Mon, Feb 27, 2017 at 05:20:31PM +0100, Michal Hocko wrote:
> > [CC Rafael]
> >
> > I've got lost in the acpi indirection (again). I can see
> > acpi_device_hotplug calling lock_device_hotplug() but i cannot find a
> > path down to add_memory() which might call add_memory_resource. But the
> > patch below sounds suspicious to me. Is it possible that this could lead
> > to a deadlock. I would suspect that it is the s390 code which needs to
> > do the locking. But I would have to double check - it is really easy to
> > get lost there.
>
> To me it rather looks like bfc8c90139eb ("mem-hotplug: implement
> get/put_online_mems") introduced quite subtle and probably wrong locking
> rules.
>
> The patch introduced mem_hotplug_begin() in order to have something like
> cpu_hotplug_begin() for memory. Note that for cpu hotplug all
> cpu_hotplug_begin() calls are serialized by cpu_maps_update_begin().
>
> Especially this makes sure that active_writer can only be changed by one
> process. (See also Dan's commit which introduced the lock_device_hotplug()
> calls: https://marc.info/?l=linux-kernel&m=148693912419972&w=2 )
>
> If you look at the above commit bfc8c90139eb: there is nothing like
> cpu_maps_update_begin() for memory. And therefore it's possible to have
> concurrent writers to active_writer.
>
> It looks like now lock_device_hotplug() is supposed to be the new
> cpu_maps_update_begin() for memory. But.. this looks like a mess, unless I
> read the code completely wrong ;)
[Full quote since I now hopefully use a non-bouncing email address from
Vladimir]
Since it is anything but obvious why Dan wrote in changelog of b5d24fda9c3d
("mm, devm_memremap_pages: hold device_hotplug lock over
mem_hotplug_{begin, done}") that write accesses to
mem_hotplug.active_writer are coordinated via lock_device_hotplug() I'd
rather propose a new private memory_add_remove_lock which has similar
semantics like the cpu_add_remove_lock for cpu hotplug (see patch below).
However instead of sprinkling locking/unlocking of that new lock around all
calls of mem_hotplug_begin() and mem_hotplug_end() simply include locking
and unlocking into these two functions.
This still allows get_online_mems() and put_online_mems() to work, while at
the same time preventing mem_hotplug.active_writer corruption.
Any opinions?
---
kernel/memremap.c | 4 ----
mm/memory_hotplug.c | 6 +++++-
2 files changed, 5 insertions(+), 5 deletions(-)
diff --git a/kernel/memremap.c b/kernel/memremap.c
index 06123234f118..07e85e5229da 100644
--- a/kernel/memremap.c
+++ b/kernel/memremap.c
@@ -247,11 +247,9 @@ static void devm_memremap_pages_release(struct device *dev, void *data)
align_start = res->start & ~(SECTION_SIZE - 1);
align_size = ALIGN(resource_size(res), SECTION_SIZE);
- lock_device_hotplug();
mem_hotplug_begin();
arch_remove_memory(align_start, align_size);
mem_hotplug_done();
- unlock_device_hotplug();
untrack_pfn(NULL, PHYS_PFN(align_start), align_size);
pgmap_radix_release(res);
@@ -364,11 +362,9 @@ void *devm_memremap_pages(struct device *dev, struct resource *res,
if (error)
goto err_pfn_remap;
- lock_device_hotplug();
mem_hotplug_begin();
error = arch_add_memory(nid, align_start, align_size, true);
mem_hotplug_done();
- unlock_device_hotplug();
if (error)
goto err_add_memory;
diff --git a/mm/memory_hotplug.c b/mm/memory_hotplug.c
index 1d3ed58f92ab..6ee6e6a17310 100644
--- a/mm/memory_hotplug.c
+++ b/mm/memory_hotplug.c
@@ -124,9 +124,12 @@ void put_online_mems(void)
}
+/* Needed to serialize write accesses to mem_hotplug.active_writer. */
+static DEFINE_MUTEX(memory_add_remove_lock);
+
void mem_hotplug_begin(void)
{
- assert_held_device_hotplug();
+ mutex_lock(&memory_add_remove_lock);
mem_hotplug.active_writer = current;
@@ -146,6 +149,7 @@ void mem_hotplug_done(void)
mem_hotplug.active_writer = NULL;
mutex_unlock(&mem_hotplug.lock);
memhp_lock_release();
+ mutex_unlock(&memory_add_remove_lock);
}
/* add this memory to iomem resource */
--
2.8.4
> > On Sun 26-02-17 12:42:44, Sebastian Ott wrote:
> > > With 4.10.0-10265-gc4f3f22 the following warning is triggered on s390:
> > >
> > > WARNING: CPU: 6 PID: 1 at drivers/base/core.c:643 assert_held_device_hotplug+0x4a/0x58
> > > [ 5.731214] Call Trace:
> > > [ 5.731219] ([<000000000067b8b0>] assert_held_device_hotplug+0x40/0x58)
> > > [ 5.731225] [<0000000000337914>] mem_hotplug_begin+0x34/0xc8
> > > [ 5.731231] [<00000000008b897e>] add_memory_resource+0x7e/0x1f8
> > > [ 5.731236] [<00000000008b8bd2>] add_memory+0xda/0x130
> > > [ 5.731243] [<0000000000d7f0dc>] add_memory_merged+0x15c/0x178
> > > [ 5.731247] [<0000000000d7f3a6>] sclp_detect_standby_memory+0x2ae/0x2f8
> > > [ 5.731252] [<00000000001002ba>] do_one_initcall+0xa2/0x150
> > > [ 5.731258] [<0000000000d3adc0>] kernel_init_freeable+0x228/0x2d8
> > > [ 5.731263] [<00000000008b6572>] kernel_init+0x2a/0x140
> > > [ 5.731267] [<00000000008c3972>] kernel_thread_starter+0x6/0xc
> > > [ 5.731272] [<00000000008c396c>] kernel_thread_starter+0x0/0xc
> > > [ 5.731276] no locks held by swapper/0/1.
> > > [ 5.731280] Last Breaking-Event-Address:
> > > [ 5.731285] [<000000000067b8b6>] assert_held_device_hotplug+0x46/0x58
> > > [ 5.731292] ---[ end trace 46480df21194c96a ]---
> >
> > such an informtion belongs to the changelog
> >
> > > ----->8
> > > mm, add_memory_resource: hold device_hotplug lock over mem_hotplug_{begin, done}
> > >
> > > With commit 3fc219241 ("mm: validate device_hotplug is held for memory hotplug")
> > > a lock assertion was added to mem_hotplug_begin() which led to a warning
> > > when add_memory() is called. Fix this by acquiring device_hotplug_lock in
> > > add_memory_resource().
> > >
> > > Signed-off-by: Sebastian Ott <sebott@linux.vnet.ibm.com>
> > > ---
> > > mm/memory_hotplug.c | 2 ++
> > > 1 file changed, 2 insertions(+)
> > >
> > > diff --git a/mm/memory_hotplug.c b/mm/memory_hotplug.c
> > > index 1d3ed58..c633bbc 100644
> > > --- a/mm/memory_hotplug.c
> > > +++ b/mm/memory_hotplug.c
> > > @@ -1361,6 +1361,7 @@ int __ref add_memory_resource(int nid, struct resource *res, bool online)
> > > new_pgdat = !p;
> > > }
> > >
> > > + lock_device_hotplug();
> > > mem_hotplug_begin();
> > >
> > > /*
> > > @@ -1416,6 +1417,7 @@ int __ref add_memory_resource(int nid, struct resource *res, bool online)
> > >
> > > out:
> > > mem_hotplug_done();
> > > + unlock_device_hotplug();
> > > return ret;
> > > }
> > > EXPORT_SYMBOL_GPL(add_memory_resource);
> > > --
> > > 2.3.0
> > >
> > > --
> > > To unsubscribe, send a message with 'unsubscribe linux-mm' in
> > > the body to majordomo@kvack.org. For more info on Linux MM,
> > > see: http://www.linux-mm.org/ .
> > > Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
> >
> > --
> > Michal Hocko
> > SUSE Labs
> >
>
> --
> To unsubscribe, send a message with 'unsubscribe linux-mm' in
> the body to majordomo@kvack.org. For more info on Linux MM,
> see: http://www.linux-mm.org/ .
> Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply related [flat|nested] 18+ messages in thread
* Re: [PATCH] mm, add_memory_resource: hold device_hotplug lock over mem_hotplug_{begin, done}
@ 2017-03-01 12:51 ` Heiko Carstens
0 siblings, 0 replies; 18+ messages in thread
From: Heiko Carstens @ 2017-03-01 12:51 UTC (permalink / raw)
To: Michal Hocko, Sebastian Ott, Dan Williams, linux-mm,
linux-kernel, Andrew Morton, Rafael J. Wysocki, Vladimir Davydov,
Ben Hutchings
On Tue, Feb 28, 2017 at 12:57:29PM +0100, Heiko Carstens wrote:
> On Mon, Feb 27, 2017 at 05:20:31PM +0100, Michal Hocko wrote:
> > [CC Rafael]
> >
> > I've got lost in the acpi indirection (again). I can see
> > acpi_device_hotplug calling lock_device_hotplug() but i cannot find a
> > path down to add_memory() which might call add_memory_resource. But the
> > patch below sounds suspicious to me. Is it possible that this could lead
> > to a deadlock. I would suspect that it is the s390 code which needs to
> > do the locking. But I would have to double check - it is really easy to
> > get lost there.
>
> To me it rather looks like bfc8c90139eb ("mem-hotplug: implement
> get/put_online_mems") introduced quite subtle and probably wrong locking
> rules.
>
> The patch introduced mem_hotplug_begin() in order to have something like
> cpu_hotplug_begin() for memory. Note that for cpu hotplug all
> cpu_hotplug_begin() calls are serialized by cpu_maps_update_begin().
>
> Especially this makes sure that active_writer can only be changed by one
> process. (See also Dan's commit which introduced the lock_device_hotplug()
> calls: https://marc.info/?l=linux-kernel&m=148693912419972&w=2 )
>
> If you look at the above commit bfc8c90139eb: there is nothing like
> cpu_maps_update_begin() for memory. And therefore it's possible to have
> concurrent writers to active_writer.
>
> It looks like now lock_device_hotplug() is supposed to be the new
> cpu_maps_update_begin() for memory. But.. this looks like a mess, unless I
> read the code completely wrong ;)
[Full quote since I now hopefully use a non-bouncing email address from
Vladimir]
Since it is anything but obvious why Dan wrote in changelog of b5d24fda9c3d
("mm, devm_memremap_pages: hold device_hotplug lock over
mem_hotplug_{begin, done}") that write accesses to
mem_hotplug.active_writer are coordinated via lock_device_hotplug() I'd
rather propose a new private memory_add_remove_lock which has similar
semantics like the cpu_add_remove_lock for cpu hotplug (see patch below).
However instead of sprinkling locking/unlocking of that new lock around all
calls of mem_hotplug_begin() and mem_hotplug_end() simply include locking
and unlocking into these two functions.
This still allows get_online_mems() and put_online_mems() to work, while at
the same time preventing mem_hotplug.active_writer corruption.
Any opinions?
---
kernel/memremap.c | 4 ----
mm/memory_hotplug.c | 6 +++++-
2 files changed, 5 insertions(+), 5 deletions(-)
diff --git a/kernel/memremap.c b/kernel/memremap.c
index 06123234f118..07e85e5229da 100644
--- a/kernel/memremap.c
+++ b/kernel/memremap.c
@@ -247,11 +247,9 @@ static void devm_memremap_pages_release(struct device *dev, void *data)
align_start = res->start & ~(SECTION_SIZE - 1);
align_size = ALIGN(resource_size(res), SECTION_SIZE);
- lock_device_hotplug();
mem_hotplug_begin();
arch_remove_memory(align_start, align_size);
mem_hotplug_done();
- unlock_device_hotplug();
untrack_pfn(NULL, PHYS_PFN(align_start), align_size);
pgmap_radix_release(res);
@@ -364,11 +362,9 @@ void *devm_memremap_pages(struct device *dev, struct resource *res,
if (error)
goto err_pfn_remap;
- lock_device_hotplug();
mem_hotplug_begin();
error = arch_add_memory(nid, align_start, align_size, true);
mem_hotplug_done();
- unlock_device_hotplug();
if (error)
goto err_add_memory;
diff --git a/mm/memory_hotplug.c b/mm/memory_hotplug.c
index 1d3ed58f92ab..6ee6e6a17310 100644
--- a/mm/memory_hotplug.c
+++ b/mm/memory_hotplug.c
@@ -124,9 +124,12 @@ void put_online_mems(void)
}
+/* Needed to serialize write accesses to mem_hotplug.active_writer. */
+static DEFINE_MUTEX(memory_add_remove_lock);
+
void mem_hotplug_begin(void)
{
- assert_held_device_hotplug();
+ mutex_lock(&memory_add_remove_lock);
mem_hotplug.active_writer = current;
@@ -146,6 +149,7 @@ void mem_hotplug_done(void)
mem_hotplug.active_writer = NULL;
mutex_unlock(&mem_hotplug.lock);
memhp_lock_release();
+ mutex_unlock(&memory_add_remove_lock);
}
/* add this memory to iomem resource */
--
2.8.4
> > On Sun 26-02-17 12:42:44, Sebastian Ott wrote:
> > > With 4.10.0-10265-gc4f3f22 the following warning is triggered on s390:
> > >
> > > WARNING: CPU: 6 PID: 1 at drivers/base/core.c:643 assert_held_device_hotplug+0x4a/0x58
> > > [ 5.731214] Call Trace:
> > > [ 5.731219] ([<000000000067b8b0>] assert_held_device_hotplug+0x40/0x58)
> > > [ 5.731225] [<0000000000337914>] mem_hotplug_begin+0x34/0xc8
> > > [ 5.731231] [<00000000008b897e>] add_memory_resource+0x7e/0x1f8
> > > [ 5.731236] [<00000000008b8bd2>] add_memory+0xda/0x130
> > > [ 5.731243] [<0000000000d7f0dc>] add_memory_merged+0x15c/0x178
> > > [ 5.731247] [<0000000000d7f3a6>] sclp_detect_standby_memory+0x2ae/0x2f8
> > > [ 5.731252] [<00000000001002ba>] do_one_initcall+0xa2/0x150
> > > [ 5.731258] [<0000000000d3adc0>] kernel_init_freeable+0x228/0x2d8
> > > [ 5.731263] [<00000000008b6572>] kernel_init+0x2a/0x140
> > > [ 5.731267] [<00000000008c3972>] kernel_thread_starter+0x6/0xc
> > > [ 5.731272] [<00000000008c396c>] kernel_thread_starter+0x0/0xc
> > > [ 5.731276] no locks held by swapper/0/1.
> > > [ 5.731280] Last Breaking-Event-Address:
> > > [ 5.731285] [<000000000067b8b6>] assert_held_device_hotplug+0x46/0x58
> > > [ 5.731292] ---[ end trace 46480df21194c96a ]---
> >
> > such an informtion belongs to the changelog
> >
> > > ----->8
> > > mm, add_memory_resource: hold device_hotplug lock over mem_hotplug_{begin, done}
> > >
> > > With commit 3fc219241 ("mm: validate device_hotplug is held for memory hotplug")
> > > a lock assertion was added to mem_hotplug_begin() which led to a warning
> > > when add_memory() is called. Fix this by acquiring device_hotplug_lock in
> > > add_memory_resource().
> > >
> > > Signed-off-by: Sebastian Ott <sebott@linux.vnet.ibm.com>
> > > ---
> > > mm/memory_hotplug.c | 2 ++
> > > 1 file changed, 2 insertions(+)
> > >
> > > diff --git a/mm/memory_hotplug.c b/mm/memory_hotplug.c
> > > index 1d3ed58..c633bbc 100644
> > > --- a/mm/memory_hotplug.c
> > > +++ b/mm/memory_hotplug.c
> > > @@ -1361,6 +1361,7 @@ int __ref add_memory_resource(int nid, struct resource *res, bool online)
> > > new_pgdat = !p;
> > > }
> > >
> > > + lock_device_hotplug();
> > > mem_hotplug_begin();
> > >
> > > /*
> > > @@ -1416,6 +1417,7 @@ int __ref add_memory_resource(int nid, struct resource *res, bool online)
> > >
> > > out:
> > > mem_hotplug_done();
> > > + unlock_device_hotplug();
> > > return ret;
> > > }
> > > EXPORT_SYMBOL_GPL(add_memory_resource);
> > > --
> > > 2.3.0
> > >
> > > --
> > > To unsubscribe, send a message with 'unsubscribe linux-mm' in
> > > the body to majordomo@kvack.org. For more info on Linux MM,
> > > see: http://www.linux-mm.org/ .
> > > Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
> >
> > --
> > Michal Hocko
> > SUSE Labs
> >
>
> --
> To unsubscribe, send a message with 'unsubscribe linux-mm' in
> the body to majordomo@kvack.org. For more info on Linux MM,
> see: http://www.linux-mm.org/ .
> Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply related [flat|nested] 18+ messages in thread
* Re: [PATCH] mm, add_memory_resource: hold device_hotplug lock over mem_hotplug_{begin, done}
2017-03-01 12:51 ` Heiko Carstens
@ 2017-03-01 15:52 ` Dan Williams
-1 siblings, 0 replies; 18+ messages in thread
From: Dan Williams @ 2017-03-01 15:52 UTC (permalink / raw)
To: Heiko Carstens
Cc: Michal Hocko, Sebastian Ott, Linux MM, linux-kernel,
Andrew Morton, Rafael J. Wysocki, Vladimir Davydov,
Ben Hutchings
On Wed, Mar 1, 2017 at 4:51 AM, Heiko Carstens
<heiko.carstens@de.ibm.com> wrote:
> On Tue, Feb 28, 2017 at 12:57:29PM +0100, Heiko Carstens wrote:
>> On Mon, Feb 27, 2017 at 05:20:31PM +0100, Michal Hocko wrote:
>> > [CC Rafael]
>> >
>> > I've got lost in the acpi indirection (again). I can see
>> > acpi_device_hotplug calling lock_device_hotplug() but i cannot find a
>> > path down to add_memory() which might call add_memory_resource. But the
>> > patch below sounds suspicious to me. Is it possible that this could lead
>> > to a deadlock. I would suspect that it is the s390 code which needs to
>> > do the locking. But I would have to double check - it is really easy to
>> > get lost there.
>>
>> To me it rather looks like bfc8c90139eb ("mem-hotplug: implement
>> get/put_online_mems") introduced quite subtle and probably wrong locking
>> rules.
>>
>> The patch introduced mem_hotplug_begin() in order to have something like
>> cpu_hotplug_begin() for memory. Note that for cpu hotplug all
>> cpu_hotplug_begin() calls are serialized by cpu_maps_update_begin().
>>
>> Especially this makes sure that active_writer can only be changed by one
>> process. (See also Dan's commit which introduced the lock_device_hotplug()
>> calls: https://marc.info/?l=linux-kernel&m=148693912419972&w=2 )
>>
>> If you look at the above commit bfc8c90139eb: there is nothing like
>> cpu_maps_update_begin() for memory. And therefore it's possible to have
>> concurrent writers to active_writer.
>>
>> It looks like now lock_device_hotplug() is supposed to be the new
>> cpu_maps_update_begin() for memory. But.. this looks like a mess, unless I
>> read the code completely wrong ;)
>
> [Full quote since I now hopefully use a non-bouncing email address from
> Vladimir]
>
> Since it is anything but obvious why Dan wrote in changelog of b5d24fda9c3d
> ("mm, devm_memremap_pages: hold device_hotplug lock over
> mem_hotplug_{begin, done}") that write accesses to
> mem_hotplug.active_writer are coordinated via lock_device_hotplug() I'd
> rather propose a new private memory_add_remove_lock which has similar
> semantics like the cpu_add_remove_lock for cpu hotplug (see patch below).
>
> However instead of sprinkling locking/unlocking of that new lock around all
> calls of mem_hotplug_begin() and mem_hotplug_end() simply include locking
> and unlocking into these two functions.
>
> This still allows get_online_mems() and put_online_mems() to work, while at
> the same time preventing mem_hotplug.active_writer corruption.
>
> Any opinions?
Sorry, yes, I didn't make it clear that I derived that locking
requirement from store_mem_state() and its usage of
lock_device_hotplug_sysfs().
That routine is trying very hard not trip the soft-lockup detector. It
seems like that wants to be an interruptible wait.
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [PATCH] mm, add_memory_resource: hold device_hotplug lock over mem_hotplug_{begin, done}
@ 2017-03-01 15:52 ` Dan Williams
0 siblings, 0 replies; 18+ messages in thread
From: Dan Williams @ 2017-03-01 15:52 UTC (permalink / raw)
To: Heiko Carstens
Cc: Michal Hocko, Sebastian Ott, Linux MM, linux-kernel,
Andrew Morton, Rafael J. Wysocki, Vladimir Davydov,
Ben Hutchings
On Wed, Mar 1, 2017 at 4:51 AM, Heiko Carstens
<heiko.carstens@de.ibm.com> wrote:
> On Tue, Feb 28, 2017 at 12:57:29PM +0100, Heiko Carstens wrote:
>> On Mon, Feb 27, 2017 at 05:20:31PM +0100, Michal Hocko wrote:
>> > [CC Rafael]
>> >
>> > I've got lost in the acpi indirection (again). I can see
>> > acpi_device_hotplug calling lock_device_hotplug() but i cannot find a
>> > path down to add_memory() which might call add_memory_resource. But the
>> > patch below sounds suspicious to me. Is it possible that this could lead
>> > to a deadlock. I would suspect that it is the s390 code which needs to
>> > do the locking. But I would have to double check - it is really easy to
>> > get lost there.
>>
>> To me it rather looks like bfc8c90139eb ("mem-hotplug: implement
>> get/put_online_mems") introduced quite subtle and probably wrong locking
>> rules.
>>
>> The patch introduced mem_hotplug_begin() in order to have something like
>> cpu_hotplug_begin() for memory. Note that for cpu hotplug all
>> cpu_hotplug_begin() calls are serialized by cpu_maps_update_begin().
>>
>> Especially this makes sure that active_writer can only be changed by one
>> process. (See also Dan's commit which introduced the lock_device_hotplug()
>> calls: https://marc.info/?l=linux-kernel&m=148693912419972&w=2 )
>>
>> If you look at the above commit bfc8c90139eb: there is nothing like
>> cpu_maps_update_begin() for memory. And therefore it's possible to have
>> concurrent writers to active_writer.
>>
>> It looks like now lock_device_hotplug() is supposed to be the new
>> cpu_maps_update_begin() for memory. But.. this looks like a mess, unless I
>> read the code completely wrong ;)
>
> [Full quote since I now hopefully use a non-bouncing email address from
> Vladimir]
>
> Since it is anything but obvious why Dan wrote in changelog of b5d24fda9c3d
> ("mm, devm_memremap_pages: hold device_hotplug lock over
> mem_hotplug_{begin, done}") that write accesses to
> mem_hotplug.active_writer are coordinated via lock_device_hotplug() I'd
> rather propose a new private memory_add_remove_lock which has similar
> semantics like the cpu_add_remove_lock for cpu hotplug (see patch below).
>
> However instead of sprinkling locking/unlocking of that new lock around all
> calls of mem_hotplug_begin() and mem_hotplug_end() simply include locking
> and unlocking into these two functions.
>
> This still allows get_online_mems() and put_online_mems() to work, while at
> the same time preventing mem_hotplug.active_writer corruption.
>
> Any opinions?
Sorry, yes, I didn't make it clear that I derived that locking
requirement from store_mem_state() and its usage of
lock_device_hotplug_sysfs().
That routine is trying very hard not trip the soft-lockup detector. It
seems like that wants to be an interruptible wait.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [PATCH] mm, add_memory_resource: hold device_hotplug lock over mem_hotplug_{begin, done}
2017-03-01 15:52 ` Dan Williams
@ 2017-03-01 17:04 ` Heiko Carstens
-1 siblings, 0 replies; 18+ messages in thread
From: Heiko Carstens @ 2017-03-01 17:04 UTC (permalink / raw)
To: Dan Williams
Cc: Michal Hocko, Sebastian Ott, Linux MM, linux-kernel,
Andrew Morton, Rafael J. Wysocki, Vladimir Davydov,
Ben Hutchings
On Wed, Mar 01, 2017 at 07:52:18AM -0800, Dan Williams wrote:
> On Wed, Mar 1, 2017 at 4:51 AM, Heiko Carstens
> <heiko.carstens@de.ibm.com> wrote:
> > Since it is anything but obvious why Dan wrote in changelog of b5d24fda9c3d
> > ("mm, devm_memremap_pages: hold device_hotplug lock over
> > mem_hotplug_{begin, done}") that write accesses to
> > mem_hotplug.active_writer are coordinated via lock_device_hotplug() I'd
> > rather propose a new private memory_add_remove_lock which has similar
> > semantics like the cpu_add_remove_lock for cpu hotplug (see patch below).
> >
> > However instead of sprinkling locking/unlocking of that new lock around all
> > calls of mem_hotplug_begin() and mem_hotplug_end() simply include locking
> > and unlocking into these two functions.
> >
> > This still allows get_online_mems() and put_online_mems() to work, while at
> > the same time preventing mem_hotplug.active_writer corruption.
> >
> > Any opinions?
>
> Sorry, yes, I didn't make it clear that I derived that locking
> requirement from store_mem_state() and its usage of
> lock_device_hotplug_sysfs().
>
> That routine is trying very hard not trip the soft-lockup detector. It
> seems like that wants to be an interruptible wait.
If you look at commit 5e33bc4165f3 ("driver core / ACPI: Avoid device hot
remove locking issues") then lock_device_hotplug_sysfs() was introduced to
avoid a different subtle deadlock, but it also sleeps uninterruptible, but
not for more than 5ms ;)
However I'm not sure if the device hotplug lock should also be used to fix
an unrelated bug that was introduced with the get_online_mems() /
put_online_mems() interface. Should it?
If so, we need to sprinkle around a couple of lock_device_hotplug() calls
near mem_hotplug_begin() calls, like Sebastian already started, and give it
additional semantics (protecting mem_hotplug.active_writer), and hope it
doesn't lead to deadlocks anywhere.
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [PATCH] mm, add_memory_resource: hold device_hotplug lock over mem_hotplug_{begin, done}
@ 2017-03-01 17:04 ` Heiko Carstens
0 siblings, 0 replies; 18+ messages in thread
From: Heiko Carstens @ 2017-03-01 17:04 UTC (permalink / raw)
To: Dan Williams
Cc: Michal Hocko, Sebastian Ott, Linux MM, linux-kernel,
Andrew Morton, Rafael J. Wysocki, Vladimir Davydov,
Ben Hutchings
On Wed, Mar 01, 2017 at 07:52:18AM -0800, Dan Williams wrote:
> On Wed, Mar 1, 2017 at 4:51 AM, Heiko Carstens
> <heiko.carstens@de.ibm.com> wrote:
> > Since it is anything but obvious why Dan wrote in changelog of b5d24fda9c3d
> > ("mm, devm_memremap_pages: hold device_hotplug lock over
> > mem_hotplug_{begin, done}") that write accesses to
> > mem_hotplug.active_writer are coordinated via lock_device_hotplug() I'd
> > rather propose a new private memory_add_remove_lock which has similar
> > semantics like the cpu_add_remove_lock for cpu hotplug (see patch below).
> >
> > However instead of sprinkling locking/unlocking of that new lock around all
> > calls of mem_hotplug_begin() and mem_hotplug_end() simply include locking
> > and unlocking into these two functions.
> >
> > This still allows get_online_mems() and put_online_mems() to work, while at
> > the same time preventing mem_hotplug.active_writer corruption.
> >
> > Any opinions?
>
> Sorry, yes, I didn't make it clear that I derived that locking
> requirement from store_mem_state() and its usage of
> lock_device_hotplug_sysfs().
>
> That routine is trying very hard not trip the soft-lockup detector. It
> seems like that wants to be an interruptible wait.
If you look at commit 5e33bc4165f3 ("driver core / ACPI: Avoid device hot
remove locking issues") then lock_device_hotplug_sysfs() was introduced to
avoid a different subtle deadlock, but it also sleeps uninterruptible, but
not for more than 5ms ;)
However I'm not sure if the device hotplug lock should also be used to fix
an unrelated bug that was introduced with the get_online_mems() /
put_online_mems() interface. Should it?
If so, we need to sprinkle around a couple of lock_device_hotplug() calls
near mem_hotplug_begin() calls, like Sebastian already started, and give it
additional semantics (protecting mem_hotplug.active_writer), and hope it
doesn't lead to deadlocks anywhere.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [PATCH] mm, add_memory_resource: hold device_hotplug lock over mem_hotplug_{begin, done}
2017-03-01 17:04 ` Heiko Carstens
@ 2017-03-01 22:55 ` Dan Williams
-1 siblings, 0 replies; 18+ messages in thread
From: Dan Williams @ 2017-03-01 22:55 UTC (permalink / raw)
To: Heiko Carstens
Cc: Michal Hocko, Sebastian Ott, Linux MM, linux-kernel,
Andrew Morton, Rafael J. Wysocki, Vladimir Davydov,
Ben Hutchings
On Wed, Mar 1, 2017 at 9:04 AM, Heiko Carstens
<heiko.carstens@de.ibm.com> wrote:
> On Wed, Mar 01, 2017 at 07:52:18AM -0800, Dan Williams wrote:
>> On Wed, Mar 1, 2017 at 4:51 AM, Heiko Carstens
>> <heiko.carstens@de.ibm.com> wrote:
>> > Since it is anything but obvious why Dan wrote in changelog of b5d24fda9c3d
>> > ("mm, devm_memremap_pages: hold device_hotplug lock over
>> > mem_hotplug_{begin, done}") that write accesses to
>> > mem_hotplug.active_writer are coordinated via lock_device_hotplug() I'd
>> > rather propose a new private memory_add_remove_lock which has similar
>> > semantics like the cpu_add_remove_lock for cpu hotplug (see patch below).
>> >
>> > However instead of sprinkling locking/unlocking of that new lock around all
>> > calls of mem_hotplug_begin() and mem_hotplug_end() simply include locking
>> > and unlocking into these two functions.
>> >
>> > This still allows get_online_mems() and put_online_mems() to work, while at
>> > the same time preventing mem_hotplug.active_writer corruption.
>> >
>> > Any opinions?
>>
>> Sorry, yes, I didn't make it clear that I derived that locking
>> requirement from store_mem_state() and its usage of
>> lock_device_hotplug_sysfs().
>>
>> That routine is trying very hard not trip the soft-lockup detector. It
>> seems like that wants to be an interruptible wait.
>
> If you look at commit 5e33bc4165f3 ("driver core / ACPI: Avoid device hot
> remove locking issues") then lock_device_hotplug_sysfs() was introduced to
> avoid a different subtle deadlock, but it also sleeps uninterruptible, but
> not for more than 5ms ;)
>
> However I'm not sure if the device hotplug lock should also be used to fix
> an unrelated bug that was introduced with the get_online_mems() /
> put_online_mems() interface. Should it?
No, I don't think it should.
I like your proposed direction of creating a new lock internal to
mem_hotplug_begin() to protect active_writer, and stop relying on
lock_device_hotplug to serve this purpose.
> If so, we need to sprinkle around a couple of lock_device_hotplug() calls
> near mem_hotplug_begin() calls, like Sebastian already started, and give it
> additional semantics (protecting mem_hotplug.active_writer), and hope it
> doesn't lead to deadlocks anywhere.
I'll put your proposed patch through some testing.
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [PATCH] mm, add_memory_resource: hold device_hotplug lock over mem_hotplug_{begin, done}
@ 2017-03-01 22:55 ` Dan Williams
0 siblings, 0 replies; 18+ messages in thread
From: Dan Williams @ 2017-03-01 22:55 UTC (permalink / raw)
To: Heiko Carstens
Cc: Michal Hocko, Sebastian Ott, Linux MM, linux-kernel,
Andrew Morton, Rafael J. Wysocki, Vladimir Davydov,
Ben Hutchings
On Wed, Mar 1, 2017 at 9:04 AM, Heiko Carstens
<heiko.carstens@de.ibm.com> wrote:
> On Wed, Mar 01, 2017 at 07:52:18AM -0800, Dan Williams wrote:
>> On Wed, Mar 1, 2017 at 4:51 AM, Heiko Carstens
>> <heiko.carstens@de.ibm.com> wrote:
>> > Since it is anything but obvious why Dan wrote in changelog of b5d24fda9c3d
>> > ("mm, devm_memremap_pages: hold device_hotplug lock over
>> > mem_hotplug_{begin, done}") that write accesses to
>> > mem_hotplug.active_writer are coordinated via lock_device_hotplug() I'd
>> > rather propose a new private memory_add_remove_lock which has similar
>> > semantics like the cpu_add_remove_lock for cpu hotplug (see patch below).
>> >
>> > However instead of sprinkling locking/unlocking of that new lock around all
>> > calls of mem_hotplug_begin() and mem_hotplug_end() simply include locking
>> > and unlocking into these two functions.
>> >
>> > This still allows get_online_mems() and put_online_mems() to work, while at
>> > the same time preventing mem_hotplug.active_writer corruption.
>> >
>> > Any opinions?
>>
>> Sorry, yes, I didn't make it clear that I derived that locking
>> requirement from store_mem_state() and its usage of
>> lock_device_hotplug_sysfs().
>>
>> That routine is trying very hard not trip the soft-lockup detector. It
>> seems like that wants to be an interruptible wait.
>
> If you look at commit 5e33bc4165f3 ("driver core / ACPI: Avoid device hot
> remove locking issues") then lock_device_hotplug_sysfs() was introduced to
> avoid a different subtle deadlock, but it also sleeps uninterruptible, but
> not for more than 5ms ;)
>
> However I'm not sure if the device hotplug lock should also be used to fix
> an unrelated bug that was introduced with the get_online_mems() /
> put_online_mems() interface. Should it?
No, I don't think it should.
I like your proposed direction of creating a new lock internal to
mem_hotplug_begin() to protect active_writer, and stop relying on
lock_device_hotplug to serve this purpose.
> If so, we need to sprinkle around a couple of lock_device_hotplug() calls
> near mem_hotplug_begin() calls, like Sebastian already started, and give it
> additional semantics (protecting mem_hotplug.active_writer), and hope it
> doesn't lead to deadlocks anywhere.
I'll put your proposed patch through some testing.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [PATCH] mm, add_memory_resource: hold device_hotplug lock over mem_hotplug_{begin, done}
2017-03-01 22:55 ` Dan Williams
@ 2017-03-06 8:22 ` Heiko Carstens
-1 siblings, 0 replies; 18+ messages in thread
From: Heiko Carstens @ 2017-03-06 8:22 UTC (permalink / raw)
To: Dan Williams
Cc: Michal Hocko, Sebastian Ott, Linux MM, linux-kernel,
Andrew Morton, Rafael J. Wysocki, Vladimir Davydov,
Ben Hutchings
Hello Dan,
> > If you look at commit 5e33bc4165f3 ("driver core / ACPI: Avoid device hot
> > remove locking issues") then lock_device_hotplug_sysfs() was introduced to
> > avoid a different subtle deadlock, but it also sleeps uninterruptible, but
> > not for more than 5ms ;)
> >
> > However I'm not sure if the device hotplug lock should also be used to fix
> > an unrelated bug that was introduced with the get_online_mems() /
> > put_online_mems() interface. Should it?
>
> No, I don't think it should.
>
> I like your proposed direction of creating a new lock internal to
> mem_hotplug_begin() to protect active_writer, and stop relying on
> lock_device_hotplug to serve this purpose.
>
> > If so, we need to sprinkle around a couple of lock_device_hotplug() calls
> > near mem_hotplug_begin() calls, like Sebastian already started, and give it
> > additional semantics (protecting mem_hotplug.active_writer), and hope it
> > doesn't lead to deadlocks anywhere.
>
> I'll put your proposed patch through some testing.
On s390 it _seems_ to work. Did it pass your testing too?
If so I would send a patch with proper patch description for inclusion.
Thanks,
Heiko
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [PATCH] mm, add_memory_resource: hold device_hotplug lock over mem_hotplug_{begin, done}
@ 2017-03-06 8:22 ` Heiko Carstens
0 siblings, 0 replies; 18+ messages in thread
From: Heiko Carstens @ 2017-03-06 8:22 UTC (permalink / raw)
To: Dan Williams
Cc: Michal Hocko, Sebastian Ott, Linux MM, linux-kernel,
Andrew Morton, Rafael J. Wysocki, Vladimir Davydov,
Ben Hutchings
Hello Dan,
> > If you look at commit 5e33bc4165f3 ("driver core / ACPI: Avoid device hot
> > remove locking issues") then lock_device_hotplug_sysfs() was introduced to
> > avoid a different subtle deadlock, but it also sleeps uninterruptible, but
> > not for more than 5ms ;)
> >
> > However I'm not sure if the device hotplug lock should also be used to fix
> > an unrelated bug that was introduced with the get_online_mems() /
> > put_online_mems() interface. Should it?
>
> No, I don't think it should.
>
> I like your proposed direction of creating a new lock internal to
> mem_hotplug_begin() to protect active_writer, and stop relying on
> lock_device_hotplug to serve this purpose.
>
> > If so, we need to sprinkle around a couple of lock_device_hotplug() calls
> > near mem_hotplug_begin() calls, like Sebastian already started, and give it
> > additional semantics (protecting mem_hotplug.active_writer), and hope it
> > doesn't lead to deadlocks anywhere.
>
> I'll put your proposed patch through some testing.
On s390 it _seems_ to work. Did it pass your testing too?
If so I would send a patch with proper patch description for inclusion.
Thanks,
Heiko
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [PATCH] mm, add_memory_resource: hold device_hotplug lock over mem_hotplug_{begin, done}
2017-03-06 8:22 ` Heiko Carstens
@ 2017-03-09 6:26 ` Dan Williams
-1 siblings, 0 replies; 18+ messages in thread
From: Dan Williams @ 2017-03-09 6:26 UTC (permalink / raw)
To: Heiko Carstens
Cc: Michal Hocko, Sebastian Ott, Linux MM, linux-kernel,
Andrew Morton, Rafael J. Wysocki, Vladimir Davydov,
Ben Hutchings
On Mon, Mar 6, 2017 at 12:22 AM, Heiko Carstens
<heiko.carstens@de.ibm.com> wrote:
> Hello Dan,
>
>> > If you look at commit 5e33bc4165f3 ("driver core / ACPI: Avoid device hot
>> > remove locking issues") then lock_device_hotplug_sysfs() was introduced to
>> > avoid a different subtle deadlock, but it also sleeps uninterruptible, but
>> > not for more than 5ms ;)
>> >
>> > However I'm not sure if the device hotplug lock should also be used to fix
>> > an unrelated bug that was introduced with the get_online_mems() /
>> > put_online_mems() interface. Should it?
>>
>> No, I don't think it should.
>>
>> I like your proposed direction of creating a new lock internal to
>> mem_hotplug_begin() to protect active_writer, and stop relying on
>> lock_device_hotplug to serve this purpose.
>>
>> > If so, we need to sprinkle around a couple of lock_device_hotplug() calls
>> > near mem_hotplug_begin() calls, like Sebastian already started, and give it
>> > additional semantics (protecting mem_hotplug.active_writer), and hope it
>> > doesn't lead to deadlocks anywhere.
>>
>> I'll put your proposed patch through some testing.
>
> On s390 it _seems_ to work. Did it pass your testing too?
> If so I would send a patch with proper patch description for inclusion.
Looks ok here. No lockdep warnings running it through it paces with
the persistent memory use case.
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [PATCH] mm, add_memory_resource: hold device_hotplug lock over mem_hotplug_{begin, done}
@ 2017-03-09 6:26 ` Dan Williams
0 siblings, 0 replies; 18+ messages in thread
From: Dan Williams @ 2017-03-09 6:26 UTC (permalink / raw)
To: Heiko Carstens
Cc: Michal Hocko, Sebastian Ott, Linux MM, linux-kernel,
Andrew Morton, Rafael J. Wysocki, Vladimir Davydov,
Ben Hutchings
On Mon, Mar 6, 2017 at 12:22 AM, Heiko Carstens
<heiko.carstens@de.ibm.com> wrote:
> Hello Dan,
>
>> > If you look at commit 5e33bc4165f3 ("driver core / ACPI: Avoid device hot
>> > remove locking issues") then lock_device_hotplug_sysfs() was introduced to
>> > avoid a different subtle deadlock, but it also sleeps uninterruptible, but
>> > not for more than 5ms ;)
>> >
>> > However I'm not sure if the device hotplug lock should also be used to fix
>> > an unrelated bug that was introduced with the get_online_mems() /
>> > put_online_mems() interface. Should it?
>>
>> No, I don't think it should.
>>
>> I like your proposed direction of creating a new lock internal to
>> mem_hotplug_begin() to protect active_writer, and stop relying on
>> lock_device_hotplug to serve this purpose.
>>
>> > If so, we need to sprinkle around a couple of lock_device_hotplug() calls
>> > near mem_hotplug_begin() calls, like Sebastian already started, and give it
>> > additional semantics (protecting mem_hotplug.active_writer), and hope it
>> > doesn't lead to deadlocks anywhere.
>>
>> I'll put your proposed patch through some testing.
>
> On s390 it _seems_ to work. Did it pass your testing too?
> If so I would send a patch with proper patch description for inclusion.
Looks ok here. No lockdep warnings running it through it paces with
the persistent memory use case.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 18+ messages in thread
end of thread, other threads:[~2017-03-09 6:36 UTC | newest]
Thread overview: 18+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2017-02-26 11:42 [PATCH] mm, add_memory_resource: hold device_hotplug lock over mem_hotplug_{begin, done} Sebastian Ott
2017-02-26 11:42 ` Sebastian Ott
2017-02-27 16:20 ` Michal Hocko
2017-02-27 16:20 ` Michal Hocko
2017-02-28 11:57 ` Heiko Carstens
2017-02-28 11:57 ` Heiko Carstens
2017-03-01 12:51 ` Heiko Carstens
2017-03-01 12:51 ` Heiko Carstens
2017-03-01 15:52 ` Dan Williams
2017-03-01 15:52 ` Dan Williams
2017-03-01 17:04 ` Heiko Carstens
2017-03-01 17:04 ` Heiko Carstens
2017-03-01 22:55 ` Dan Williams
2017-03-01 22:55 ` Dan Williams
2017-03-06 8:22 ` Heiko Carstens
2017-03-06 8:22 ` Heiko Carstens
2017-03-09 6:26 ` Dan Williams
2017-03-09 6:26 ` Dan Williams
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.