[PATCH] malloc: notify primary process about hotplug in secondary

[PATCH] malloc: notify primary process about hotplug in secondary

[Seth Howell]

When secondary process hotplugs memory, it sends a request
to primary, which then performs the real mmap() and sends
sync requests to all secondary processes. Upon receiving
such sync request, each secondary process will notify the
upper layers of hotplugged memory (and will call all
locally registered event callbacks).

In the end we'll end up with memory event callbacks fired
in all the processes except the primary, which is a bug.

This gets critical if memory is hotplugged while a VFIO
device is attached, as the VFIO memory registration -
which is done from a memory event callback present in the
primary process only - is never called.

After this patch, a primary process fires memory event
callbacks before secondary processes start their
synchronizations - both for hotplug and hotremove.

Change-Id: I60de33913f58bc2454069c3844826c92cb043fff
Signed-off-by: Seth Howell <seth.howell@intel.com>
Signed-off-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
---
 lib/librte_eal/common/malloc_mp.c | 8 ++++++++
 1 file changed, 8 insertions(+)

diff --git a/lib/librte_eal/common/malloc_mp.c b/lib/librte_eal/common/malloc_mp.c
index 5f2d4e0be..f3a13353b 100644
--- a/lib/librte_eal/common/malloc_mp.c
+++ b/lib/librte_eal/common/malloc_mp.c
@@ -209,6 +209,8 @@ handle_alloc_request(const struct malloc_mp_req *m,
 
 	map_addr = ms[0]->addr;
 
+	eal_memalloc_mem_event_notify(RTE_MEM_EVENT_ALLOC, map_addr, alloc_sz);
+
 	/* we have succeeded in allocating memory, but we still need to sync
 	 * with other processes. however, since DPDK IPC is single-threaded, we
 	 * send an asynchronous request and exit this callback.
@@ -258,6 +260,9 @@ handle_request(const struct rte_mp_msg *msg, const void *peer __rte_unused)
 	if (m->t == REQ_TYPE_ALLOC) {
 		ret = handle_alloc_request(m, entry);
 	} else if (m->t == REQ_TYPE_FREE) {
+		eal_memalloc_mem_event_notify(RTE_MEM_EVENT_FREE,
+				m->free_req.addr, m->free_req.len);
+
 		ret = malloc_heap_free_pages(m->free_req.addr,
 				m->free_req.len);
 	} else {
@@ -436,6 +441,9 @@ handle_sync_response(const struct rte_mp_msg *request,
 		memset(&rb_msg, 0, sizeof(rb_msg));
 
 		/* we've failed to sync, so do a rollback */
+		eal_memalloc_mem_event_notify(RTE_MEM_EVENT_FREE,
+				state->map_addr, state->map_len);
+
 		rollback_expand_heap(state->ms, state->ms_len, state->elem,
 				state->map_addr, state->map_len);
 
-- 
2.17.2

Re: [PATCH] malloc: notify primary process about hotplug in secondary

[Burakov, Anatoly]

On 04-Dec-18 5:06 PM, Seth Howell wrote:
> When secondary process hotplugs memory, it sends a request
> to primary, which then performs the real mmap() and sends
> sync requests to all secondary processes. Upon receiving
> such sync request, each secondary process will notify the
> upper layers of hotplugged memory (and will call all
> locally registered event callbacks).
> 
> In the end we'll end up with memory event callbacks fired
> in all the processes except the primary, which is a bug.
> 
> This gets critical if memory is hotplugged while a VFIO
> device is attached, as the VFIO memory registration -
> which is done from a memory event callback present in the
> primary process only - is never called.
> 
> After this patch, a primary process fires memory event
> callbacks before secondary processes start their
> synchronizations - both for hotplug and hotremove.
> 
> Change-Id: I60de33913f58bc2454069c3844826c92cb043fff

This is internal tag, please don't include it in the patch. Also, misses 
Fixes: tag, and Cc: stable@dpdk.org (since this bug goes back to 18.05).

> Signed-off-by: Seth Howell <seth.howell@intel.com>
> Signed-off-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
> ---

Otherwise,

Reviewed-by: Anatoly Burakov <anatoly.burakov@intel.com>

-- 
Thanks,
Anatoly

[PATCH] malloc: notify primary process about hotplug in secondary

[Seth Howell]

When secondary process hotplugs memory, it sends a request
to primary, which then performs the real mmap() and sends
sync requests to all secondary processes. Upon receiving
such sync request, each secondary process will notify the
upper layers of hotplugged memory (and will call all
locally registered event callbacks).

In the end we'll end up with memory event callbacks fired
in all the processes except the primary, which is a bug.

This gets critical if memory is hotplugged while a VFIO
device is attached, as the VFIO memory registration -
which is done from a memory event callback present in the
primary process only - is never called.

After this patch, a primary process fires memory event
callbacks before secondary processes start their
synchronizations - both for hotplug and hotremove.

Fixes: 07dcbfe0101f ("malloc: support multiprocess memory hotplug")

Signed-off-by: Seth Howell <seth.howell@intel.com>
Signed-off-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
---
 lib/librte_eal/common/malloc_mp.c | 8 ++++++++
 1 file changed, 8 insertions(+)

diff --git a/lib/librte_eal/common/malloc_mp.c b/lib/librte_eal/common/malloc_mp.c
index 5f2d4e0be..f3a13353b 100644
--- a/lib/librte_eal/common/malloc_mp.c
+++ b/lib/librte_eal/common/malloc_mp.c
@@ -209,6 +209,8 @@ handle_alloc_request(const struct malloc_mp_req *m,
 
 	map_addr = ms[0]->addr;
 
+	eal_memalloc_mem_event_notify(RTE_MEM_EVENT_ALLOC, map_addr, alloc_sz);
+
 	/* we have succeeded in allocating memory, but we still need to sync
 	 * with other processes. however, since DPDK IPC is single-threaded, we
 	 * send an asynchronous request and exit this callback.
@@ -258,6 +260,9 @@ handle_request(const struct rte_mp_msg *msg, const void *peer __rte_unused)
 	if (m->t == REQ_TYPE_ALLOC) {
 		ret = handle_alloc_request(m, entry);
 	} else if (m->t == REQ_TYPE_FREE) {
+		eal_memalloc_mem_event_notify(RTE_MEM_EVENT_FREE,
+				m->free_req.addr, m->free_req.len);
+
 		ret = malloc_heap_free_pages(m->free_req.addr,
 				m->free_req.len);
 	} else {
@@ -436,6 +441,9 @@ handle_sync_response(const struct rte_mp_msg *request,
 		memset(&rb_msg, 0, sizeof(rb_msg));
 
 		/* we've failed to sync, so do a rollback */
+		eal_memalloc_mem_event_notify(RTE_MEM_EVENT_FREE,
+				state->map_addr, state->map_len);
+
 		rollback_expand_heap(state->ms, state->ms_len, state->elem,
 				state->map_addr, state->map_len);
 
-- 
2.17.2

[PATCH v2] malloc: notify primary process about hotplug in secondary

[Seth Howell]

When secondary process hotplugs memory, it sends a request
to primary, which then performs the real mmap() and sends
sync requests to all secondary processes. Upon receiving
such sync request, each secondary process will notify the
upper layers of hotplugged memory (and will call all
locally registered event callbacks).

In the end we'll end up with memory event callbacks fired
in all the processes except the primary, which is a bug.

This gets critical if memory is hotplugged while a VFIO
device is attached, as the VFIO memory registration -
which is done from a memory event callback present in the
primary process only - is never called.

After this patch, a primary process fires memory event
callbacks before secondary processes start their
synchronizations - both for hotplug and hotremove.

Fixes: 07dcbfe0101f ("malloc: support multiprocess memory hotplug")

Signed-off-by: Seth Howell <seth.howell@intel.com>
Signed-off-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
---
 lib/librte_eal/common/malloc_mp.c | 8 ++++++++
 1 file changed, 8 insertions(+)

diff --git a/lib/librte_eal/common/malloc_mp.c b/lib/librte_eal/common/malloc_mp.c
index 5f2d4e0be..f3a13353b 100644
--- a/lib/librte_eal/common/malloc_mp.c
+++ b/lib/librte_eal/common/malloc_mp.c
@@ -209,6 +209,8 @@ handle_alloc_request(const struct malloc_mp_req *m,
 
 	map_addr = ms[0]->addr;
 
+	eal_memalloc_mem_event_notify(RTE_MEM_EVENT_ALLOC, map_addr, alloc_sz);
+
 	/* we have succeeded in allocating memory, but we still need to sync
 	 * with other processes. however, since DPDK IPC is single-threaded, we
 	 * send an asynchronous request and exit this callback.
@@ -258,6 +260,9 @@ handle_request(const struct rte_mp_msg *msg, const void *peer __rte_unused)
 	if (m->t == REQ_TYPE_ALLOC) {
 		ret = handle_alloc_request(m, entry);
 	} else if (m->t == REQ_TYPE_FREE) {
+		eal_memalloc_mem_event_notify(RTE_MEM_EVENT_FREE,
+				m->free_req.addr, m->free_req.len);
+
 		ret = malloc_heap_free_pages(m->free_req.addr,
 				m->free_req.len);
 	} else {
@@ -436,6 +441,9 @@ handle_sync_response(const struct rte_mp_msg *request,
 		memset(&rb_msg, 0, sizeof(rb_msg));
 
 		/* we've failed to sync, so do a rollback */
+		eal_memalloc_mem_event_notify(RTE_MEM_EVENT_FREE,
+				state->map_addr, state->map_len);
+
 		rollback_expand_heap(state->ms, state->ms_len, state->elem,
 				state->map_addr, state->map_len);
 
-- 
2.17.2

Re: [PATCH v2] malloc: notify primary process about hotplug in secondary

[Burakov, Anatoly]

On 07-Dec-18 8:10 PM, Seth Howell wrote:
> When secondary process hotplugs memory, it sends a request
> to primary, which then performs the real mmap() and sends
> sync requests to all secondary processes. Upon receiving
> such sync request, each secondary process will notify the
> upper layers of hotplugged memory (and will call all
> locally registered event callbacks).
> 
> In the end we'll end up with memory event callbacks fired
> in all the processes except the primary, which is a bug.
> 
> This gets critical if memory is hotplugged while a VFIO
> device is attached, as the VFIO memory registration -
> which is done from a memory event callback present in the
> primary process only - is never called.
> 
> After this patch, a primary process fires memory event
> callbacks before secondary processes start their
> synchronizations - both for hotplug and hotremove.
> 
> Fixes: 07dcbfe0101f ("malloc: support multiprocess memory hotplug")
> 
> Signed-off-by: Seth Howell <seth.howell@intel.com>
> Signed-off-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
> ---

Lost my review tag...

Reviewed-by: Anatoly Burakov <anatoly.burakov@intel.com>

-- 
Thanks,
Anatoly

Re: [dpdk-stable] [PATCH v2] malloc: notify primary process about hotplug in secondary

[Thomas Monjalon]

10/12/2018 11:49, Burakov, Anatoly:
> On 07-Dec-18 8:10 PM, Seth Howell wrote:
> > When secondary process hotplugs memory, it sends a request
> > to primary, which then performs the real mmap() and sends
> > sync requests to all secondary processes. Upon receiving
> > such sync request, each secondary process will notify the
> > upper layers of hotplugged memory (and will call all
> > locally registered event callbacks).
> > 
> > In the end we'll end up with memory event callbacks fired
> > in all the processes except the primary, which is a bug.
> > 
> > This gets critical if memory is hotplugged while a VFIO
> > device is attached, as the VFIO memory registration -
> > which is done from a memory event callback present in the
> > primary process only - is never called.
> > 
> > After this patch, a primary process fires memory event
> > callbacks before secondary processes start their
> > synchronizations - both for hotplug and hotremove.
> > 
> > Fixes: 07dcbfe0101f ("malloc: support multiprocess memory hotplug")
> > 
> > Signed-off-by: Seth Howell <seth.howell@intel.com>
> > Signed-off-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
> > ---
> 
> Lost my review tag...
> 
> Reviewed-by: Anatoly Burakov <anatoly.burakov@intel.com>

+Cc: stable@dpdk.org

Applied, thanks