[PATCH 0/2] ppc:Fix the memory initialization for IBM POWER

[PATCH 0/2] ppc:Fix the memory initialization for IBM POWER

[Chao Zhu]

Due to mmap implementation on IBM POWER, the secondary process memory
initialization may fail (mmap will not respect the required address hints).
This patch sets add the fixes and guides to fix this problem.

Chao Zhu (2):
  eal/ppc: fix mmap for memory initialization
  doc/guides: Add hugepage reserve instructions for IBM POWER

 doc/guides/linux_gsg/sys_reqs.rst        | 6 ++++++
 lib/librte_eal/linuxapp/eal/eal_memory.c | 8 ++++++++
 2 files changed, 14 insertions(+)

-- 
1.8.3.1

[PATCH 1/2] eal/ppc: fix mmap for memory initialization

[Chao Zhu]

On IBM POWER platform, when mapping /dev/zero file to hugepage memory
space, mmap will not respect the requested address hint. This will cause
the memory initilization for the second process fails. This patch adds
the required mmap flags to make it work. Beside this, users need to set
the nr_overcommit_hugepages to expand the VA range. When
doing the initilization, users need to set both nr_hugepages and
nr_overcommit_hugepages to the same value, like 64, 128, etc.

Signed-off-by: Chao Zhu <chaozhu@linux.vnet.ibm.com>
---
 lib/librte_eal/linuxapp/eal/eal_memory.c | 8 ++++++++
 1 file changed, 8 insertions(+)

diff --git a/lib/librte_eal/linuxapp/eal/eal_memory.c b/lib/librte_eal/linuxapp/eal/eal_memory.c
index a956bb2..e06186b 100644
--- a/lib/librte_eal/linuxapp/eal/eal_memory.c
+++ b/lib/librte_eal/linuxapp/eal/eal_memory.c
@@ -313,7 +313,11 @@ int rte_xen_dom0_supported(void)
 	}
 	do {
 		addr = mmap(addr,
+#ifndef RTE_ARCH_PPC_64
 				(*size) + hugepage_sz, PROT_READ, MAP_PRIVATE, fd, 0);
+#else
+                (*size) + hugepage_sz, PROT_READ, MAP_PRIVATE | MAP_ANONYMOUS | MAP_HUGETLB, fd, 0);
+#endif
 		if (addr == MAP_FAILED)
 			*size -= hugepage_sz;
 	} while (addr == MAP_FAILED && *size > 0);
@@ -1330,7 +1334,11 @@ static int huge_wrap_sigsetjmp(void)
 		 * use mmap to get identical addresses as the primary process.
 		 */
 		base_addr = mmap(mcfg->memseg[s].addr, mcfg->memseg[s].len,
+#ifndef RTE_ARCH_PPC_64
 				 PROT_READ, MAP_PRIVATE, fd_zero, 0);
+#else
+                 PROT_READ, MAP_PRIVATE | MAP_ANONYMOUS | MAP_HUGETLB, fd_zero, 0);
+#endif
 		if (base_addr == MAP_FAILED ||
 		    base_addr != mcfg->memseg[s].addr) {
 			max_seg = s;
-- 
1.8.3.1

Re: [PATCH 1/2] eal/ppc: fix mmap for memory initialization

[Sergio Gonzalez Monroy]

Hi Chao,

You mentioned that 'mmap will not respect the requested address hint', 
how does the proposed change solves that?

Is it that hugepages map to a specific VA region, and without 
MAP_HUGETLB you may get address from wrong region?

If mmap were to respect the hinted address, we could do this change 
multi-arch without having to set overcommit hugepages?

fd = -1
addr = mmap(addr, (*size) + hugepage_sz, PROT_READ, MAP_PRIVATE | 
MAP_ANONYMOUS | MAP_HUGETLB, fd, 0)
# Free hugepages mapping
addr = mmap(addr, (*size) + hugepage_sz, PROT_READ, MAP_PRIVATE | 
MAP_ANONYMOUS, fd, 0)

What do you think?

Regards,
Sergio

On 06/04/2017 11:06, Chao Zhu wrote:
> On IBM POWER platform, when mapping /dev/zero file to hugepage memory
> space, mmap will not respect the requested address hint. This will cause
> the memory initilization for the second process fails. This patch adds
> the required mmap flags to make it work. Beside this, users need to set
> the nr_overcommit_hugepages to expand the VA range. When
> doing the initilization, users need to set both nr_hugepages and
> nr_overcommit_hugepages to the same value, like 64, 128, etc.
>
> Signed-off-by: Chao Zhu <chaozhu@linux.vnet.ibm.com>
> ---
>   lib/librte_eal/linuxapp/eal/eal_memory.c | 8 ++++++++
>   1 file changed, 8 insertions(+)
>
> diff --git a/lib/librte_eal/linuxapp/eal/eal_memory.c b/lib/librte_eal/linuxapp/eal/eal_memory.c
> index a956bb2..e06186b 100644
> --- a/lib/librte_eal/linuxapp/eal/eal_memory.c
> +++ b/lib/librte_eal/linuxapp/eal/eal_memory.c
> @@ -313,7 +313,11 @@ int rte_xen_dom0_supported(void)
>   	}
>   	do {
>   		addr = mmap(addr,
> +#ifndef RTE_ARCH_PPC_64
>   				(*size) + hugepage_sz, PROT_READ, MAP_PRIVATE, fd, 0);
> +#else
> +                (*size) + hugepage_sz, PROT_READ, MAP_PRIVATE | MAP_ANONYMOUS | MAP_HUGETLB, fd, 0);
> +#endif
>   		if (addr == MAP_FAILED)
>   			*size -= hugepage_sz;
>   	} while (addr == MAP_FAILED && *size > 0);
> @@ -1330,7 +1334,11 @@ static int huge_wrap_sigsetjmp(void)
>   		 * use mmap to get identical addresses as the primary process.
>   		 */
>   		base_addr = mmap(mcfg->memseg[s].addr, mcfg->memseg[s].len,
> +#ifndef RTE_ARCH_PPC_64
>   				 PROT_READ, MAP_PRIVATE, fd_zero, 0);
> +#else
> +                 PROT_READ, MAP_PRIVATE | MAP_ANONYMOUS | MAP_HUGETLB, fd_zero, 0);
> +#endif
>   		if (base_addr == MAP_FAILED ||
>   		    base_addr != mcfg->memseg[s].addr) {
>   			max_seg = s;

Re: [PATCH 1/2] eal/ppc: fix mmap for memory initialization

[Chao Zhu]

Sergio,

Thanks for the comments!
On POWER, if it doesn't specify the MAP_HUGETLB flag when doing mapping, it
may get the addresses from other regions. However, the address space size of
hugepages is exactly the same as the value specified by user when doing
initialization. There will be not enough space for mmap twice in DPDK.
That's why we need to set the overcommit to expand the address space.

> -----Original Message-----
> From: Sergio Gonzalez Monroy [mailto:sergio.gonzalez.monroy@intel.com]
> Sent: 2017年4月6日 20:59
> To: Chao Zhu <chaozhu@linux.vnet.ibm.com>; dev@dpdk.org
> Cc: Gowrishankar <gowrishankar.m@linux.vnet.ibm.com>;
> david.marchand@6wind.com
> Subject: Re: [PATCH 1/2] eal/ppc: fix mmap for memory initialization
> 
> Hi Chao,
> 
> You mentioned that 'mmap will not respect the requested address hint', how
> does the proposed change solves that?
> 
> Is it that hugepages map to a specific VA region, and without MAP_HUGETLB
> you may get address from wrong region?
> 
> If mmap were to respect the hinted address, we could do this change
> multi-arch without having to set overcommit hugepages?
> 
> fd = -1
> addr = mmap(addr, (*size) + hugepage_sz, PROT_READ, MAP_PRIVATE |
> MAP_ANONYMOUS | MAP_HUGETLB, fd, 0) # Free hugepages mapping addr =
> mmap(addr, (*size) + hugepage_sz, PROT_READ, MAP_PRIVATE |
> MAP_ANONYMOUS, fd, 0)
> 
> What do you think?
> 
> Regards,
> Sergio
> 
> On 06/04/2017 11:06, Chao Zhu wrote:
> > On IBM POWER platform, when mapping /dev/zero file to hugepage memory
> > space, mmap will not respect the requested address hint. This will
> > cause the memory initilization for the second process fails. This
> > patch adds the required mmap flags to make it work. Beside this, users
> > need to set the nr_overcommit_hugepages to expand the VA range. When
> > doing the initilization, users need to set both nr_hugepages and
> > nr_overcommit_hugepages to the same value, like 64, 128, etc.
> >
> > Signed-off-by: Chao Zhu <chaozhu@linux.vnet.ibm.com>
> > ---
> >   lib/librte_eal/linuxapp/eal/eal_memory.c | 8 ++++++++
> >   1 file changed, 8 insertions(+)
> >
> > diff --git a/lib/librte_eal/linuxapp/eal/eal_memory.c
> > b/lib/librte_eal/linuxapp/eal/eal_memory.c
> > index a956bb2..e06186b 100644
> > --- a/lib/librte_eal/linuxapp/eal/eal_memory.c
> > +++ b/lib/librte_eal/linuxapp/eal/eal_memory.c
> > @@ -313,7 +313,11 @@ int rte_xen_dom0_supported(void)
> >   	}
> >   	do {
> >   		addr = mmap(addr,
> > +#ifndef RTE_ARCH_PPC_64
> >   				(*size) + hugepage_sz, PROT_READ,
MAP_PRIVATE, fd,
> 0);
> > +#else
> > +                (*size) + hugepage_sz, PROT_READ, MAP_PRIVATE |
> > +MAP_ANONYMOUS | MAP_HUGETLB, fd, 0); #endif
> >   		if (addr == MAP_FAILED)
> >   			*size -= hugepage_sz;
> >   	} while (addr == MAP_FAILED && *size > 0); @@ -1330,7 +1334,11
> @@
> > static int huge_wrap_sigsetjmp(void)
> >   		 * use mmap to get identical addresses as the primary
process.
> >   		 */
> >   		base_addr = mmap(mcfg->memseg[s].addr,
> mcfg->memseg[s].len,
> > +#ifndef RTE_ARCH_PPC_64
> >   				 PROT_READ, MAP_PRIVATE, fd_zero, 0);
> > +#else
> > +                 PROT_READ, MAP_PRIVATE | MAP_ANONYMOUS |
> > +MAP_HUGETLB, fd_zero, 0); #endif
> >   		if (base_addr == MAP_FAILED ||
> >   		    base_addr != mcfg->memseg[s].addr) {
> >   			max_seg = s;
> 

Re: [PATCH 1/2] eal/ppc: fix mmap for memory initialization

[Sergio Gonzalez Monroy]

On 06/04/2017 11:06, Chao Zhu wrote:
> On IBM POWER platform, when mapping /dev/zero file to hugepage memory
> space, mmap will not respect the requested address hint. This will cause
> the memory initilization for the second process fails. This patch adds
> the required mmap flags to make it work. Beside this, users need to set
> the nr_overcommit_hugepages to expand the VA range. When
> doing the initilization, users need to set both nr_hugepages and
> nr_overcommit_hugepages to the same value, like 64, 128, etc.
>
> Signed-off-by: Chao Zhu <chaozhu@linux.vnet.ibm.com>
> ---
>   lib/librte_eal/linuxapp/eal/eal_memory.c | 8 ++++++++
>   1 file changed, 8 insertions(+)
>
> diff --git a/lib/librte_eal/linuxapp/eal/eal_memory.c b/lib/librte_eal/linuxapp/eal/eal_memory.c
> index a956bb2..e06186b 100644
> --- a/lib/librte_eal/linuxapp/eal/eal_memory.c
> +++ b/lib/librte_eal/linuxapp/eal/eal_memory.c
> @@ -313,7 +313,11 @@ int rte_xen_dom0_supported(void)
>   	}
>   	do {
>   		addr = mmap(addr,
> +#ifndef RTE_ARCH_PPC_64
>   				(*size) + hugepage_sz, PROT_READ, MAP_PRIVATE, fd, 0);
> +#else
> +                (*size) + hugepage_sz, PROT_READ, MAP_PRIVATE | MAP_ANONYMOUS | MAP_HUGETLB, fd, 0);
> +#endif
>   		if (addr == MAP_FAILED)
>   			*size -= hugepage_sz;
>   	} while (addr == MAP_FAILED && *size > 0);
> @@ -1330,7 +1334,11 @@ static int huge_wrap_sigsetjmp(void)
>   		 * use mmap to get identical addresses as the primary process.
>   		 */
>   		base_addr = mmap(mcfg->memseg[s].addr, mcfg->memseg[s].len,
> +#ifndef RTE_ARCH_PPC_64
>   				 PROT_READ, MAP_PRIVATE, fd_zero, 0);
> +#else
> +                 PROT_READ, MAP_PRIVATE | MAP_ANONYMOUS | MAP_HUGETLB, fd_zero, 0);
> +#endif
>   		if (base_addr == MAP_FAILED ||
>   		    base_addr != mcfg->memseg[s].addr) {
>   			max_seg = s;


Acked-by: Sergio Gonzalez Monroy <sergio.gonzalez.monroy@intel.com>

Re: [PATCH 1/2] eal/ppc: fix mmap for memory initialization

[Thomas Monjalon]

13/04/2017 10:14, Sergio Gonzalez Monroy:
> On 06/04/2017 11:06, Chao Zhu wrote:
> > On IBM POWER platform, when mapping /dev/zero file to hugepage memory
> > space, mmap will not respect the requested address hint. This will cause
> > the memory initilization for the second process fails. This patch adds
> > the required mmap flags to make it work. Beside this, users need to set
> > the nr_overcommit_hugepages to expand the VA range. When
> > doing the initilization, users need to set both nr_hugepages and
> > nr_overcommit_hugepages to the same value, like 64, 128, etc.
> > 
> > Signed-off-by: Chao Zhu <chaozhu@linux.vnet.ibm.com>
> 
> Acked-by: Sergio Gonzalez Monroy <sergio.gonzalez.monroy@intel.com>

Series fixed, squashed and applied, thanks

Re: [PATCH 1/2] eal/ppc: fix mmap for memory initialization

[Thomas Monjalon]

06/04/2017 12:06, Chao Zhu:
> On IBM POWER platform, when mapping /dev/zero file to hugepage memory
> space, mmap will not respect the requested address hint. This will cause
> the memory initilization for the second process fails. This patch adds
> the required mmap flags to make it work. Beside this, users need to set
> the nr_overcommit_hugepages to expand the VA range. When
> doing the initilization, users need to set both nr_hugepages and
> nr_overcommit_hugepages to the same value, like 64, 128, etc.
> 
> Signed-off-by: Chao Zhu <chaozhu@linux.vnet.ibm.com>
> ---
>  lib/librte_eal/linuxapp/eal/eal_memory.c | 8 ++++++++
>  1 file changed, 8 insertions(+)
> 
> diff --git a/lib/librte_eal/linuxapp/eal/eal_memory.c
> b/lib/librte_eal/linuxapp/eal/eal_memory.c index a956bb2..e06186b 100644
> --- a/lib/librte_eal/linuxapp/eal/eal_memory.c
> +++ b/lib/librte_eal/linuxapp/eal/eal_memory.c
> @@ -313,7 +313,11 @@ int rte_xen_dom0_supported(void)
>  	}
>  	do {
>  		addr = mmap(addr,
> +#ifndef RTE_ARCH_PPC_64
>  				(*size) + hugepage_sz, PROT_READ, MAP_PRIVATE, fd, 0);
> +#else
> +                (*size) + hugepage_sz, PROT_READ, MAP_PRIVATE |
> MAP_ANONYMOUS | MAP_HUGETLB, fd, 0); +#endif
>  		if (addr == MAP_FAILED)
>  			*size -= hugepage_sz;
>  	} while (addr == MAP_FAILED && *size > 0);
> @@ -1330,7 +1334,11 @@ static int huge_wrap_sigsetjmp(void)
>  		 * use mmap to get identical addresses as the primary process.
>  		 */
>  		base_addr = mmap(mcfg->memseg[s].addr, mcfg->memseg[s].len,
> +#ifndef RTE_ARCH_PPC_64
>  				 PROT_READ, MAP_PRIVATE, fd_zero, 0);
> +#else
> +                 PROT_READ, MAP_PRIVATE | MAP_ANONYMOUS | MAP_HUGETLB,
> fd_zero, 0); +#endif
>  		if (base_addr == MAP_FAILED ||
>  		    base_addr != mcfg->memseg[s].addr) {
>  			max_seg = s;

Indentation and line length are wrong.
Changed to this:

--- a/lib/librte_eal/linuxapp/eal/eal_memory.c
+++ b/lib/librte_eal/linuxapp/eal/eal_memory.c
@@ -331,7 +331,13 @@ get_virtual_area(size_t *size, size_t hugepage_sz)
        }
        do {
                addr = mmap(addr,
-                               (*size) + hugepage_sz, PROT_READ, MAP_PRIVATE, 
fd, 0);
+                               (*size) + hugepage_sz, PROT_READ,
+#ifdef RTE_ARCH_PPC_64
+                               MAP_PRIVATE | MAP_ANONYMOUS | MAP_HUGETLB,
+#else
+                               MAP_PRIVATE,
+#endif
+                               fd, 0);
                if (addr == MAP_FAILED)
                        *size -= hugepage_sz;
        } while (addr == MAP_FAILED && *size > 0);
@@ -1359,7 +1365,13 @@ rte_eal_hugepage_attach(void)
                 * use mmap to get identical addresses as the primary process.
                 */
                base_addr = mmap(mcfg->memseg[s].addr, mcfg->memseg[s].len,
-                                PROT_READ, MAP_PRIVATE, fd_zero, 0);
+                                PROT_READ,
+#ifdef RTE_ARCH_PPC_64
+                                MAP_PRIVATE | MAP_ANONYMOUS | MAP_HUGETLB,
+#else
+                                MAP_PRIVATE,
+#endif
+                                fd_zero, 0);
                if (base_addr == MAP_FAILED ||
                    base_addr != mcfg->memseg[s].addr) {
                        max_seg = s;

[PATCH 2/2] doc/guides: Add hugepage reserve instructions for IBM POWER

[Chao Zhu]

This patch adds additional instructions for hugepage reservation on IBM
POWER.

Signed-off-by: Chao Zhu <chaozhu@linux.vnet.ibm.com>
---
 doc/guides/linux_gsg/sys_reqs.rst | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/doc/guides/linux_gsg/sys_reqs.rst b/doc/guides/linux_gsg/sys_reqs.rst
index 61222c6..3a28c9e 100644
--- a/doc/guides/linux_gsg/sys_reqs.rst
+++ b/doc/guides/linux_gsg/sys_reqs.rst
@@ -200,6 +200,12 @@ On a NUMA machine, pages should be allocated explicitly on separate nodes::
 
     For 1G pages, it is not possible to reserve the hugepage memory after the system has booted.
 
+    On IBM POWER system, the nr_overcommit_hugepages should be set to the same value as nr_hugepages.
+    For example, if the required page number is 128, the following commands are used::
+
+        echo 128 > /sys/kernel/mm/hugepages/hugepages-16384kB/nr_hugepages
+        echo 128 > /sys/kernel/mm/hugepages/hugepages-16384kB/nr_overcommit_hugepages
+
 Using Hugepages with the DPDK
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
-- 
1.8.3.1

Re: [PATCH 2/2] doc/guides: Add hugepage reserve instructions for IBM POWER

[Mcnamara, John]

> -----Original Message-----
> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Chao Zhu
> Sent: Thursday, April 6, 2017 11:06 AM
> To: dev@dpdk.org
> Cc: Gowrishankar <gowrishankar.m@linux.vnet.ibm.com>; Gonzalez Monroy,
> Sergio <sergio.gonzalez.monroy@intel.com>; david.marchand@6wind.com
> Subject: [dpdk-dev] [PATCH 2/2] doc/guides: Add hugepage reserve
> instructions for IBM POWER
> 
> This patch adds additional instructions for hugepage reservation on IBM
> POWER.
> 
> Signed-off-by: Chao Zhu <chaozhu@linux.vnet.ibm.com>

Acked-by: John McNamara <john.mcnamara@intel.com>