long initialization of rte_eal_hugepage_init

long initialization of rte_eal_hugepage_init

[王志克]

Hi All,

I observed that rte_eal_hugepage_init() will take quite long time if there are lots of huge pages. Example I have 500 1G huge pages, and it takes about 2 minutes. That is too long especially for application restart case.

If the application only needs limited huge page while the host have lots of huge pages, the algorithm is not so efficent. Example, we only need 1G memory from each socket.

What is the proposal from DPDK community? Any solution?

Note I tried version dpdk 16.11.

Br,
Wang Zhike

Re: long initialization of rte_eal_hugepage_init

[Stephen Hemminger]

Linux zeros huge pages by default. There was a fix in later releases

On Sep 5, 2017 8:24 PM, "王志克" <wangzhike@jd.com> wrote:

> Hi All,
>
> I observed that rte_eal_hugepage_init() will take quite long time if there
> are lots of huge pages. Example I have 500 1G huge pages, and it takes
> about 2 minutes. That is too long especially for application restart case.
>
> If the application only needs limited huge page while the host have lots
> of huge pages, the algorithm is not so efficent. Example, we only need 1G
> memory from each socket.
>
> What is the proposal from DPDK community? Any solution?
>
> Note I tried version dpdk 16.11.
>
> Br,
> Wang Zhike
>

Re: long initialization of rte_eal_hugepage_init

[Pavan Nikhilesh Bhagavatula]

On Wed, Sep 06, 2017 at 03:24:52AM +0000, 王志克 wrote:
> Hi All,
>
> I observed that rte_eal_hugepage_init() will take quite long time if there are lots of huge pages. Example I have 500 1G huge pages, and it takes about 2 minutes. That is too long especially for application restart case.
>
> If the application only needs limited huge page while the host have lots of huge pages, the algorithm is not so efficent. Example, we only need 1G memory from each socket.
>

There is a EAL option --socket-mem which can be used to limit the memory
aquired from each socket.

> What is the proposal from DPDK community? Any solution?
>
> Note I tried version dpdk 16.11.
>
> Br,
> Wang Zhike

-Pavan

Re: long initialization of rte_eal_hugepage_init

[Tan, Jianfeng]

> -----Original Message-----
> From: users [mailto:users-bounces@dpdk.org] On Behalf Of ???
> Sent: Wednesday, September 6, 2017 11:25 AM
> To: users@dpdk.org; dev@dpdk.org
> Subject: [dpdk-users] long initialization of rte_eal_hugepage_init
> 
> Hi All,
> 
> I observed that rte_eal_hugepage_init() will take quite long time if there are
> lots of huge pages. Example I have 500 1G huge pages, and it takes about 2
> minutes. That is too long especially for application restart case.
> 
> If the application only needs limited huge page while the host have lots of
> huge pages, the algorithm is not so efficent. Example, we only need 1G
> memory from each socket.
> 
> What is the proposal from DPDK community? Any solution?

You can mount hugetlbfs with "size" option + use "--socket-mem" option in DPDK to restrict the memory to be used. 

Thanks,
Jianfeng

> 
> Note I tried version dpdk 16.11.
> 
> Br,
> Wang Zhike

Re: long initialization of rte_eal_hugepage_init

[王志克]

Do you mean "pagesize" when you say "size" option? I have specified the pagesize as 1G.
Also, I already use "--socket-mem " to specify that the application only needs 1G per NUMA node.

The problem is that map_all_hugepages() would map all free huge pages, and then select the proper ones. If I have 500 free huge pages (each 1G), and application only needs 1G per NUMA socket, it is unreasonable for such mapping.

My use case is OVS+DPDK. The OVS+DPDK would only need 2G, and other application (Qemu/VM) would use the other huge pages.

Br,
Wang Zhike


-----Original Message-----
From: Tan, Jianfeng [mailto:jianfeng.tan@intel.com] 
Sent: Wednesday, September 06, 2017 12:36 PM
To: 王志克; users@dpdk.org; dev@dpdk.org
Subject: RE: long initialization of rte_eal_hugepage_init



> -----Original Message-----
> From: users [mailto:users-bounces@dpdk.org] On Behalf Of ???
> Sent: Wednesday, September 6, 2017 11:25 AM
> To: users@dpdk.org; dev@dpdk.org
> Subject: [dpdk-users] long initialization of rte_eal_hugepage_init
> 
> Hi All,
> 
> I observed that rte_eal_hugepage_init() will take quite long time if there are
> lots of huge pages. Example I have 500 1G huge pages, and it takes about 2
> minutes. That is too long especially for application restart case.
> 
> If the application only needs limited huge page while the host have lots of
> huge pages, the algorithm is not so efficent. Example, we only need 1G
> memory from each socket.
> 
> What is the proposal from DPDK community? Any solution?

You can mount hugetlbfs with "size" option + use "--socket-mem" option in DPDK to restrict the memory to be used. 

Thanks,
Jianfeng

> 
> Note I tried version dpdk 16.11.
> 
> Br,
> Wang Zhike

Re: long initialization of rte_eal_hugepage_init

[王志克]

Hi Stephen,

Do you means “disable zero huge page” would improve the performance?  How can the memory be guarantee<http://www.baidu.com/link?url=OcSiFdTLN-XzcXbWcNS7WKEDAs5KPRf5SoQeihstSK0eIPPoRsFICa7XLymTk-ln_XJ5mXmGU9C4srI6Nwax6IgorIeptfF9NvgooO1z4B3>d to be allocated? Would it introduce function issue?

I checked below commit, and I guess the commit at least means the “zero the huge page” is needed.

commit 5ce3ace1de458e2ded1b408acfe59c15cf9863f1
Author: Zhihong Wang <zhihong.wang@intel.com>
Date:   Sun Nov 22 14:13:35 2015 -0500

    eal: remove unnecessary hugepage zero-filling

    The kernel fills new allocated (huge) pages with zeros.
    DPDK just has to populate page tables to trigger the allocation.

    Signed-off-by: Zhihong Wang <zhihong.wang@intel.com>
    Acked-by: Stephen Hemminger <stephen@networkplumber.org>

From: Stephen Hemminger [mailto:stephen@networkplumber.org]
Sent: Wednesday, September 06, 2017 12:24 PM
To: 王志克
Cc: dev@dpdk.org; users@dpdk.org
Subject: Re: [dpdk-dev] long initialization of rte_eal_hugepage_init

Linux zeros huge pages by default. There was a fix in later releases

On Sep 5, 2017 8:24 PM, "王志克" <wangzhike@jd.com<mailto:wangzhike@jd.com>> wrote:
Hi All,

I observed that rte_eal_hugepage_init() will take quite long time if there are lots of huge pages. Example I have 500 1G huge pages, and it takes about 2 minutes. That is too long especially for application restart case.

If the application only needs limited huge page while the host have lots of huge pages, the algorithm is not so efficent. Example, we only need 1G memory from each socket.

What is the proposal from DPDK community? Any solution?

Note I tried version dpdk 16.11.

Br,
Wang Zhike

Re: long initialization of rte_eal_hugepage_init

[Tan, Jianfeng]

> -----Original Message-----
> From: 王志克 [mailto:wangzhike@jd.com]
> Sent: Wednesday, September 6, 2017 2:03 PM
> To: Tan, Jianfeng; users@dpdk.org; dev@dpdk.org
> Subject: RE: long initialization of rte_eal_hugepage_init
> 
> Do you mean "pagesize" when you say "size" option? I have specified the
> pagesize as 1G.

No, I mean "size". I mean adding another hugetlbfs with total size = what you need for your app. And with another DPDK option "--huge-dir", we can avoid allocating all free hugepages.

If you want to allocate memory on different sockets, e.g., --socket-mem 1024,1024, you need a newer DPDK with below commit by Ilya Maximets:
commit 1b72605d241 ("mem: balanced allocation of hugepages").

Thanks,
Jianfeng

> Also, I already use "--socket-mem " to specify that the application only needs
> 1G per NUMA node.
> 
> The problem is that map_all_hugepages() would map all free huge pages,
> and then select the proper ones. If I have 500 free huge pages (each 1G), and
> application only needs 1G per NUMA socket, it is unreasonable for such
> mapping.
> 
> My use case is OVS+DPDK. The OVS+DPDK would only need 2G, and other
> application (Qemu/VM) would use the other huge pages.
> 
> Br,
> Wang Zhike
> 
> 
> -----Original Message-----
> From: Tan, Jianfeng [mailto:jianfeng.tan@intel.com]
> Sent: Wednesday, September 06, 2017 12:36 PM
> To: 王志克; users@dpdk.org; dev@dpdk.org
> Subject: RE: long initialization of rte_eal_hugepage_init
> 
> 
> 
> > -----Original Message-----
> > From: users [mailto:users-bounces@dpdk.org] On Behalf Of ???
> > Sent: Wednesday, September 6, 2017 11:25 AM
> > To: users@dpdk.org; dev@dpdk.org
> > Subject: [dpdk-users] long initialization of rte_eal_hugepage_init
> >
> > Hi All,
> >
> > I observed that rte_eal_hugepage_init() will take quite long time if there
> are
> > lots of huge pages. Example I have 500 1G huge pages, and it takes about 2
> > minutes. That is too long especially for application restart case.
> >
> > If the application only needs limited huge page while the host have lots of
> > huge pages, the algorithm is not so efficent. Example, we only need 1G
> > memory from each socket.
> >
> > What is the proposal from DPDK community? Any solution?
> 
> You can mount hugetlbfs with "size" option + use "--socket-mem" option in
> DPDK to restrict the memory to be used.
> 
> Thanks,
> Jianfeng
> 
> >
> > Note I tried version dpdk 16.11.
> >
> > Br,
> > Wang Zhike

Re: long initialization of rte_eal_hugepage_init

[Sergio Gonzalez Monroy]

On 06/09/2017 05:35, Pavan Nikhilesh Bhagavatula wrote:
> On Wed, Sep 06, 2017 at 03:24:52AM +0000, 王志克 wrote:
>> Hi All,
>>
>> I observed that rte_eal_hugepage_init() will take quite long time if there are lots of huge pages. Example I have 500 1G huge pages, and it takes about 2 minutes. That is too long especially for application restart case.
>>
>> If the application only needs limited huge page while the host have lots of huge pages, the algorithm is not so efficent. Example, we only need 1G memory from each socket.
>>
> There is a EAL option --socket-mem which can be used to limit the memory
> aquired from each socket.
>
>> What is the proposal from DPDK community? Any solution?
>>
>> Note I tried version dpdk 16.11.
>>
>> Br,
>> Wang Zhike
> -Pavan

Since DPDK 17.08+ we use libnuma to first get the amount of pages we 
need from each socket, then as many more as we can.
So you can setup your huge page mount point or cgroups to limit the 
amount of pages you can get.

So basically:
1. setup mount quota or cgroup limit
2. use --socket-mem option to limit amount per socket

Note that pre-17.08 we did not have libnuma support so chances are that 
if you have a low quota/limit and need memory from both sockets it would 
likely fail to allocate the request.

Thanks,
Sergio

Re: long initialization of rte_eal_hugepage_init

[王志克]

Thanks Jianfeng for your suggestion. I get the point.

Br,
Wang Zhike

-----Original Message-----
From: Tan, Jianfeng [mailto:jianfeng.tan@intel.com] 
Sent: Wednesday, September 06, 2017 3:18 PM
To: 王志克; users@dpdk.org; dev@dpdk.org
Subject: RE: long initialization of rte_eal_hugepage_init



> -----Original Message-----
> From: 王志克 [mailto:wangzhike@jd.com]
> Sent: Wednesday, September 6, 2017 2:03 PM
> To: Tan, Jianfeng; users@dpdk.org; dev@dpdk.org
> Subject: RE: long initialization of rte_eal_hugepage_init
> 
> Do you mean "pagesize" when you say "size" option? I have specified the
> pagesize as 1G.

No, I mean "size". I mean adding another hugetlbfs with total size = what you need for your app. And with another DPDK option "--huge-dir", we can avoid allocating all free hugepages.

If you want to allocate memory on different sockets, e.g., --socket-mem 1024,1024, you need a newer DPDK with below commit by Ilya Maximets:
commit 1b72605d241 ("mem: balanced allocation of hugepages").

Thanks,
Jianfeng

> Also, I already use "--socket-mem " to specify that the application only needs
> 1G per NUMA node.
> 
> The problem is that map_all_hugepages() would map all free huge pages,
> and then select the proper ones. If I have 500 free huge pages (each 1G), and
> application only needs 1G per NUMA socket, it is unreasonable for such
> mapping.
> 
> My use case is OVS+DPDK. The OVS+DPDK would only need 2G, and other
> application (Qemu/VM) would use the other huge pages.
> 
> Br,
> Wang Zhike
> 
> 
> -----Original Message-----
> From: Tan, Jianfeng [mailto:jianfeng.tan@intel.com]
> Sent: Wednesday, September 06, 2017 12:36 PM
> To: 王志克; users@dpdk.org; dev@dpdk.org
> Subject: RE: long initialization of rte_eal_hugepage_init
> 
> 
> 
> > -----Original Message-----
> > From: users [mailto:users-bounces@dpdk.org] On Behalf Of ???
> > Sent: Wednesday, September 6, 2017 11:25 AM
> > To: users@dpdk.org; dev@dpdk.org
> > Subject: [dpdk-users] long initialization of rte_eal_hugepage_init
> >
> > Hi All,
> >
> > I observed that rte_eal_hugepage_init() will take quite long time if there
> are
> > lots of huge pages. Example I have 500 1G huge pages, and it takes about 2
> > minutes. That is too long especially for application restart case.
> >
> > If the application only needs limited huge page while the host have lots of
> > huge pages, the algorithm is not so efficent. Example, we only need 1G
> > memory from each socket.
> >
> > What is the proposal from DPDK community? Any solution?
> 
> You can mount hugetlbfs with "size" option + use "--socket-mem" option in
> DPDK to restrict the memory to be used.
> 
> Thanks,
> Jianfeng
> 
> >
> > Note I tried version dpdk 16.11.
> >
> > Br,
> > Wang Zhike

Re: long initialization of rte_eal_hugepage_init

[王志克]

Thanks Sergio. It really helps.

Br,
Wang Zhike

-----Original Message-----
From: Sergio Gonzalez Monroy [mailto:sergio.gonzalez.monroy@intel.com] 
Sent: Wednesday, September 06, 2017 3:37 PM
To: Pavan Nikhilesh Bhagavatula; 王志克
Cc: dev@dpdk.org
Subject: Re: [dpdk-dev] long initialization of rte_eal_hugepage_init

On 06/09/2017 05:35, Pavan Nikhilesh Bhagavatula wrote:
> On Wed, Sep 06, 2017 at 03:24:52AM +0000, 王志克 wrote:
>> Hi All,
>>
>> I observed that rte_eal_hugepage_init() will take quite long time if there are lots of huge pages. Example I have 500 1G huge pages, and it takes about 2 minutes. That is too long especially for application restart case.
>>
>> If the application only needs limited huge page while the host have lots of huge pages, the algorithm is not so efficent. Example, we only need 1G memory from each socket.
>>
> There is a EAL option --socket-mem which can be used to limit the memory
> aquired from each socket.
>
>> What is the proposal from DPDK community? Any solution?
>>
>> Note I tried version dpdk 16.11.
>>
>> Br,
>> Wang Zhike
> -Pavan

Since DPDK 17.08+ we use libnuma to first get the amount of pages we 
need from each socket, then as many more as we can.
So you can setup your huge page mount point or cgroups to limit the 
amount of pages you can get.

So basically:
1. setup mount quota or cgroup limit
2. use --socket-mem option to limit amount per socket

Note that pre-17.08 we did not have libnuma support so chances are that 
if you have a low quota/limit and need memory from both sockets it would 
likely fail to allocate the request.

Thanks,
Sergio