[PATCH 0/2] Reduce DPDK initialization time

[PATCH 0/2] Reduce DPDK initialization time

[Zhihong Wang]

This patch aims to reduce DPDK initialization time, which is important in cases such as micro service.

Changes are:

1. Reduce timer initialization time

2. Remove unnecessary hugepage zero-filling operations

With this patch:

1. Timer initialization time can be reduced by 4/10 second

2. Memory initialization time can be reduced nearly by half

The 2nd topic has been brought up before in this thread:
http://dpdk.org/dev/patchwork/patch/4219/

--------------
Changes in v1:

1. Use macro in sleep time initialization

2. Update commit message according to code change

--------------
Changes in RFC v2:

1. Use MAP_POPULATE flag to populate page tables

2. Add comments to avoid future misunderstanding

Zhihong Wang (2):
  lib/librte_eal: Reduce timer initialization time
  lib/librte_eal: Remove unnecessary hugepage zero-filling

 lib/librte_eal/linuxapp/eal/eal_memory.c | 20 ++++++--------------
 lib/librte_eal/linuxapp/eal/eal_timer.c  |  2 +-
 2 files changed, 7 insertions(+), 15 deletions(-)

-- 
2.5.0

[PATCH 1/2] lib/librte_eal: Reduce timer initialization time

[Zhihong Wang]

Changing from 1/2 second to 1/10 doesn't compromise the precision, and a 4/10 second is worth saving.

Signed-off-by: Zhihong Wang <zhihong.wang@intel.com>
---
 lib/librte_eal/linuxapp/eal/eal_timer.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/lib/librte_eal/linuxapp/eal/eal_timer.c b/lib/librte_eal/linuxapp/eal/eal_timer.c
index e0642de..b40afa0 100644
--- a/lib/librte_eal/linuxapp/eal/eal_timer.c
+++ b/lib/librte_eal/linuxapp/eal/eal_timer.c
@@ -271,7 +271,7 @@ get_tsc_freq(void)
 #ifdef CLOCK_MONOTONIC_RAW
 #define NS_PER_SEC 1E9
 
-	struct timespec sleeptime = {.tv_nsec = 5E8 }; /* 1/2 second */
+	struct timespec sleeptime = {.tv_nsec = NS_PER_SEC / 10 }; /* 1/10 second */
 
 	struct timespec t_start, t_end;
 	uint64_t tsc_hz;
-- 
2.5.0

[PATCH 2/2] lib/librte_eal: Remove unnecessary hugepage zero-filling

[Zhihong Wang]

The kernel fills new allocated (huge) pages with zeros.
DPDK just has to populate page tables to trigger the allocation.

Signed-off-by: Zhihong Wang <zhihong.wang@intel.com>
---
 lib/librte_eal/linuxapp/eal/eal_memory.c | 20 ++++++--------------
 1 file changed, 6 insertions(+), 14 deletions(-)

diff --git a/lib/librte_eal/linuxapp/eal/eal_memory.c b/lib/librte_eal/linuxapp/eal/eal_memory.c
index 0de75cd..21a5146 100644
--- a/lib/librte_eal/linuxapp/eal/eal_memory.c
+++ b/lib/librte_eal/linuxapp/eal/eal_memory.c
@@ -399,8 +399,10 @@ map_all_hugepages(struct hugepage_file *hugepg_tbl,
 			return -1;
 		}
 
+		/* map the segment, and populate page tables,
+		 * the kernel fills this segment with zeros */
 		virtaddr = mmap(vma_addr, hugepage_sz, PROT_READ | PROT_WRITE,
-				MAP_SHARED, fd, 0);
+				MAP_SHARED | MAP_POPULATE, fd, 0);
 		if (virtaddr == MAP_FAILED) {
 			RTE_LOG(ERR, EAL, "%s(): mmap failed: %s\n", __func__,
 					strerror(errno));
@@ -410,7 +412,6 @@ map_all_hugepages(struct hugepage_file *hugepg_tbl,
 
 		if (orig) {
 			hugepg_tbl[i].orig_va = virtaddr;
-			memset(virtaddr, 0, hugepage_sz);
 		}
 		else {
 			hugepg_tbl[i].final_va = virtaddr;
@@ -529,22 +530,16 @@ remap_all_hugepages(struct hugepage_file *hugepg_tbl, struct hugepage_info *hpi)
 
 			old_addr = vma_addr;
 
-			/* map new, bigger segment */
+			/* map new, bigger segment, and populate page tables,
+			 * the kernel fills this segment with zeros */
 			vma_addr = mmap(vma_addr, total_size,
-					PROT_READ | PROT_WRITE, MAP_SHARED, fd, 0);
+					PROT_READ | PROT_WRITE, MAP_SHARED | MAP_POPULATE, fd, 0);
 
 			if (vma_addr == MAP_FAILED || vma_addr != old_addr) {
 				RTE_LOG(ERR, EAL, "%s(): mmap failed: %s\n", __func__, strerror(errno));
 				close(fd);
 				return -1;
 			}
-
-			/* touch the page. this is needed because kernel postpones mapping
-			 * creation until the first page fault. with this, we pin down
-			 * the page and it is marked as used and gets into process' pagemap.
-			 */
-			for (offset = 0; offset < total_size; offset += hugepage_sz)
-				*((volatile uint8_t*) RTE_PTR_ADD(vma_addr, offset));
 		}
 
 		/* set shared flock on the file. */
@@ -592,9 +587,6 @@ remap_all_hugepages(struct hugepage_file *hugepg_tbl, struct hugepage_info *hpi)
 			}
 		}
 
-		/* zero out the whole segment */
-		memset(hugepg_tbl[page_idx].final_va, 0, total_size);
-
 		page_idx++;
 	}
 
-- 
2.5.0

Re: [PATCH 2/2] lib/librte_eal: Remove unnecessary hugepage zero-filling

[Stephen Hemminger]

On Sun, 22 Nov 2015 14:13:35 -0500
Zhihong Wang <zhihong.wang@intel.com> wrote:

> The kernel fills new allocated (huge) pages with zeros.
> DPDK just has to populate page tables to trigger the allocation.
> 
> Signed-off-by: Zhihong Wang <zhihong.wang@intel.com>
> ---
>  lib/librte_eal/linuxapp/eal/eal_memory.c | 20 ++++++--------------
>  1 file changed, 6 insertions(+), 14 deletions(-)
> 
> diff --git a/lib/librte_eal/linuxapp/eal/eal_memory.c b/lib/librte_eal/linuxapp/eal/eal_memory.c
> index 0de75cd..21a5146 100644
> --- a/lib/librte_eal/linuxapp/eal/eal_memory.c
> +++ b/lib/librte_eal/linuxapp/eal/eal_memory.c
> @@ -399,8 +399,10 @@ map_all_hugepages(struct hugepage_file *hugepg_tbl,
>  			return -1;
>  		}
>  
> +		/* map the segment, and populate page tables,
> +		 * the kernel fills this segment with zeros */
>  		virtaddr = mmap(vma_addr, hugepage_sz, PROT_READ | PROT_WRITE,
> -				MAP_SHARED, fd, 0);
> +				MAP_SHARED | MAP_POPULATE, fd, 0);
>  		if (virtaddr == MAP_FAILED) {
>  			RTE_LOG(ERR, EAL, "%s(): mmap failed: %s\n", __func__,
>  					strerror(errno));
> @@ -410,7 +412,6 @@ map_all_hugepages(struct hugepage_file *hugepg_tbl,
>  
>  		if (orig) {
>  			hugepg_tbl[i].orig_va = virtaddr;
> -			memset(virtaddr, 0, hugepage_sz);
>  		}
>  		else {
>  			hugepg_tbl[i].final_va = virtaddr;
> @@ -529,22 +530,16 @@ remap_all_hugepages(struct hugepage_file *hugepg_tbl, struct hugepage_info *hpi)
>  
>  			old_addr = vma_addr;
>  
> -			/* map new, bigger segment */
> +			/* map new, bigger segment, and populate page tables,
> +			 * the kernel fills this segment with zeros */
>  			vma_addr = mmap(vma_addr, total_size,
> -					PROT_READ | PROT_WRITE, MAP_SHARED, fd, 0);
> +					PROT_READ | PROT_WRITE, MAP_SHARED | MAP_POPULATE, fd, 0);
>  
>  			if (vma_addr == MAP_FAILED || vma_addr != old_addr) {
>  				RTE_LOG(ERR, EAL, "%s(): mmap failed: %s\n", __func__, strerror(errno));
>  				close(fd);
>  				return -1;
>  			}
> -
> -			/* touch the page. this is needed because kernel postpones mapping
> -			 * creation until the first page fault. with this, we pin down
> -			 * the page and it is marked as used and gets into process' pagemap.
> -			 */
> -			for (offset = 0; offset < total_size; offset += hugepage_sz)
> -				*((volatile uint8_t*) RTE_PTR_ADD(vma_addr, offset));
>  		}
>  
>  		/* set shared flock on the file. */
> @@ -592,9 +587,6 @@ remap_all_hugepages(struct hugepage_file *hugepg_tbl, struct hugepage_info *hpi)
>  			}
>  		}
>  
> -		/* zero out the whole segment */
> -		memset(hugepg_tbl[page_idx].final_va, 0, total_size);
> -
>  		page_idx++;
>  	}
>  

Nice, especially on slow machines or with large memory.

Acked-by: Stephen Hemminger <stephen@networkplumber.org>

Re: [PATCH 2/2] lib/librte_eal: Remove unnecessary hugepage zero-filling

[Thomas Monjalon]

2015-11-22 18:28, Stephen Hemminger:
> On Sun, 22 Nov 2015 14:13:35 -0500
> Zhihong Wang <zhihong.wang@intel.com> wrote:
> 
> > The kernel fills new allocated (huge) pages with zeros.
> > DPDK just has to populate page tables to trigger the allocation.
> > 
> > Signed-off-by: Zhihong Wang <zhihong.wang@intel.com>
> 
> Nice, especially on slow machines or with large memory.
> 
> Acked-by: Stephen Hemminger <stephen@networkplumber.org>

Yes very nice.
I think it's too late to integrate this change which can have some
unpredictable side effects.
Do you agree to wait for 2.3?

Re: [PATCH 2/2] lib/librte_eal: Remove unnecessary hugepage zero-filling

[Stephen Hemminger]

On Tue, 24 Nov 2015 22:13:28 +0100
Thomas Monjalon <thomas.monjalon@6wind.com> wrote:

> 2015-11-22 18:28, Stephen Hemminger:
> > On Sun, 22 Nov 2015 14:13:35 -0500
> > Zhihong Wang <zhihong.wang@intel.com> wrote:
> > 
> > > The kernel fills new allocated (huge) pages with zeros.
> > > DPDK just has to populate page tables to trigger the allocation.
> > > 
> > > Signed-off-by: Zhihong Wang <zhihong.wang@intel.com>
> > 
> > Nice, especially on slow machines or with large memory.
> > 
> > Acked-by: Stephen Hemminger <stephen@networkplumber.org>
> 
> Yes very nice.
> I think it's too late to integrate this change which can have some
> unpredictable side effects.
> Do you agree to wait for 2.3?

What side effects? Either it is zero or it is not.
Only some broken architecture would have an issue.

Re: [PATCH 2/2] lib/librte_eal: Remove unnecessary hugepage zero-filling

[Thomas Monjalon]

2015-11-24 14:44, Stephen Hemminger:
> On Tue, 24 Nov 2015 22:13:28 +0100
> Thomas Monjalon <thomas.monjalon@6wind.com> wrote:
> 
> > 2015-11-22 18:28, Stephen Hemminger:
> > > On Sun, 22 Nov 2015 14:13:35 -0500
> > > Zhihong Wang <zhihong.wang@intel.com> wrote:
> > > 
> > > > The kernel fills new allocated (huge) pages with zeros.
> > > > DPDK just has to populate page tables to trigger the allocation.
> > > > 
> > > > Signed-off-by: Zhihong Wang <zhihong.wang@intel.com>
> > > 
> > > Nice, especially on slow machines or with large memory.
> > > 
> > > Acked-by: Stephen Hemminger <stephen@networkplumber.org>
> > 
> > Yes very nice.
> > I think it's too late to integrate this change which can have some
> > unpredictable side effects.
> > Do you agree to wait for 2.3?
> 
> What side effects? Either it is zero or it is not.
> Only some broken architecture would have an issue.

I mean it changes the memory allocator behaviour. It's not something we
want to discover a new bug just before the release.
This kind of important change must be integrated at the beginning of the
release cycle.
I'm asking for opinions because it would be really nice to have.

Re: [PATCH 2/2] lib/librte_eal: Remove unnecessary hugepage zero-filling

[Yuanhan Liu]

On Wed, Nov 25, 2015 at 12:04:16AM +0100, Thomas Monjalon wrote:
> 2015-11-24 14:44, Stephen Hemminger:
> > On Tue, 24 Nov 2015 22:13:28 +0100
> > Thomas Monjalon <thomas.monjalon@6wind.com> wrote:
> > 
> > > 2015-11-22 18:28, Stephen Hemminger:
> > > > On Sun, 22 Nov 2015 14:13:35 -0500
> > > > Zhihong Wang <zhihong.wang@intel.com> wrote:
> > > > 
> > > > > The kernel fills new allocated (huge) pages with zeros.
> > > > > DPDK just has to populate page tables to trigger the allocation.
> > > > > 
> > > > > Signed-off-by: Zhihong Wang <zhihong.wang@intel.com>
> > > > 
> > > > Nice, especially on slow machines or with large memory.
> > > > 
> > > > Acked-by: Stephen Hemminger <stephen@networkplumber.org>
> > > 
> > > Yes very nice.
> > > I think it's too late to integrate this change which can have some
> > > unpredictable side effects.
> > > Do you agree to wait for 2.3?
> > 
> > What side effects? Either it is zero or it is not.
> > Only some broken architecture would have an issue.
> 
> I mean it changes the memory allocator behaviour. It's not something we
> want to discover a new bug just before the release.
> This kind of important change must be integrated at the beginning of the
> release cycle.

+ 1

And it could be a new feature (or highlight) of 2.3: reduced dpdk
startup time by ... :)

	--yliu

Re: [PATCH 2/2] lib/librte_eal: Remove unnecessary hugepage zero-filling

[Wang, Zhihong]

> -----Original Message-----
> From: Thomas Monjalon [mailto:thomas.monjalon@6wind.com]
> Sent: Wednesday, November 25, 2015 7:04 AM
> To: Stephen Hemminger <stephen@networkplumber.org>
> Cc: Wang, Zhihong <zhihong.wang@intel.com>; dev@dpdk.org
> Subject: Re: [dpdk-dev] [PATCH 2/2] lib/librte_eal: Remove unnecessary
> hugepage zero-filling
> 
> 2015-11-24 14:44, Stephen Hemminger:
> > On Tue, 24 Nov 2015 22:13:28 +0100
> > Thomas Monjalon <thomas.monjalon@6wind.com> wrote:
> >
> > > 2015-11-22 18:28, Stephen Hemminger:
> > > > On Sun, 22 Nov 2015 14:13:35 -0500 Zhihong Wang
> > > > <zhihong.wang@intel.com> wrote:
> > > >
> > > > > The kernel fills new allocated (huge) pages with zeros.
> > > > > DPDK just has to populate page tables to trigger the allocation.
> > > > >
> > > > > Signed-off-by: Zhihong Wang <zhihong.wang@intel.com>
> > > >
> > > > Nice, especially on slow machines or with large memory.
> > > >
> > > > Acked-by: Stephen Hemminger <stephen@networkplumber.org>
> > >
> > > Yes very nice.
> > > I think it's too late to integrate this change which can have some
> > > unpredictable side effects.
> > > Do you agree to wait for 2.3?
> >
> > What side effects? Either it is zero or it is not.
> > Only some broken architecture would have an issue.
> 
> I mean it changes the memory allocator behaviour. It's not something we want to
> discover a new bug just before the release.
> This kind of important change must be integrated at the beginning of the release
> cycle.
> I'm asking for opinions because it would be really nice to have.

Literally this patch doesn't change anything, it just keeps DPDK from zero-filling pages again which have just been zero-filled.
It would be nice to have this patch in DPDK 2.2 since it can reduce the startup time nearly by half for hugepage cases.
But I understand longer merge/test window make it safer for a release.
It makes sense either way.

Re: [PATCH 0/2] Reduce DPDK initialization time

[Thomas Monjalon]

2015-11-22 14:13, Zhihong Wang:
> This patch aims to reduce DPDK initialization time, which is important in cases such as micro service.
> 
> Changes are:
> 
> 1. Reduce timer initialization time
> 
> 2. Remove unnecessary hugepage zero-filling operations
> 
> With this patch:
> 
> 1. Timer initialization time can be reduced by 4/10 second
> 
> 2. Memory initialization time can be reduced nearly by half
> 
> The 2nd topic has been brought up before in this thread:
> http://dpdk.org/dev/patchwork/patch/4219/

Applied, thanks