* [PATCH 0/2] Reduce DPDK initialization time
@ 2015-11-22 19:13 Zhihong Wang
2015-11-22 19:13 ` [PATCH 1/2] lib/librte_eal: Reduce timer " Zhihong Wang
` (2 more replies)
0 siblings, 3 replies; 10+ messages in thread
From: Zhihong Wang @ 2015-11-22 19:13 UTC (permalink / raw)
To: dev
This patch aims to reduce DPDK initialization time, which is important in cases such as micro service.
Changes are:
1. Reduce timer initialization time
2. Remove unnecessary hugepage zero-filling operations
With this patch:
1. Timer initialization time can be reduced by 4/10 second
2. Memory initialization time can be reduced nearly by half
The 2nd topic has been brought up before in this thread:
http://dpdk.org/dev/patchwork/patch/4219/
--------------
Changes in v1:
1. Use macro in sleep time initialization
2. Update commit message according to code change
--------------
Changes in RFC v2:
1. Use MAP_POPULATE flag to populate page tables
2. Add comments to avoid future misunderstanding
Zhihong Wang (2):
lib/librte_eal: Reduce timer initialization time
lib/librte_eal: Remove unnecessary hugepage zero-filling
lib/librte_eal/linuxapp/eal/eal_memory.c | 20 ++++++--------------
lib/librte_eal/linuxapp/eal/eal_timer.c | 2 +-
2 files changed, 7 insertions(+), 15 deletions(-)
--
2.5.0
^ permalink raw reply [flat|nested] 10+ messages in thread* [PATCH 1/2] lib/librte_eal: Reduce timer initialization time 2015-11-22 19:13 [PATCH 0/2] Reduce DPDK initialization time Zhihong Wang @ 2015-11-22 19:13 ` Zhihong Wang 2015-11-22 19:13 ` [PATCH 2/2] lib/librte_eal: Remove unnecessary hugepage zero-filling Zhihong Wang 2016-01-21 14:59 ` [PATCH 0/2] Reduce DPDK initialization time Thomas Monjalon 2 siblings, 0 replies; 10+ messages in thread From: Zhihong Wang @ 2015-11-22 19:13 UTC (permalink / raw) To: dev Changing from 1/2 second to 1/10 doesn't compromise the precision, and a 4/10 second is worth saving. Signed-off-by: Zhihong Wang <zhihong.wang@intel.com> --- lib/librte_eal/linuxapp/eal/eal_timer.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/librte_eal/linuxapp/eal/eal_timer.c b/lib/librte_eal/linuxapp/eal/eal_timer.c index e0642de..b40afa0 100644 --- a/lib/librte_eal/linuxapp/eal/eal_timer.c +++ b/lib/librte_eal/linuxapp/eal/eal_timer.c @@ -271,7 +271,7 @@ get_tsc_freq(void) #ifdef CLOCK_MONOTONIC_RAW #define NS_PER_SEC 1E9 - struct timespec sleeptime = {.tv_nsec = 5E8 }; /* 1/2 second */ + struct timespec sleeptime = {.tv_nsec = NS_PER_SEC / 10 }; /* 1/10 second */ struct timespec t_start, t_end; uint64_t tsc_hz; -- 2.5.0 ^ permalink raw reply related [flat|nested] 10+ messages in thread
* [PATCH 2/2] lib/librte_eal: Remove unnecessary hugepage zero-filling 2015-11-22 19:13 [PATCH 0/2] Reduce DPDK initialization time Zhihong Wang 2015-11-22 19:13 ` [PATCH 1/2] lib/librte_eal: Reduce timer " Zhihong Wang @ 2015-11-22 19:13 ` Zhihong Wang 2015-11-23 2:28 ` Stephen Hemminger 2016-01-21 14:59 ` [PATCH 0/2] Reduce DPDK initialization time Thomas Monjalon 2 siblings, 1 reply; 10+ messages in thread From: Zhihong Wang @ 2015-11-22 19:13 UTC (permalink / raw) To: dev The kernel fills new allocated (huge) pages with zeros. DPDK just has to populate page tables to trigger the allocation. Signed-off-by: Zhihong Wang <zhihong.wang@intel.com> --- lib/librte_eal/linuxapp/eal/eal_memory.c | 20 ++++++-------------- 1 file changed, 6 insertions(+), 14 deletions(-) diff --git a/lib/librte_eal/linuxapp/eal/eal_memory.c b/lib/librte_eal/linuxapp/eal/eal_memory.c index 0de75cd..21a5146 100644 --- a/lib/librte_eal/linuxapp/eal/eal_memory.c +++ b/lib/librte_eal/linuxapp/eal/eal_memory.c @@ -399,8 +399,10 @@ map_all_hugepages(struct hugepage_file *hugepg_tbl, return -1; } + /* map the segment, and populate page tables, + * the kernel fills this segment with zeros */ virtaddr = mmap(vma_addr, hugepage_sz, PROT_READ | PROT_WRITE, - MAP_SHARED, fd, 0); + MAP_SHARED | MAP_POPULATE, fd, 0); if (virtaddr == MAP_FAILED) { RTE_LOG(ERR, EAL, "%s(): mmap failed: %s\n", __func__, strerror(errno)); @@ -410,7 +412,6 @@ map_all_hugepages(struct hugepage_file *hugepg_tbl, if (orig) { hugepg_tbl[i].orig_va = virtaddr; - memset(virtaddr, 0, hugepage_sz); } else { hugepg_tbl[i].final_va = virtaddr; @@ -529,22 +530,16 @@ remap_all_hugepages(struct hugepage_file *hugepg_tbl, struct hugepage_info *hpi) old_addr = vma_addr; - /* map new, bigger segment */ + /* map new, bigger segment, and populate page tables, + * the kernel fills this segment with zeros */ vma_addr = mmap(vma_addr, total_size, - PROT_READ | PROT_WRITE, MAP_SHARED, fd, 0); + PROT_READ | PROT_WRITE, MAP_SHARED | MAP_POPULATE, fd, 0); if (vma_addr == MAP_FAILED || vma_addr != old_addr) { RTE_LOG(ERR, EAL, "%s(): mmap failed: %s\n", __func__, strerror(errno)); close(fd); return -1; } - - /* touch the page. this is needed because kernel postpones mapping - * creation until the first page fault. with this, we pin down - * the page and it is marked as used and gets into process' pagemap. - */ - for (offset = 0; offset < total_size; offset += hugepage_sz) - *((volatile uint8_t*) RTE_PTR_ADD(vma_addr, offset)); } /* set shared flock on the file. */ @@ -592,9 +587,6 @@ remap_all_hugepages(struct hugepage_file *hugepg_tbl, struct hugepage_info *hpi) } } - /* zero out the whole segment */ - memset(hugepg_tbl[page_idx].final_va, 0, total_size); - page_idx++; } -- 2.5.0 ^ permalink raw reply related [flat|nested] 10+ messages in thread
* Re: [PATCH 2/2] lib/librte_eal: Remove unnecessary hugepage zero-filling 2015-11-22 19:13 ` [PATCH 2/2] lib/librte_eal: Remove unnecessary hugepage zero-filling Zhihong Wang @ 2015-11-23 2:28 ` Stephen Hemminger 2015-11-24 21:13 ` Thomas Monjalon 0 siblings, 1 reply; 10+ messages in thread From: Stephen Hemminger @ 2015-11-23 2:28 UTC (permalink / raw) To: Zhihong Wang; +Cc: dev On Sun, 22 Nov 2015 14:13:35 -0500 Zhihong Wang <zhihong.wang@intel.com> wrote: > The kernel fills new allocated (huge) pages with zeros. > DPDK just has to populate page tables to trigger the allocation. > > Signed-off-by: Zhihong Wang <zhihong.wang@intel.com> > --- > lib/librte_eal/linuxapp/eal/eal_memory.c | 20 ++++++-------------- > 1 file changed, 6 insertions(+), 14 deletions(-) > > diff --git a/lib/librte_eal/linuxapp/eal/eal_memory.c b/lib/librte_eal/linuxapp/eal/eal_memory.c > index 0de75cd..21a5146 100644 > --- a/lib/librte_eal/linuxapp/eal/eal_memory.c > +++ b/lib/librte_eal/linuxapp/eal/eal_memory.c > @@ -399,8 +399,10 @@ map_all_hugepages(struct hugepage_file *hugepg_tbl, > return -1; > } > > + /* map the segment, and populate page tables, > + * the kernel fills this segment with zeros */ > virtaddr = mmap(vma_addr, hugepage_sz, PROT_READ | PROT_WRITE, > - MAP_SHARED, fd, 0); > + MAP_SHARED | MAP_POPULATE, fd, 0); > if (virtaddr == MAP_FAILED) { > RTE_LOG(ERR, EAL, "%s(): mmap failed: %s\n", __func__, > strerror(errno)); > @@ -410,7 +412,6 @@ map_all_hugepages(struct hugepage_file *hugepg_tbl, > > if (orig) { > hugepg_tbl[i].orig_va = virtaddr; > - memset(virtaddr, 0, hugepage_sz); > } > else { > hugepg_tbl[i].final_va = virtaddr; > @@ -529,22 +530,16 @@ remap_all_hugepages(struct hugepage_file *hugepg_tbl, struct hugepage_info *hpi) > > old_addr = vma_addr; > > - /* map new, bigger segment */ > + /* map new, bigger segment, and populate page tables, > + * the kernel fills this segment with zeros */ > vma_addr = mmap(vma_addr, total_size, > - PROT_READ | PROT_WRITE, MAP_SHARED, fd, 0); > + PROT_READ | PROT_WRITE, MAP_SHARED | MAP_POPULATE, fd, 0); > > if (vma_addr == MAP_FAILED || vma_addr != old_addr) { > RTE_LOG(ERR, EAL, "%s(): mmap failed: %s\n", __func__, strerror(errno)); > close(fd); > return -1; > } > - > - /* touch the page. this is needed because kernel postpones mapping > - * creation until the first page fault. with this, we pin down > - * the page and it is marked as used and gets into process' pagemap. > - */ > - for (offset = 0; offset < total_size; offset += hugepage_sz) > - *((volatile uint8_t*) RTE_PTR_ADD(vma_addr, offset)); > } > > /* set shared flock on the file. */ > @@ -592,9 +587,6 @@ remap_all_hugepages(struct hugepage_file *hugepg_tbl, struct hugepage_info *hpi) > } > } > > - /* zero out the whole segment */ > - memset(hugepg_tbl[page_idx].final_va, 0, total_size); > - > page_idx++; > } > Nice, especially on slow machines or with large memory. Acked-by: Stephen Hemminger <stephen@networkplumber.org> ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [PATCH 2/2] lib/librte_eal: Remove unnecessary hugepage zero-filling 2015-11-23 2:28 ` Stephen Hemminger @ 2015-11-24 21:13 ` Thomas Monjalon 2015-11-24 22:44 ` Stephen Hemminger 0 siblings, 1 reply; 10+ messages in thread From: Thomas Monjalon @ 2015-11-24 21:13 UTC (permalink / raw) To: Stephen Hemminger, Zhihong Wang; +Cc: dev 2015-11-22 18:28, Stephen Hemminger: > On Sun, 22 Nov 2015 14:13:35 -0500 > Zhihong Wang <zhihong.wang@intel.com> wrote: > > > The kernel fills new allocated (huge) pages with zeros. > > DPDK just has to populate page tables to trigger the allocation. > > > > Signed-off-by: Zhihong Wang <zhihong.wang@intel.com> > > Nice, especially on slow machines or with large memory. > > Acked-by: Stephen Hemminger <stephen@networkplumber.org> Yes very nice. I think it's too late to integrate this change which can have some unpredictable side effects. Do you agree to wait for 2.3? ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [PATCH 2/2] lib/librte_eal: Remove unnecessary hugepage zero-filling 2015-11-24 21:13 ` Thomas Monjalon @ 2015-11-24 22:44 ` Stephen Hemminger 2015-11-24 23:04 ` Thomas Monjalon 0 siblings, 1 reply; 10+ messages in thread From: Stephen Hemminger @ 2015-11-24 22:44 UTC (permalink / raw) To: Thomas Monjalon; +Cc: dev On Tue, 24 Nov 2015 22:13:28 +0100 Thomas Monjalon <thomas.monjalon@6wind.com> wrote: > 2015-11-22 18:28, Stephen Hemminger: > > On Sun, 22 Nov 2015 14:13:35 -0500 > > Zhihong Wang <zhihong.wang@intel.com> wrote: > > > > > The kernel fills new allocated (huge) pages with zeros. > > > DPDK just has to populate page tables to trigger the allocation. > > > > > > Signed-off-by: Zhihong Wang <zhihong.wang@intel.com> > > > > Nice, especially on slow machines or with large memory. > > > > Acked-by: Stephen Hemminger <stephen@networkplumber.org> > > Yes very nice. > I think it's too late to integrate this change which can have some > unpredictable side effects. > Do you agree to wait for 2.3? What side effects? Either it is zero or it is not. Only some broken architecture would have an issue. ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [PATCH 2/2] lib/librte_eal: Remove unnecessary hugepage zero-filling 2015-11-24 22:44 ` Stephen Hemminger @ 2015-11-24 23:04 ` Thomas Monjalon 2015-11-25 1:57 ` Yuanhan Liu 2015-11-25 1:59 ` Wang, Zhihong 0 siblings, 2 replies; 10+ messages in thread From: Thomas Monjalon @ 2015-11-24 23:04 UTC (permalink / raw) To: Stephen Hemminger; +Cc: dev 2015-11-24 14:44, Stephen Hemminger: > On Tue, 24 Nov 2015 22:13:28 +0100 > Thomas Monjalon <thomas.monjalon@6wind.com> wrote: > > > 2015-11-22 18:28, Stephen Hemminger: > > > On Sun, 22 Nov 2015 14:13:35 -0500 > > > Zhihong Wang <zhihong.wang@intel.com> wrote: > > > > > > > The kernel fills new allocated (huge) pages with zeros. > > > > DPDK just has to populate page tables to trigger the allocation. > > > > > > > > Signed-off-by: Zhihong Wang <zhihong.wang@intel.com> > > > > > > Nice, especially on slow machines or with large memory. > > > > > > Acked-by: Stephen Hemminger <stephen@networkplumber.org> > > > > Yes very nice. > > I think it's too late to integrate this change which can have some > > unpredictable side effects. > > Do you agree to wait for 2.3? > > What side effects? Either it is zero or it is not. > Only some broken architecture would have an issue. I mean it changes the memory allocator behaviour. It's not something we want to discover a new bug just before the release. This kind of important change must be integrated at the beginning of the release cycle. I'm asking for opinions because it would be really nice to have. ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [PATCH 2/2] lib/librte_eal: Remove unnecessary hugepage zero-filling 2015-11-24 23:04 ` Thomas Monjalon @ 2015-11-25 1:57 ` Yuanhan Liu 2015-11-25 1:59 ` Wang, Zhihong 1 sibling, 0 replies; 10+ messages in thread From: Yuanhan Liu @ 2015-11-25 1:57 UTC (permalink / raw) To: Thomas Monjalon; +Cc: dev On Wed, Nov 25, 2015 at 12:04:16AM +0100, Thomas Monjalon wrote: > 2015-11-24 14:44, Stephen Hemminger: > > On Tue, 24 Nov 2015 22:13:28 +0100 > > Thomas Monjalon <thomas.monjalon@6wind.com> wrote: > > > > > 2015-11-22 18:28, Stephen Hemminger: > > > > On Sun, 22 Nov 2015 14:13:35 -0500 > > > > Zhihong Wang <zhihong.wang@intel.com> wrote: > > > > > > > > > The kernel fills new allocated (huge) pages with zeros. > > > > > DPDK just has to populate page tables to trigger the allocation. > > > > > > > > > > Signed-off-by: Zhihong Wang <zhihong.wang@intel.com> > > > > > > > > Nice, especially on slow machines or with large memory. > > > > > > > > Acked-by: Stephen Hemminger <stephen@networkplumber.org> > > > > > > Yes very nice. > > > I think it's too late to integrate this change which can have some > > > unpredictable side effects. > > > Do you agree to wait for 2.3? > > > > What side effects? Either it is zero or it is not. > > Only some broken architecture would have an issue. > > I mean it changes the memory allocator behaviour. It's not something we > want to discover a new bug just before the release. > This kind of important change must be integrated at the beginning of the > release cycle. + 1 And it could be a new feature (or highlight) of 2.3: reduced dpdk startup time by ... :) --yliu ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [PATCH 2/2] lib/librte_eal: Remove unnecessary hugepage zero-filling 2015-11-24 23:04 ` Thomas Monjalon 2015-11-25 1:57 ` Yuanhan Liu @ 2015-11-25 1:59 ` Wang, Zhihong 1 sibling, 0 replies; 10+ messages in thread From: Wang, Zhihong @ 2015-11-25 1:59 UTC (permalink / raw) To: Thomas Monjalon, Stephen Hemminger; +Cc: dev > -----Original Message----- > From: Thomas Monjalon [mailto:thomas.monjalon@6wind.com] > Sent: Wednesday, November 25, 2015 7:04 AM > To: Stephen Hemminger <stephen@networkplumber.org> > Cc: Wang, Zhihong <zhihong.wang@intel.com>; dev@dpdk.org > Subject: Re: [dpdk-dev] [PATCH 2/2] lib/librte_eal: Remove unnecessary > hugepage zero-filling > > 2015-11-24 14:44, Stephen Hemminger: > > On Tue, 24 Nov 2015 22:13:28 +0100 > > Thomas Monjalon <thomas.monjalon@6wind.com> wrote: > > > > > 2015-11-22 18:28, Stephen Hemminger: > > > > On Sun, 22 Nov 2015 14:13:35 -0500 Zhihong Wang > > > > <zhihong.wang@intel.com> wrote: > > > > > > > > > The kernel fills new allocated (huge) pages with zeros. > > > > > DPDK just has to populate page tables to trigger the allocation. > > > > > > > > > > Signed-off-by: Zhihong Wang <zhihong.wang@intel.com> > > > > > > > > Nice, especially on slow machines or with large memory. > > > > > > > > Acked-by: Stephen Hemminger <stephen@networkplumber.org> > > > > > > Yes very nice. > > > I think it's too late to integrate this change which can have some > > > unpredictable side effects. > > > Do you agree to wait for 2.3? > > > > What side effects? Either it is zero or it is not. > > Only some broken architecture would have an issue. > > I mean it changes the memory allocator behaviour. It's not something we want to > discover a new bug just before the release. > This kind of important change must be integrated at the beginning of the release > cycle. > I'm asking for opinions because it would be really nice to have. Literally this patch doesn't change anything, it just keeps DPDK from zero-filling pages again which have just been zero-filled. It would be nice to have this patch in DPDK 2.2 since it can reduce the startup time nearly by half for hugepage cases. But I understand longer merge/test window make it safer for a release. It makes sense either way. ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [PATCH 0/2] Reduce DPDK initialization time 2015-11-22 19:13 [PATCH 0/2] Reduce DPDK initialization time Zhihong Wang 2015-11-22 19:13 ` [PATCH 1/2] lib/librte_eal: Reduce timer " Zhihong Wang 2015-11-22 19:13 ` [PATCH 2/2] lib/librte_eal: Remove unnecessary hugepage zero-filling Zhihong Wang @ 2016-01-21 14:59 ` Thomas Monjalon 2 siblings, 0 replies; 10+ messages in thread From: Thomas Monjalon @ 2016-01-21 14:59 UTC (permalink / raw) To: Zhihong Wang; +Cc: dev 2015-11-22 14:13, Zhihong Wang: > This patch aims to reduce DPDK initialization time, which is important in cases such as micro service. > > Changes are: > > 1. Reduce timer initialization time > > 2. Remove unnecessary hugepage zero-filling operations > > With this patch: > > 1. Timer initialization time can be reduced by 4/10 second > > 2. Memory initialization time can be reduced nearly by half > > The 2nd topic has been brought up before in this thread: > http://dpdk.org/dev/patchwork/patch/4219/ Applied, thanks ^ permalink raw reply [flat|nested] 10+ messages in thread
end of thread, other threads:[~2016-01-21 15:00 UTC | newest] Thread overview: 10+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2015-11-22 19:13 [PATCH 0/2] Reduce DPDK initialization time Zhihong Wang 2015-11-22 19:13 ` [PATCH 1/2] lib/librte_eal: Reduce timer " Zhihong Wang 2015-11-22 19:13 ` [PATCH 2/2] lib/librte_eal: Remove unnecessary hugepage zero-filling Zhihong Wang 2015-11-23 2:28 ` Stephen Hemminger 2015-11-24 21:13 ` Thomas Monjalon 2015-11-24 22:44 ` Stephen Hemminger 2015-11-24 23:04 ` Thomas Monjalon 2015-11-25 1:57 ` Yuanhan Liu 2015-11-25 1:59 ` Wang, Zhihong 2016-01-21 14:59 ` [PATCH 0/2] Reduce DPDK initialization time Thomas Monjalon
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.