* [PATCH 0/1][PULL] bitbake: parsing time improvement @ 2011-08-29 7:33 Dongxiao Xu 2011-08-29 7:33 ` [PATCH 1/1] data_smart.py: make use of expand cache in getVar() Dongxiao Xu 0 siblings, 1 reply; 8+ messages in thread From: Dongxiao Xu @ 2011-08-29 7:33 UTC (permalink / raw) To: bitbake-devel Hi Richard, This pull request is to improve parsing time by getting use of the expand cache, please help to review and pull. Thanks, Dongxiao The following changes since commit 0423587db09f6f28cf9d801f5657a84157f42dbe: hob: disable some menu entries whilst build is in progress (2011-08-24 19:32:42 -0700) are available in the git repository at: git://git.pokylinux.org/poky-contrib dxu4/parse http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=dxu4/parse Dongxiao Xu (1): data_smart.py: make use of expand cache in getVar() lib/bb/data_smart.py | 7 ++++++- 1 files changed, 6 insertions(+), 1 deletions(-) ^ permalink raw reply [flat|nested] 8+ messages in thread
* [PATCH 1/1] data_smart.py: make use of expand cache in getVar() 2011-08-29 7:33 [PATCH 0/1][PULL] bitbake: parsing time improvement Dongxiao Xu @ 2011-08-29 7:33 ` Dongxiao Xu 2011-08-29 12:55 ` Richard Purdie 0 siblings, 1 reply; 8+ messages in thread From: Dongxiao Xu @ 2011-08-29 7:33 UTC (permalink / raw) To: bitbake-devel Currently if passing expand=True to getVar() function, it will pass the handling to getVarFlag(), which doesn't get any benefit from the expand cache. Call the expand() function separately in getVar() to make use of the expand cache, which can decrease the parsing time by 40%. (from current 49s to 27s) Signed-off-by: Dongxiao Xu <dongxiao.xu@intel.com> --- lib/bb/data_smart.py | 7 ++++++- 1 files changed, 6 insertions(+), 1 deletions(-) diff --git a/lib/bb/data_smart.py b/lib/bb/data_smart.py index 301f9e3..d8ba24f 100644 --- a/lib/bb/data_smart.py +++ b/lib/bb/data_smart.py @@ -268,7 +268,12 @@ class DataSmart(MutableMapping): self.dict[var]["content"] = value def getVar(self, var, expand=False, noweakdefault=False): - return self.getVarFlag(var, "content", expand, noweakdefault) + value = self.getVarFlag(var, "content", False, noweakdefault) + + # Call expand() separately to make use of the expand cache + if expand and value: + return self.expand(value, var) + return value def renameVar(self, key, newkey): """ -- 1.7.1 ^ permalink raw reply related [flat|nested] 8+ messages in thread
* Re: [PATCH 1/1] data_smart.py: make use of expand cache in getVar() 2011-08-29 7:33 ` [PATCH 1/1] data_smart.py: make use of expand cache in getVar() Dongxiao Xu @ 2011-08-29 12:55 ` Richard Purdie 2011-08-29 13:07 ` Xu, Dongxiao 0 siblings, 1 reply; 8+ messages in thread From: Richard Purdie @ 2011-08-29 12:55 UTC (permalink / raw) To: Dongxiao Xu; +Cc: bitbake-devel On Mon, 2011-08-29 at 15:33 +0800, Dongxiao Xu wrote: > Currently if passing expand=True to getVar() function, it will pass the > handling to getVarFlag(), which doesn't get any benefit from the expand > cache. > > Call the expand() function separately in getVar() to make use of the > expand cache, which can decrease the parsing time by 40%. > (from current 49s to 27s) > > Signed-off-by: Dongxiao Xu <dongxiao.xu@intel.com> > --- > lib/bb/data_smart.py | 7 ++++++- > 1 files changed, 6 insertions(+), 1 deletions(-) > > diff --git a/lib/bb/data_smart.py b/lib/bb/data_smart.py > index 301f9e3..d8ba24f 100644 > --- a/lib/bb/data_smart.py > +++ b/lib/bb/data_smart.py > @@ -268,7 +268,12 @@ class DataSmart(MutableMapping): > self.dict[var]["content"] = value > > def getVar(self, var, expand=False, noweakdefault=False): > - return self.getVarFlag(var, "content", expand, noweakdefault) > + value = self.getVarFlag(var, "content", False, noweakdefault) > + > + # Call expand() separately to make use of the expand cache > + if expand and value: > + return self.expand(value, var) > + return value > Why can't we have getVarFlag use the expand cache? Cheers, Richard ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [PATCH 1/1] data_smart.py: make use of expand cache in getVar() 2011-08-29 12:55 ` Richard Purdie @ 2011-08-29 13:07 ` Xu, Dongxiao 2011-08-29 13:13 ` Richard Purdie 0 siblings, 1 reply; 8+ messages in thread From: Xu, Dongxiao @ 2011-08-29 13:07 UTC (permalink / raw) To: Richard Purdie; +Cc: bitbake-devel Hi Richard, > -----Original Message----- > From: Richard Purdie [mailto:richard.purdie@linuxfoundation.org] > Sent: Monday, August 29, 2011 8:55 PM > To: Xu, Dongxiao > Cc: bitbake-devel@lists.openembedded.org > Subject: Re: [bitbake-devel] [PATCH 1/1] data_smart.py: make use of expand > cache in getVar() > > On Mon, 2011-08-29 at 15:33 +0800, Dongxiao Xu wrote: > > Currently if passing expand=True to getVar() function, it will pass > > the handling to getVarFlag(), which doesn't get any benefit from the > > expand cache. > > > > Call the expand() function separately in getVar() to make use of the > > expand cache, which can decrease the parsing time by 40%. > > (from current 49s to 27s) > > > > Signed-off-by: Dongxiao Xu <dongxiao.xu@intel.com> > > --- > > lib/bb/data_smart.py | 7 ++++++- > > 1 files changed, 6 insertions(+), 1 deletions(-) > > > > diff --git a/lib/bb/data_smart.py b/lib/bb/data_smart.py index > > 301f9e3..d8ba24f 100644 > > --- a/lib/bb/data_smart.py > > +++ b/lib/bb/data_smart.py > > @@ -268,7 +268,12 @@ class DataSmart(MutableMapping): > > self.dict[var]["content"] = value > > > > def getVar(self, var, expand=False, noweakdefault=False): > > - return self.getVarFlag(var, "content", expand, noweakdefault) > > + value = self.getVarFlag(var, "content", False, noweakdefault) > > + > > + # Call expand() separately to make use of the expand cache > > + if expand and value: > > + return self.expand(value, var) > > + return value > > > > Why can't we have getVarFlag use the expand cache? For expand cache, the format is a list: expand_cache[varname]. It doesn't contain information of flag values. Thanks, Dongxiao > > Cheers, > > Richard ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [PATCH 1/1] data_smart.py: make use of expand cache in getVar() 2011-08-29 13:07 ` Xu, Dongxiao @ 2011-08-29 13:13 ` Richard Purdie 2011-08-29 13:15 ` Xu, Dongxiao 2011-08-29 14:21 ` Xu, Dongxiao 0 siblings, 2 replies; 8+ messages in thread From: Richard Purdie @ 2011-08-29 13:13 UTC (permalink / raw) To: Xu, Dongxiao; +Cc: bitbake-devel On Mon, 2011-08-29 at 21:07 +0800, Xu, Dongxiao wrote: > Hi Richard, > > > -----Original Message----- > > From: Richard Purdie [mailto:richard.purdie@linuxfoundation.org] > > Sent: Monday, August 29, 2011 8:55 PM > > To: Xu, Dongxiao > > Cc: bitbake-devel@lists.openembedded.org > > Subject: Re: [bitbake-devel] [PATCH 1/1] data_smart.py: make use of expand > > cache in getVar() > > > > On Mon, 2011-08-29 at 15:33 +0800, Dongxiao Xu wrote: > > > Currently if passing expand=True to getVar() function, it will pass > > > the handling to getVarFlag(), which doesn't get any benefit from the > > > expand cache. > > > > > > Call the expand() function separately in getVar() to make use of the > > > expand cache, which can decrease the parsing time by 40%. > > > (from current 49s to 27s) > > > > > > Signed-off-by: Dongxiao Xu <dongxiao.xu@intel.com> > > > --- > > > lib/bb/data_smart.py | 7 ++++++- > > > 1 files changed, 6 insertions(+), 1 deletions(-) > > > > > > diff --git a/lib/bb/data_smart.py b/lib/bb/data_smart.py index > > > 301f9e3..d8ba24f 100644 > > > --- a/lib/bb/data_smart.py > > > +++ b/lib/bb/data_smart.py > > > @@ -268,7 +268,12 @@ class DataSmart(MutableMapping): > > > self.dict[var]["content"] = value > > > > > > def getVar(self, var, expand=False, noweakdefault=False): > > > - return self.getVarFlag(var, "content", expand, noweakdefault) > > > + value = self.getVarFlag(var, "content", False, noweakdefault) > > > + > > > + # Call expand() separately to make use of the expand cache > > > + if expand and value: > > > + return self.expand(value, var) > > > + return value > > > > > > > Why can't we have getVarFlag use the expand cache? > > For expand cache, the format is a list: expand_cache[varname]. > It doesn't contain information of flag values. Good point. I'm left wondering why we don't key that cache on the unexpanded value rather than a specific variable name (or name+flag). I'll take this patch but its something we should consider... Cheers, Richard ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [PATCH 1/1] data_smart.py: make use of expand cache in getVar() 2011-08-29 13:13 ` Richard Purdie @ 2011-08-29 13:15 ` Xu, Dongxiao 2011-08-29 14:21 ` Xu, Dongxiao 1 sibling, 0 replies; 8+ messages in thread From: Xu, Dongxiao @ 2011-08-29 13:15 UTC (permalink / raw) To: Richard Purdie; +Cc: bitbake-devel > -----Original Message----- > From: Richard Purdie [mailto:richard.purdie@linuxfoundation.org] > Sent: Monday, August 29, 2011 9:13 PM > To: Xu, Dongxiao > Cc: bitbake-devel@lists.openembedded.org > Subject: RE: [bitbake-devel] [PATCH 1/1] data_smart.py: make use of expand > cache in getVar() > > On Mon, 2011-08-29 at 21:07 +0800, Xu, Dongxiao wrote: > > Hi Richard, > > > > > -----Original Message----- > > > From: Richard Purdie [mailto:richard.purdie@linuxfoundation.org] > > > Sent: Monday, August 29, 2011 8:55 PM > > > To: Xu, Dongxiao > > > Cc: bitbake-devel@lists.openembedded.org > > > Subject: Re: [bitbake-devel] [PATCH 1/1] data_smart.py: make use of > > > expand cache in getVar() > > > > > > On Mon, 2011-08-29 at 15:33 +0800, Dongxiao Xu wrote: > > > > Currently if passing expand=True to getVar() function, it will > > > > pass the handling to getVarFlag(), which doesn't get any benefit > > > > from the expand cache. > > > > > > > > Call the expand() function separately in getVar() to make use of > > > > the expand cache, which can decrease the parsing time by 40%. > > > > (from current 49s to 27s) > > > > > > > > Signed-off-by: Dongxiao Xu <dongxiao.xu@intel.com> > > > > --- > > > > lib/bb/data_smart.py | 7 ++++++- > > > > 1 files changed, 6 insertions(+), 1 deletions(-) > > > > > > > > diff --git a/lib/bb/data_smart.py b/lib/bb/data_smart.py index > > > > 301f9e3..d8ba24f 100644 > > > > --- a/lib/bb/data_smart.py > > > > +++ b/lib/bb/data_smart.py > > > > @@ -268,7 +268,12 @@ class DataSmart(MutableMapping): > > > > self.dict[var]["content"] = value > > > > > > > > def getVar(self, var, expand=False, noweakdefault=False): > > > > - return self.getVarFlag(var, "content", expand, noweakdefault) > > > > + value = self.getVarFlag(var, "content", False, > > > > + noweakdefault) > > > > + > > > > + # Call expand() separately to make use of the expand cache > > > > + if expand and value: > > > > + return self.expand(value, var) > > > > + return value > > > > > > > > > > Why can't we have getVarFlag use the expand cache? > > > > For expand cache, the format is a list: expand_cache[varname]. > > It doesn't contain information of flag values. > > Good point. > > I'm left wondering why we don't key that cache on the unexpanded value rather > than a specific variable name (or name+flag). > > I'll take this patch but its something we should consider... Thanks, I will step into this point further and see if (name+flag) type of cache could benefit us more. Thanks, Dongxiao > > Cheers, > > Richard > > ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [PATCH 1/1] data_smart.py: make use of expand cache in getVar() 2011-08-29 13:13 ` Richard Purdie 2011-08-29 13:15 ` Xu, Dongxiao @ 2011-08-29 14:21 ` Xu, Dongxiao 2011-08-30 21:09 ` Richard Purdie 1 sibling, 1 reply; 8+ messages in thread From: Xu, Dongxiao @ 2011-08-29 14:21 UTC (permalink / raw) To: Richard Purdie; +Cc: bitbake-devel Hi Richard, > -----Original Message----- > From: Xu, Dongxiao > Sent: Monday, August 29, 2011 9:16 PM > To: Richard Purdie > Cc: bitbake-devel@lists.openembedded.org > Subject: RE: [bitbake-devel] [PATCH 1/1] data_smart.py: make use of expand > cache in getVar() > > > -----Original Message----- > > From: Richard Purdie [mailto:richard.purdie@linuxfoundation.org] > > Sent: Monday, August 29, 2011 9:13 PM > > To: Xu, Dongxiao > > Cc: bitbake-devel@lists.openembedded.org > > Subject: RE: [bitbake-devel] [PATCH 1/1] data_smart.py: make use of > > expand cache in getVar() > > > > On Mon, 2011-08-29 at 21:07 +0800, Xu, Dongxiao wrote: > > > Hi Richard, > > > > > > > -----Original Message----- > > > > From: Richard Purdie [mailto:richard.purdie@linuxfoundation.org] > > > > Sent: Monday, August 29, 2011 8:55 PM > > > > To: Xu, Dongxiao > > > > Cc: bitbake-devel@lists.openembedded.org > > > > Subject: Re: [bitbake-devel] [PATCH 1/1] data_smart.py: make use > > > > of expand cache in getVar() > > > > > > > > On Mon, 2011-08-29 at 15:33 +0800, Dongxiao Xu wrote: > > > > > Currently if passing expand=True to getVar() function, it will > > > > > pass the handling to getVarFlag(), which doesn't get any benefit > > > > > from the expand cache. > > > > > > > > > > Call the expand() function separately in getVar() to make use of > > > > > the expand cache, which can decrease the parsing time by 40%. > > > > > (from current 49s to 27s) > > > > > > > > > > Signed-off-by: Dongxiao Xu <dongxiao.xu@intel.com> > > > > > --- > > > > > lib/bb/data_smart.py | 7 ++++++- > > > > > 1 files changed, 6 insertions(+), 1 deletions(-) > > > > > > > > > > diff --git a/lib/bb/data_smart.py b/lib/bb/data_smart.py index > > > > > 301f9e3..d8ba24f 100644 > > > > > --- a/lib/bb/data_smart.py > > > > > +++ b/lib/bb/data_smart.py > > > > > @@ -268,7 +268,12 @@ class DataSmart(MutableMapping): > > > > > self.dict[var]["content"] = value > > > > > > > > > > def getVar(self, var, expand=False, noweakdefault=False): > > > > > - return self.getVarFlag(var, "content", expand, noweakdefault) > > > > > + value = self.getVarFlag(var, "content", False, > > > > > + noweakdefault) > > > > > + > > > > > + # Call expand() separately to make use of the expand cache > > > > > + if expand and value: > > > > > + return self.expand(value, var) > > > > > + return value > > > > > > > > > > > > > Why can't we have getVarFlag use the expand cache? > > > > > > For expand cache, the format is a list: expand_cache[varname]. > > > It doesn't contain information of flag values. > > > > Good point. > > > > I'm left wondering why we don't key that cache on the unexpanded value > > rather than a specific variable name (or name+flag). > > > > I'll take this patch but its something we should consider... > > Thanks, I will step into this point further and see if (name+flag) type of cache > could benefit us more. Sorry I must misunderstand your point just now... For caching of unexpanded value, there are too many/uncertainty of types, like: ${TUNE_ARCH} ${@bb.parse.BBHandler.vars_from_file(bb.data.getVar('FILE',d),d)[0] or 'defaultpkgname'} ${@bb.utils.contains("TUNE_FEATURES", "m32", "${X86ARCH32}", "" ,d)}${@bb.utils.contains("TUNE_FEATURES", "mx32", "${X86ARCH64}", "" ,d)}${@bb.utils.contains("TUNE_FEATURES", "m64", "${X86ARCH64}", "" ,d)} For simple variables, it is easy to cache, however for the expressions, it may not suitable to cache. Besides, I also tried the (name+flag) types of cache, but it doesn't bring benefit (from 27s to 28s). Therefore it is suppose that most of the calling should be getVar(). Those specifically getting flag value should be not much. Thanks, Dongxiao > > Thanks, > Dongxiao > > > > > Cheers, > > > > Richard > > > > ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [PATCH 1/1] data_smart.py: make use of expand cache in getVar() 2011-08-29 14:21 ` Xu, Dongxiao @ 2011-08-30 21:09 ` Richard Purdie 0 siblings, 0 replies; 8+ messages in thread From: Richard Purdie @ 2011-08-30 21:09 UTC (permalink / raw) To: Xu, Dongxiao; +Cc: bitbake-devel On Mon, 2011-08-29 at 22:21 +0800, Xu, Dongxiao wrote: > > -----Original Message----- > > From: Xu, Dongxiao > > Sent: Monday, August 29, 2011 9:16 PM > > To: Richard Purdie > > Cc: bitbake-devel@lists.openembedded.org > > Subject: RE: [bitbake-devel] [PATCH 1/1] data_smart.py: make use of expand > > cache in getVar() > > > > > -----Original Message----- > > > From: Richard Purdie [mailto:richard.purdie@linuxfoundation.org] > > > Sent: Monday, August 29, 2011 9:13 PM > > > To: Xu, Dongxiao > > > Cc: bitbake-devel@lists.openembedded.org > > > Subject: RE: [bitbake-devel] [PATCH 1/1] data_smart.py: make use of > > > expand cache in getVar() > > > > > > > > > I'm left wondering why we don't key that cache on the unexpanded value > > > rather than a specific variable name (or name+flag). > > > > > > I'll take this patch but its something we should consider... > > > > Thanks, I will step into this point further and see if (name+flag) type of cache > > could benefit us more. > > Sorry I must misunderstand your point just now... > > For caching of unexpanded value, there are too many/uncertainty of types, like: > > ${TUNE_ARCH} > ${@bb.parse.BBHandler.vars_from_file(bb.data.getVar('FILE',d),d)[0] or 'defaultpkgname'} > ${@bb.utils.contains("TUNE_FEATURES", "m32", "${X86ARCH32}", "" ,d)}${@bb.utils.contains("TUNE_FEATURES", "mx32", "${X86ARCH64}", "" ,d)}${@bb.utils.contains("TUNE_FEATURES", "m64", "${X86ARCH64}", "" ,d)} > > For simple variables, it is easy to cache, however for the expressions, it may not suitable to cache. It shouldn't really matter what the expression is, we should just be able to cache the value it evaluates to. We invalidate the cache upon any write to the data store that can potentially change a variables contents. This shouldn't really be any different to the current behaviour of the code (where caching doesn't cause a problem). Cheers, Richard ^ permalink raw reply [flat|nested] 8+ messages in thread
end of thread, other threads:[~2011-08-30 21:14 UTC | newest] Thread overview: 8+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2011-08-29 7:33 [PATCH 0/1][PULL] bitbake: parsing time improvement Dongxiao Xu 2011-08-29 7:33 ` [PATCH 1/1] data_smart.py: make use of expand cache in getVar() Dongxiao Xu 2011-08-29 12:55 ` Richard Purdie 2011-08-29 13:07 ` Xu, Dongxiao 2011-08-29 13:13 ` Richard Purdie 2011-08-29 13:15 ` Xu, Dongxiao 2011-08-29 14:21 ` Xu, Dongxiao 2011-08-30 21:09 ` Richard Purdie
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.