[PATCH 0/1][PULL] bitbake: parsing time improvement

[PATCH 0/1][PULL] bitbake: parsing time improvement

[Dongxiao Xu]

Hi Richard,

This pull request is to improve parsing time by getting use of the
expand cache, please help to review and pull.

Thanks,
Dongxiao

The following changes since commit 0423587db09f6f28cf9d801f5657a84157f42dbe:

  hob: disable some menu entries whilst build is in progress (2011-08-24 19:32:42 -0700)

are available in the git repository at:
  git://git.pokylinux.org/poky-contrib dxu4/parse
  http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=dxu4/parse

Dongxiao Xu (1):
  data_smart.py: make use of expand cache in getVar()

 lib/bb/data_smart.py |    7 ++++++-
 1 files changed, 6 insertions(+), 1 deletions(-)

[PATCH 1/1] data_smart.py: make use of expand cache in getVar()

[Dongxiao Xu]

Currently if passing expand=True to getVar() function, it will pass the
handling to getVarFlag(), which doesn't get any benefit from the expand
cache.

Call the expand() function separately in getVar() to make use of the
expand cache, which can decrease the parsing time by 40%.
(from current 49s to 27s)

Signed-off-by: Dongxiao Xu <dongxiao.xu@intel.com>
---
 lib/bb/data_smart.py |    7 ++++++-
 1 files changed, 6 insertions(+), 1 deletions(-)

diff --git a/lib/bb/data_smart.py b/lib/bb/data_smart.py
index 301f9e3..d8ba24f 100644
--- a/lib/bb/data_smart.py
+++ b/lib/bb/data_smart.py
@@ -268,7 +268,12 @@ class DataSmart(MutableMapping):
         self.dict[var]["content"] = value
 
     def getVar(self, var, expand=False, noweakdefault=False):
-        return self.getVarFlag(var, "content", expand, noweakdefault)
+        value = self.getVarFlag(var, "content", False, noweakdefault)
+
+        # Call expand() separately to make use of the expand cache
+        if expand and value:
+            return self.expand(value, var)
+        return value
 
     def renameVar(self, key, newkey):
         """
-- 
1.7.1

Re: [PATCH 1/1] data_smart.py: make use of expand cache in getVar()

[Richard Purdie]

On Mon, 2011-08-29 at 15:33 +0800, Dongxiao Xu wrote:
> Currently if passing expand=True to getVar() function, it will pass the
> handling to getVarFlag(), which doesn't get any benefit from the expand
> cache.
> 
> Call the expand() function separately in getVar() to make use of the
> expand cache, which can decrease the parsing time by 40%.
> (from current 49s to 27s)
> 
> Signed-off-by: Dongxiao Xu <dongxiao.xu@intel.com>
> ---
>  lib/bb/data_smart.py |    7 ++++++-
>  1 files changed, 6 insertions(+), 1 deletions(-)
> 
> diff --git a/lib/bb/data_smart.py b/lib/bb/data_smart.py
> index 301f9e3..d8ba24f 100644
> --- a/lib/bb/data_smart.py
> +++ b/lib/bb/data_smart.py
> @@ -268,7 +268,12 @@ class DataSmart(MutableMapping):
>          self.dict[var]["content"] = value
>  
>      def getVar(self, var, expand=False, noweakdefault=False):
> -        return self.getVarFlag(var, "content", expand, noweakdefault)
> +        value = self.getVarFlag(var, "content", False, noweakdefault)
> +
> +        # Call expand() separately to make use of the expand cache
> +        if expand and value:
> +            return self.expand(value, var)
> +        return value
>  

Why can't we have getVarFlag use the expand cache?

Cheers,

Richard

Re: [PATCH 1/1] data_smart.py: make use of expand cache in getVar()

[Xu, Dongxiao]

Hi Richard,

> -----Original Message-----
> From: Richard Purdie [mailto:richard.purdie@linuxfoundation.org]
> Sent: Monday, August 29, 2011 8:55 PM
> To: Xu, Dongxiao
> Cc: bitbake-devel@lists.openembedded.org
> Subject: Re: [bitbake-devel] [PATCH 1/1] data_smart.py: make use of expand
> cache in getVar()
> 
> On Mon, 2011-08-29 at 15:33 +0800, Dongxiao Xu wrote:
> > Currently if passing expand=True to getVar() function, it will pass
> > the handling to getVarFlag(), which doesn't get any benefit from the
> > expand cache.
> >
> > Call the expand() function separately in getVar() to make use of the
> > expand cache, which can decrease the parsing time by 40%.
> > (from current 49s to 27s)
> >
> > Signed-off-by: Dongxiao Xu <dongxiao.xu@intel.com>
> > ---
> >  lib/bb/data_smart.py |    7 ++++++-
> >  1 files changed, 6 insertions(+), 1 deletions(-)
> >
> > diff --git a/lib/bb/data_smart.py b/lib/bb/data_smart.py index
> > 301f9e3..d8ba24f 100644
> > --- a/lib/bb/data_smart.py
> > +++ b/lib/bb/data_smart.py
> > @@ -268,7 +268,12 @@ class DataSmart(MutableMapping):
> >          self.dict[var]["content"] = value
> >
> >      def getVar(self, var, expand=False, noweakdefault=False):
> > -        return self.getVarFlag(var, "content", expand, noweakdefault)
> > +        value = self.getVarFlag(var, "content", False, noweakdefault)
> > +
> > +        # Call expand() separately to make use of the expand cache
> > +        if expand and value:
> > +            return self.expand(value, var)
> > +        return value
> >
> 
> Why can't we have getVarFlag use the expand cache?

For expand cache, the format is a list: expand_cache[varname].
It doesn't contain information of flag values.

Thanks,
Dongxiao

> 
> Cheers,
> 
> Richard

Re: [PATCH 1/1] data_smart.py: make use of expand cache in getVar()

[Richard Purdie]

On Mon, 2011-08-29 at 21:07 +0800, Xu, Dongxiao wrote:
> Hi Richard,
> 
> > -----Original Message-----
> > From: Richard Purdie [mailto:richard.purdie@linuxfoundation.org]
> > Sent: Monday, August 29, 2011 8:55 PM
> > To: Xu, Dongxiao
> > Cc: bitbake-devel@lists.openembedded.org
> > Subject: Re: [bitbake-devel] [PATCH 1/1] data_smart.py: make use of expand
> > cache in getVar()
> > 
> > On Mon, 2011-08-29 at 15:33 +0800, Dongxiao Xu wrote:
> > > Currently if passing expand=True to getVar() function, it will pass
> > > the handling to getVarFlag(), which doesn't get any benefit from the
> > > expand cache.
> > >
> > > Call the expand() function separately in getVar() to make use of the
> > > expand cache, which can decrease the parsing time by 40%.
> > > (from current 49s to 27s)
> > >
> > > Signed-off-by: Dongxiao Xu <dongxiao.xu@intel.com>
> > > ---
> > >  lib/bb/data_smart.py |    7 ++++++-
> > >  1 files changed, 6 insertions(+), 1 deletions(-)
> > >
> > > diff --git a/lib/bb/data_smart.py b/lib/bb/data_smart.py index
> > > 301f9e3..d8ba24f 100644
> > > --- a/lib/bb/data_smart.py
> > > +++ b/lib/bb/data_smart.py
> > > @@ -268,7 +268,12 @@ class DataSmart(MutableMapping):
> > >          self.dict[var]["content"] = value
> > >
> > >      def getVar(self, var, expand=False, noweakdefault=False):
> > > -        return self.getVarFlag(var, "content", expand, noweakdefault)
> > > +        value = self.getVarFlag(var, "content", False, noweakdefault)
> > > +
> > > +        # Call expand() separately to make use of the expand cache
> > > +        if expand and value:
> > > +            return self.expand(value, var)
> > > +        return value
> > >
> > 
> > Why can't we have getVarFlag use the expand cache?
> 
> For expand cache, the format is a list: expand_cache[varname].
> It doesn't contain information of flag values.

Good point.

I'm left wondering why we don't key that cache on the unexpanded value
rather than a specific variable name (or name+flag).

I'll take this patch but its something we should consider...

Cheers,

Richard

Re: [PATCH 1/1] data_smart.py: make use of expand cache in getVar()

[Xu, Dongxiao]

> -----Original Message-----
> From: Richard Purdie [mailto:richard.purdie@linuxfoundation.org]
> Sent: Monday, August 29, 2011 9:13 PM
> To: Xu, Dongxiao
> Cc: bitbake-devel@lists.openembedded.org
> Subject: RE: [bitbake-devel] [PATCH 1/1] data_smart.py: make use of expand
> cache in getVar()
> 
> On Mon, 2011-08-29 at 21:07 +0800, Xu, Dongxiao wrote:
> > Hi Richard,
> >
> > > -----Original Message-----
> > > From: Richard Purdie [mailto:richard.purdie@linuxfoundation.org]
> > > Sent: Monday, August 29, 2011 8:55 PM
> > > To: Xu, Dongxiao
> > > Cc: bitbake-devel@lists.openembedded.org
> > > Subject: Re: [bitbake-devel] [PATCH 1/1] data_smart.py: make use of
> > > expand cache in getVar()
> > >
> > > On Mon, 2011-08-29 at 15:33 +0800, Dongxiao Xu wrote:
> > > > Currently if passing expand=True to getVar() function, it will
> > > > pass the handling to getVarFlag(), which doesn't get any benefit
> > > > from the expand cache.
> > > >
> > > > Call the expand() function separately in getVar() to make use of
> > > > the expand cache, which can decrease the parsing time by 40%.
> > > > (from current 49s to 27s)
> > > >
> > > > Signed-off-by: Dongxiao Xu <dongxiao.xu@intel.com>
> > > > ---
> > > >  lib/bb/data_smart.py |    7 ++++++-
> > > >  1 files changed, 6 insertions(+), 1 deletions(-)
> > > >
> > > > diff --git a/lib/bb/data_smart.py b/lib/bb/data_smart.py index
> > > > 301f9e3..d8ba24f 100644
> > > > --- a/lib/bb/data_smart.py
> > > > +++ b/lib/bb/data_smart.py
> > > > @@ -268,7 +268,12 @@ class DataSmart(MutableMapping):
> > > >          self.dict[var]["content"] = value
> > > >
> > > >      def getVar(self, var, expand=False, noweakdefault=False):
> > > > -        return self.getVarFlag(var, "content", expand, noweakdefault)
> > > > +        value = self.getVarFlag(var, "content", False,
> > > > + noweakdefault)
> > > > +
> > > > +        # Call expand() separately to make use of the expand cache
> > > > +        if expand and value:
> > > > +            return self.expand(value, var)
> > > > +        return value
> > > >
> > >
> > > Why can't we have getVarFlag use the expand cache?
> >
> > For expand cache, the format is a list: expand_cache[varname].
> > It doesn't contain information of flag values.
> 
> Good point.
> 
> I'm left wondering why we don't key that cache on the unexpanded value rather
> than a specific variable name (or name+flag).
> 
> I'll take this patch but its something we should consider...

Thanks, I will step into this point further and see if (name+flag) type of cache could benefit us more.

Thanks,
Dongxiao

> 
> Cheers,
> 
> Richard
> 
> 

Re: [PATCH 1/1] data_smart.py: make use of expand cache in getVar()

[Xu, Dongxiao]

Hi Richard,

> -----Original Message-----
> From: Xu, Dongxiao
> Sent: Monday, August 29, 2011 9:16 PM
> To: Richard Purdie
> Cc: bitbake-devel@lists.openembedded.org
> Subject: RE: [bitbake-devel] [PATCH 1/1] data_smart.py: make use of expand
> cache in getVar()
> 
> > -----Original Message-----
> > From: Richard Purdie [mailto:richard.purdie@linuxfoundation.org]
> > Sent: Monday, August 29, 2011 9:13 PM
> > To: Xu, Dongxiao
> > Cc: bitbake-devel@lists.openembedded.org
> > Subject: RE: [bitbake-devel] [PATCH 1/1] data_smart.py: make use of
> > expand cache in getVar()
> >
> > On Mon, 2011-08-29 at 21:07 +0800, Xu, Dongxiao wrote:
> > > Hi Richard,
> > >
> > > > -----Original Message-----
> > > > From: Richard Purdie [mailto:richard.purdie@linuxfoundation.org]
> > > > Sent: Monday, August 29, 2011 8:55 PM
> > > > To: Xu, Dongxiao
> > > > Cc: bitbake-devel@lists.openembedded.org
> > > > Subject: Re: [bitbake-devel] [PATCH 1/1] data_smart.py: make use
> > > > of expand cache in getVar()
> > > >
> > > > On Mon, 2011-08-29 at 15:33 +0800, Dongxiao Xu wrote:
> > > > > Currently if passing expand=True to getVar() function, it will
> > > > > pass the handling to getVarFlag(), which doesn't get any benefit
> > > > > from the expand cache.
> > > > >
> > > > > Call the expand() function separately in getVar() to make use of
> > > > > the expand cache, which can decrease the parsing time by 40%.
> > > > > (from current 49s to 27s)
> > > > >
> > > > > Signed-off-by: Dongxiao Xu <dongxiao.xu@intel.com>
> > > > > ---
> > > > >  lib/bb/data_smart.py |    7 ++++++-
> > > > >  1 files changed, 6 insertions(+), 1 deletions(-)
> > > > >
> > > > > diff --git a/lib/bb/data_smart.py b/lib/bb/data_smart.py index
> > > > > 301f9e3..d8ba24f 100644
> > > > > --- a/lib/bb/data_smart.py
> > > > > +++ b/lib/bb/data_smart.py
> > > > > @@ -268,7 +268,12 @@ class DataSmart(MutableMapping):
> > > > >          self.dict[var]["content"] = value
> > > > >
> > > > >      def getVar(self, var, expand=False, noweakdefault=False):
> > > > > -        return self.getVarFlag(var, "content", expand, noweakdefault)
> > > > > +        value = self.getVarFlag(var, "content", False,
> > > > > + noweakdefault)
> > > > > +
> > > > > +        # Call expand() separately to make use of the expand cache
> > > > > +        if expand and value:
> > > > > +            return self.expand(value, var)
> > > > > +        return value
> > > > >
> > > >
> > > > Why can't we have getVarFlag use the expand cache?
> > >
> > > For expand cache, the format is a list: expand_cache[varname].
> > > It doesn't contain information of flag values.
> >
> > Good point.
> >
> > I'm left wondering why we don't key that cache on the unexpanded value
> > rather than a specific variable name (or name+flag).
> >
> > I'll take this patch but its something we should consider...
> 
> Thanks, I will step into this point further and see if (name+flag) type of cache
> could benefit us more.

Sorry I must misunderstand your point just now...

For caching of unexpanded value, there are too many/uncertainty of types, like:

${TUNE_ARCH}
${@bb.parse.BBHandler.vars_from_file(bb.data.getVar('FILE',d),d)[0] or 'defaultpkgname'}
${@bb.utils.contains("TUNE_FEATURES", "m32", "${X86ARCH32}", "" ,d)}${@bb.utils.contains("TUNE_FEATURES", "mx32", "${X86ARCH64}", "" ,d)}${@bb.utils.contains("TUNE_FEATURES", "m64", "${X86ARCH64}", "" ,d)}

For simple variables, it is easy to cache, however for the expressions, it may not suitable to cache.

Besides, I also tried the (name+flag) types of cache, but it doesn't bring benefit (from 27s to 28s). Therefore it is suppose that most of the calling should be getVar(). Those specifically getting flag value should be not much.

Thanks,
Dongxiao

> 
> Thanks,
> Dongxiao
> 
> >
> > Cheers,
> >
> > Richard
> >
> >

Re: [PATCH 1/1] data_smart.py: make use of expand cache in getVar()

[Richard Purdie]

On Mon, 2011-08-29 at 22:21 +0800, Xu, Dongxiao wrote:
> > -----Original Message-----
> > From: Xu, Dongxiao
> > Sent: Monday, August 29, 2011 9:16 PM
> > To: Richard Purdie
> > Cc: bitbake-devel@lists.openembedded.org
> > Subject: RE: [bitbake-devel] [PATCH 1/1] data_smart.py: make use of expand
> > cache in getVar()
> > 
> > > -----Original Message-----
> > > From: Richard Purdie [mailto:richard.purdie@linuxfoundation.org]
> > > Sent: Monday, August 29, 2011 9:13 PM
> > > To: Xu, Dongxiao
> > > Cc: bitbake-devel@lists.openembedded.org
> > > Subject: RE: [bitbake-devel] [PATCH 1/1] data_smart.py: make use of
> > > expand cache in getVar()
> > >
> > >
> > > I'm left wondering why we don't key that cache on the unexpanded value
> > > rather than a specific variable name (or name+flag).
> > >
> > > I'll take this patch but its something we should consider...
> > 
> > Thanks, I will step into this point further and see if (name+flag) type of cache
> > could benefit us more.
> 
> Sorry I must misunderstand your point just now...
> 
> For caching of unexpanded value, there are too many/uncertainty of types, like:
> 
> ${TUNE_ARCH}
> ${@bb.parse.BBHandler.vars_from_file(bb.data.getVar('FILE',d),d)[0] or 'defaultpkgname'}
> ${@bb.utils.contains("TUNE_FEATURES", "m32", "${X86ARCH32}", "" ,d)}${@bb.utils.contains("TUNE_FEATURES", "mx32", "${X86ARCH64}", "" ,d)}${@bb.utils.contains("TUNE_FEATURES", "m64", "${X86ARCH64}", "" ,d)}
> 
> For simple variables, it is easy to cache, however for the expressions, it may not suitable to cache.

It shouldn't really matter what the expression is, we should just be
able to cache the value it evaluates to. We invalidate the cache upon
any write to the data store that can potentially change a variables
contents.

This shouldn't really be any different to the current behaviour of the
code (where caching doesn't cause a problem).

Cheers,

Richard