[PATCH 1/3] cooker.py: Save prioritized BBFILES

[PATCH 1/3] cooker.py: Save prioritized BBFILES

[Robert Yang]

The original code saved BBFILES back to BBFILES without changes, I think that
it was for avoiding undefined BBFILES. Now save prioritized BBFILES back which
is more usable for the one which requires it such as bb.utils.get_file_layer().

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
---
 lib/bb/cooker.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/bitbake/lib/bb/cooker.py b/bitbake/lib/bb/cooker.py
index 3a58a3a33..5571e5312 100644
--- a/bitbake/lib/bb/cooker.py
+++ b/bitbake/lib/bb/cooker.py
@@ -1738,10 +1738,10 @@ class CookerCollectFiles(object):
         collectlog.debug(1, "collecting .bb files")
 
         files = (config.getVar( "BBFILES") or "").split()
-        config.setVar("BBFILES", " ".join(files))
 
         # Sort files by priority
         files.sort( key=lambda fileitem: self.calc_bbfile_priority(fileitem)[0] )
+        config.setVar("BBFILES", " ".join(files))
 
         if not len(files):
             files = self.get_bbfiles()
-- 
2.26.2

[PATCH 2/3] utils.py: get_file_layer(): Exit the loop when file is matched

[Robert Yang]

This can make "$ bitbake-layers show-recipes" save about 60% time (14min ->
6min) in my build (more than 3000 recipes)

The command "bitbake-layers show-recipes" calls bb.utils.get_file_layer() with
each recipe, and get_file_layer() compare the file with each item in BBFILES
which makes it very time consuming when there are a lot of recipes and items in
BBFILES. So exit when file is matched, it doesn't make sense to go on the loop
when file is matched.

And use fnmatchcase to replace of fnmatch since the comparison should be
case-sensitive.

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
---
 lib/bb/utils.py | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/bitbake/lib/bb/utils.py b/bitbake/lib/bb/utils.py
index 50032e50c..03568474d 100644
--- a/bitbake/lib/bb/utils.py
+++ b/bitbake/lib/bb/utils.py
@@ -1494,9 +1494,10 @@ def get_file_layer(filename, d):
     bbfiles = (d.getVar('BBFILES') or '').split()
     bbfilesmatch = False
     for bbfilesentry in bbfiles:
-        if fnmatch.fnmatch(filename, bbfilesentry):
+        if fnmatch.fnmatchcase(filename, bbfilesentry):
             bbfilesmatch = True
             result = path_to_layer(bbfilesentry)
+            break
 
     if not bbfilesmatch:
         # Probably a bbclass
-- 
2.26.2

[PATCH 3/3] utils.py: get_file_layer(): Improve performance

[Robert Yang]

The following code costs a lot of time when there are lot of layers and recipes:

     for collection in collections:
         collection_res[collection] = d.getVar('BBFILE_PATTERN_%s' % collection) or ''

My build has more than 100 layers and 3000 recipes, which calls d.getVar() 300K
(3000 * 100) times and makes 'bitbake-layers show-recipes' very slow, add a
keyword argument to get_file_layer() can fix the problem, it can save about 90%
time in my build (6min -> 40s).

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
---
 lib/bb/utils.py       | 12 +++++++++---
 lib/bblayers/query.py | 10 ++++++++--
 2 files changed, 17 insertions(+), 5 deletions(-)

diff --git a/bitbake/lib/bb/utils.py b/bitbake/lib/bb/utils.py
index 03568474d..ca5c38683 100644
--- a/bitbake/lib/bb/utils.py
+++ b/bitbake/lib/bb/utils.py
@@ -1472,14 +1472,20 @@ def edit_bblayers_conf(bblayers_conf, add, remove, edit_cb=None):
 
     return (notadded, notremoved)
 
-
-def get_file_layer(filename, d):
-    """Determine the collection (as defined by a layer's layer.conf file) containing the specified file"""
+def get_collection_res(d):
     collections = (d.getVar('BBFILE_COLLECTIONS') or '').split()
     collection_res = {}
     for collection in collections:
         collection_res[collection] = d.getVar('BBFILE_PATTERN_%s' % collection) or ''
 
+    return collection_res
+
+
+def get_file_layer(filename, d, collection_res={}):
+    """Determine the collection (as defined by a layer's layer.conf file) containing the specified file"""
+    if not collection_res:
+        collection_res = get_collection_res(d)
+
     def path_to_layer(path):
         # Use longest path so we handle nested layers
         matchlen = 0
diff --git a/bitbake/lib/bblayers/query.py b/bitbake/lib/bblayers/query.py
index ee2db0efe..be2d84c35 100644
--- a/bitbake/lib/bblayers/query.py
+++ b/bitbake/lib/bblayers/query.py
@@ -107,6 +107,8 @@ pnspec to match a specified recipe name (supports wildcards). Note that
 skipped recipes will also be listed, with a " (skipped)" suffix.
 """
 
+        self.collection_res = {}
+
         inheritlist = args.inherits.split(',') if args.inherits else []
         if inheritlist or args.pnspec or args.multiple:
             title = 'Matching recipes:'
@@ -222,7 +224,6 @@ skipped recipes will also be listed, with a " (skipped)" suffix.
                             multilayer = True
                         if prov[0] != pref[0]:
                             same_ver = False
-
                     if (multilayer or not show_overlayed_only) and (same_ver or not show_same_ver_only):
                         if not items_listed:
                             logger.plain('=== %s ===' % title)
@@ -243,8 +244,13 @@ skipped recipes will also be listed, with a " (skipped)" suffix.
         else:
             return '?'
 
+    def get_collection_res(self):
+        if not self.collection_res:
+            self.collection_res = bb.utils.get_collection_res(self.tinfoil.config_data)
+        return self.collection_res
+
     def get_file_layerdir(self, filename):
-        layer = bb.utils.get_file_layer(filename, self.tinfoil.config_data)
+        layer = bb.utils.get_file_layer(filename, self.tinfoil.config_data, self.get_collection_res())
         return self.bbfile_collections.get(layer, None)
 
     def remove_layer_prefix(self, f):
-- 
2.26.2

Re: [bitbake-devel] [PATCH 1/3] cooker.py: Save prioritized BBFILES

[Richard Purdie]

On Thu, 2020-07-30 at 02:23 -0700, Robert Yang wrote:
> The original code saved BBFILES back to BBFILES without changes, I think that
> it was for avoiding undefined BBFILES. Now save prioritized BBFILES back which
> is more usable for the one which requires it such as bb.utils.get_file_layer().
> 
> Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
> ---
>  lib/bb/cooker.py | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/bitbake/lib/bb/cooker.py b/bitbake/lib/bb/cooker.py
> index 3a58a3a33..5571e5312 100644
> --- a/bitbake/lib/bb/cooker.py
> +++ b/bitbake/lib/bb/cooker.py
> @@ -1738,10 +1738,10 @@ class CookerCollectFiles(object):
>          collectlog.debug(1, "collecting .bb files")
>  
>          files = (config.getVar( "BBFILES") or "").split()
> -        config.setVar("BBFILES", " ".join(files))

I traced that back to 2007 but didn't go further. I don't think we
should be writing this value back at all. Its going to confuse anyone
trying to understand the history of the data in the variable.

>          # Sort files by priority
>          files.sort( key=lambda fileitem: self.calc_bbfile_priority(fileitem)[0] )
> +        config.setVar("BBFILES", " ".join(files))

If we want to make this information available to the wider system, we
should perhaps have bitbake API for querying these kinds of things?

Cheers,

Richard

Re: [bitbake-devel] [PATCH 1/3] cooker.py: Save prioritized BBFILES

[Robert Yang]

On 7/30/20 6:37 PM, Richard Purdie wrote:
> On Thu, 2020-07-30 at 02:23 -0700, Robert Yang wrote:
>> The original code saved BBFILES back to BBFILES without changes, I think that
>> it was for avoiding undefined BBFILES. Now save prioritized BBFILES back which
>> is more usable for the one which requires it such as bb.utils.get_file_layer().
>>
>> Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
>> ---
>>   lib/bb/cooker.py | 2 +-
>>   1 file changed, 1 insertion(+), 1 deletion(-)
>>
>> diff --git a/bitbake/lib/bb/cooker.py b/bitbake/lib/bb/cooker.py
>> index 3a58a3a33..5571e5312 100644
>> --- a/bitbake/lib/bb/cooker.py
>> +++ b/bitbake/lib/bb/cooker.py
>> @@ -1738,10 +1738,10 @@ class CookerCollectFiles(object):
>>           collectlog.debug(1, "collecting .bb files")
>>   
>>           files = (config.getVar( "BBFILES") or "").split()
>> -        config.setVar("BBFILES", " ".join(files))
> 
> I traced that back to 2007 but didn't go further. I don't think we
> should be writing this value back at all. Its going to confuse anyone
> trying to understand the history of the data in the variable.
> 
>>           # Sort files by priority
>>           files.sort( key=lambda fileitem: self.calc_bbfile_priority(fileitem)[0] )
>> +        config.setVar("BBFILES", " ".join(files))
> 
> If we want to make this information available to the wider system, we
> should perhaps have bitbake API for querying these kinds of things?

This prioritized BBFILES can improve the performance, for example, if
BBFILES has no priority, the "bitbake-layers show-recipes"s' time would
increase from 40s to 110s in my testing.

Did you mean something like bb.utils.get_prioritized_bbfiles(), please?

// Robert


> 
> Cheers,
> 
> Richard
> 

Re: [bitbake-devel] [PATCH 2/3] utils.py: get_file_layer(): Exit the loop when file is matched

[Robert Yang]

It's strange that the cover letter was lost. I used "git send-email 000*" to
send all of the patches including cover letter, but I didn't find it in
the mailing list. So I sent it here:

Subject: [PATCH 0/3] Fix performance issues for bitbake-layers show-recipes

The ruunning time of "bitbake-layers show-recipes" increased from 40s to 14min
after the following patch:

tinfoil: Simplify remote datastore connections

The patch 2 and 3 retores the time from 14min to 40s, which saves about 95% time.

# Updated: Patch 1 also improves the performance.

// Robert

The following changes since commit 20e9df57217c5f37817653d2c3d492f2d4d37623:

   lib/oe/reproducible.py: Fix git HEAD check (2020-07-29 11:37:31 +0100)

are available in the Git repository at:

   git://git.yoctoproject.org/poky-contrib rbt/bblayers
   http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=rbt/bblayers

Robert Yang (3):
   cooker.py: Save prioritized BBFILES
   utils.py: get_file_layer(): Exit the loop when file is matched
   utils.py: get_file_layer(): Improve performance

  bitbake/lib/bb/cooker.py      |  2 +-
  bitbake/lib/bb/utils.py       | 15 +++++++++++----
  bitbake/lib/bblayers/query.py | 10 ++++++++--
  3 files changed, 20 insertions(+), 7 deletions(-)

// Robert

On 7/30/20 5:23 PM, Robert Yang wrote:
> This can make "$ bitbake-layers show-recipes" save about 60% time (14min ->
> 6min) in my build (more than 3000 recipes)
> 
> The command "bitbake-layers show-recipes" calls bb.utils.get_file_layer() with
> each recipe, and get_file_layer() compare the file with each item in BBFILES
> which makes it very time consuming when there are a lot of recipes and items in
> BBFILES. So exit when file is matched, it doesn't make sense to go on the loop
> when file is matched.
> 
> And use fnmatchcase to replace of fnmatch since the comparison should be
> case-sensitive.
> 
> Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
> ---
>   lib/bb/utils.py | 3 ++-
>   1 file changed, 2 insertions(+), 1 deletion(-)
> 
> diff --git a/bitbake/lib/bb/utils.py b/bitbake/lib/bb/utils.py
> index 50032e50c..03568474d 100644
> --- a/bitbake/lib/bb/utils.py
> +++ b/bitbake/lib/bb/utils.py
> @@ -1494,9 +1494,10 @@ def get_file_layer(filename, d):
>       bbfiles = (d.getVar('BBFILES') or '').split()
>       bbfilesmatch = False
>       for bbfilesentry in bbfiles:
> -        if fnmatch.fnmatch(filename, bbfilesentry):
> +        if fnmatch.fnmatchcase(filename, bbfilesentry):
>               bbfilesmatch = True
>               result = path_to_layer(bbfilesentry)
> +            break
>   
>       if not bbfilesmatch:
>           # Probably a bbclass
> 
> 
> 
>