* [PATCH] lib/bb: Add BB_SIGNATURE_LOCAL_DIRS_EXCLUDE to speed-up taskhash on directories
@ 2019-11-13 23:23 Aníbal Limón
2019-11-13 23:27 ` Richard Purdie
2019-11-14 2:54 ` Khem Raj
0 siblings, 2 replies; 7+ messages in thread
From: Aníbal Limón @ 2019-11-13 23:23 UTC (permalink / raw)
To: bitbake-devel
The new BB_SIGNATURE_LOCAL_DIRS_EXCLUDE allows you to specify a list
of directories to exclude when making taskhash, our specific case
is using SRC_URI that points local VCS directory.
In OE-Core this variable could be set by default to:
BB_SIGNATURE_LOCAL_DIRS_EXCLUDE = ".cvs .git .svn"
Signed-off-by: Aníbal Limón <anibal.limon@linaro.org>
---
lib/bb/checksum.py | 5 +++--
lib/bb/fetch2/__init__.py | 4 ++--
lib/bb/siggen.py | 5 +++--
3 files changed, 8 insertions(+), 6 deletions(-)
diff --git a/lib/bb/checksum.py b/lib/bb/checksum.py
index 5bc8a8fc..677020f4 100644
--- a/lib/bb/checksum.py
+++ b/lib/bb/checksum.py
@@ -74,7 +74,7 @@ class FileChecksumCache(MultiProcessCache):
else:
dest[0][h] = source[0][h]
- def get_checksums(self, filelist, pn):
+ def get_checksums(self, filelist, pn, localdirsexclude):
"""Get checksums for a list of files"""
def checksum_file(f):
@@ -90,7 +90,8 @@ class FileChecksumCache(MultiProcessCache):
if pth == "/":
bb.fatal("Refusing to checksum /")
dirchecksums = []
- for root, dirs, files in os.walk(pth):
+ for root, dirs, files in os.walk(pth, topdown=True):
+ [dirs.remove(d) for d in list(dirs) if d in localdirsexclude]
for name in files:
fullpth = os.path.join(root, name)
checksum = checksum_file(fullpth)
diff --git a/lib/bb/fetch2/__init__.py b/lib/bb/fetch2/__init__.py
index 07de6c26..731c1608 100644
--- a/lib/bb/fetch2/__init__.py
+++ b/lib/bb/fetch2/__init__.py
@@ -1197,14 +1197,14 @@ def get_checksum_file_list(d):
return " ".join(filelist)
-def get_file_checksums(filelist, pn):
+def get_file_checksums(filelist, pn, localdirsexclude):
"""Get a list of the checksums for a list of local files
Returns the checksums for a list of local files, caching the results as
it proceeds
"""
- return _checksum_cache.get_checksums(filelist, pn)
+ return _checksum_cache.get_checksums(filelist, pn, localdirsexclude)
class FetchData(object):
diff --git a/lib/bb/siggen.py b/lib/bb/siggen.py
index a4bb1ff7..f110fd72 100644
--- a/lib/bb/siggen.py
+++ b/lib/bb/siggen.py
@@ -123,6 +123,7 @@ class SignatureGeneratorBasic(SignatureGenerator):
self.unihash_cache = bb.cache.SimpleCache("1")
self.unitaskhashes = self.unihash_cache.init_cache(data, "bb_unihashes.dat", {})
+ self.localdirsexclude = (data.getVar("BB_SIGNATURE_LOCAL_DIRS_EXCLUDE") or "").split()
def init_rundepcheck(self, data):
self.taskwhitelist = data.getVar("BB_HASHTASK_WHITELIST") or None
@@ -221,9 +222,9 @@ class SignatureGeneratorBasic(SignatureGenerator):
if task in dataCache.file_checksums[fn]:
if self.checksum_cache:
- checksums = self.checksum_cache.get_checksums(dataCache.file_checksums[fn][task], recipename)
+ checksums = self.checksum_cache.get_checksums(dataCache.file_checksums[fn][task], recipename, self.localdirsexclude)
else:
- checksums = bb.fetch2.get_file_checksums(dataCache.file_checksums[fn][task], recipename)
+ checksums = bb.fetch2.get_file_checksums(dataCache.file_checksums[fn][task], recipename, self.localdirsexclude)
for (f,cs) in checksums:
self.file_checksum_values[tid].append((f,cs))
if cs:
--
2.24.0
^ permalink raw reply related [flat|nested] 7+ messages in thread
* Re: [PATCH] lib/bb: Add BB_SIGNATURE_LOCAL_DIRS_EXCLUDE to speed-up taskhash on directories
2019-11-13 23:23 [PATCH] lib/bb: Add BB_SIGNATURE_LOCAL_DIRS_EXCLUDE to speed-up taskhash on directories Aníbal Limón
@ 2019-11-13 23:27 ` Richard Purdie
2019-11-14 2:54 ` Khem Raj
1 sibling, 0 replies; 7+ messages in thread
From: Richard Purdie @ 2019-11-13 23:27 UTC (permalink / raw)
To: Aníbal Limón, bitbake-devel
On Wed, 2019-11-13 at 17:23 -0600, Aníbal Limón wrote:
> The new BB_SIGNATURE_LOCAL_DIRS_EXCLUDE allows you to specify a list
> of directories to exclude when making taskhash, our specific case
> is using SRC_URI that points local VCS directory.
>
> In OE-Core this variable could be set by default to:
>
> BB_SIGNATURE_LOCAL_DIRS_EXCLUDE = ".cvs .git .svn"
I'm wondering if we should be adventurous and make this the default
value? :)
Cheers,
Richard
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [PATCH] lib/bb: Add BB_SIGNATURE_LOCAL_DIRS_EXCLUDE to speed-up taskhash on directories
2019-11-13 23:23 [PATCH] lib/bb: Add BB_SIGNATURE_LOCAL_DIRS_EXCLUDE to speed-up taskhash on directories Aníbal Limón
2019-11-13 23:27 ` Richard Purdie
@ 2019-11-14 2:54 ` Khem Raj
2019-11-14 16:54 ` Peter Kjellerstedt
1 sibling, 1 reply; 7+ messages in thread
From: Khem Raj @ 2019-11-14 2:54 UTC (permalink / raw)
To: Aníbal Limón; +Cc: bitbake-devel
On Wed, Nov 13, 2019 at 3:23 PM Aníbal Limón <anibal.limon@linaro.org> wrote:
>
> The new BB_SIGNATURE_LOCAL_DIRS_EXCLUDE allows you to specify a list
> of directories to exclude when making taskhash, our specific case
> is using SRC_URI that points local VCS directory.
>
> In OE-Core this variable could be set by default to:
>
> BB_SIGNATURE_LOCAL_DIRS_EXCLUDE = ".cvs .git .svn"
>
I like the approach. perhaps a sane default would be nice.
> Signed-off-by: Aníbal Limón <anibal.limon@linaro.org>
> ---
> lib/bb/checksum.py | 5 +++--
> lib/bb/fetch2/__init__.py | 4 ++--
> lib/bb/siggen.py | 5 +++--
> 3 files changed, 8 insertions(+), 6 deletions(-)
>
> diff --git a/lib/bb/checksum.py b/lib/bb/checksum.py
> index 5bc8a8fc..677020f4 100644
> --- a/lib/bb/checksum.py
> +++ b/lib/bb/checksum.py
> @@ -74,7 +74,7 @@ class FileChecksumCache(MultiProcessCache):
> else:
> dest[0][h] = source[0][h]
>
> - def get_checksums(self, filelist, pn):
> + def get_checksums(self, filelist, pn, localdirsexclude):
> """Get checksums for a list of files"""
>
> def checksum_file(f):
> @@ -90,7 +90,8 @@ class FileChecksumCache(MultiProcessCache):
> if pth == "/":
> bb.fatal("Refusing to checksum /")
> dirchecksums = []
> - for root, dirs, files in os.walk(pth):
> + for root, dirs, files in os.walk(pth, topdown=True):
> + [dirs.remove(d) for d in list(dirs) if d in localdirsexclude]
> for name in files:
> fullpth = os.path.join(root, name)
> checksum = checksum_file(fullpth)
> diff --git a/lib/bb/fetch2/__init__.py b/lib/bb/fetch2/__init__.py
> index 07de6c26..731c1608 100644
> --- a/lib/bb/fetch2/__init__.py
> +++ b/lib/bb/fetch2/__init__.py
> @@ -1197,14 +1197,14 @@ def get_checksum_file_list(d):
>
> return " ".join(filelist)
>
> -def get_file_checksums(filelist, pn):
> +def get_file_checksums(filelist, pn, localdirsexclude):
> """Get a list of the checksums for a list of local files
>
> Returns the checksums for a list of local files, caching the results as
> it proceeds
>
> """
> - return _checksum_cache.get_checksums(filelist, pn)
> + return _checksum_cache.get_checksums(filelist, pn, localdirsexclude)
>
>
> class FetchData(object):
> diff --git a/lib/bb/siggen.py b/lib/bb/siggen.py
> index a4bb1ff7..f110fd72 100644
> --- a/lib/bb/siggen.py
> +++ b/lib/bb/siggen.py
> @@ -123,6 +123,7 @@ class SignatureGeneratorBasic(SignatureGenerator):
>
> self.unihash_cache = bb.cache.SimpleCache("1")
> self.unitaskhashes = self.unihash_cache.init_cache(data, "bb_unihashes.dat", {})
> + self.localdirsexclude = (data.getVar("BB_SIGNATURE_LOCAL_DIRS_EXCLUDE") or "").split()
>
> def init_rundepcheck(self, data):
> self.taskwhitelist = data.getVar("BB_HASHTASK_WHITELIST") or None
> @@ -221,9 +222,9 @@ class SignatureGeneratorBasic(SignatureGenerator):
>
> if task in dataCache.file_checksums[fn]:
> if self.checksum_cache:
> - checksums = self.checksum_cache.get_checksums(dataCache.file_checksums[fn][task], recipename)
> + checksums = self.checksum_cache.get_checksums(dataCache.file_checksums[fn][task], recipename, self.localdirsexclude)
> else:
> - checksums = bb.fetch2.get_file_checksums(dataCache.file_checksums[fn][task], recipename)
> + checksums = bb.fetch2.get_file_checksums(dataCache.file_checksums[fn][task], recipename, self.localdirsexclude)
> for (f,cs) in checksums:
> self.file_checksum_values[tid].append((f,cs))
> if cs:
> --
> 2.24.0
>
> --
> _______________________________________________
> bitbake-devel mailing list
> bitbake-devel@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/bitbake-devel
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [PATCH] lib/bb: Add BB_SIGNATURE_LOCAL_DIRS_EXCLUDE to speed-up taskhash on directories
2019-11-14 2:54 ` Khem Raj
@ 2019-11-14 16:54 ` Peter Kjellerstedt
2019-11-14 18:07 ` Anibal Limon
0 siblings, 1 reply; 7+ messages in thread
From: Peter Kjellerstedt @ 2019-11-14 16:54 UTC (permalink / raw)
To: Khem Raj, Aníbal Limón; +Cc: bitbake-devel
> -----Original Message-----
> From: bitbake-devel-bounces@lists.openembedded.org <bitbake-devel-
> bounces@lists.openembedded.org> On Behalf Of Khem Raj
> Sent: den 14 november 2019 03:55
> To: Aníbal Limón <anibal.limon@linaro.org>
> Cc: bitbake-devel <bitbake-devel@lists.openembedded.org>
> Subject: Re: [bitbake-devel] [PATCH] lib/bb: Add
> BB_SIGNATURE_LOCAL_DIRS_EXCLUDE to speed-up taskhash on directories
>
> On Wed, Nov 13, 2019 at 3:23 PM Aníbal Limón <anibal.limon@linaro.org>
> wrote:
> >
> > The new BB_SIGNATURE_LOCAL_DIRS_EXCLUDE allows you to specify a list
> > of directories to exclude when making taskhash, our specific case
> > is using SRC_URI that points local VCS directory.
> >
> > In OE-Core this variable could be set by default to:
> >
> > BB_SIGNATURE_LOCAL_DIRS_EXCLUDE = ".cvs .git .svn"
That should be:
BB_SIGNATURE_LOCAL_DIRS_EXCLUDE = "CVS .git .svn"
> I like the approach. perhaps a sane default would be nice.
//Peter
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [PATCH] lib/bb: Add BB_SIGNATURE_LOCAL_DIRS_EXCLUDE to speed-up taskhash on directories
2019-11-14 16:54 ` Peter Kjellerstedt
@ 2019-11-14 18:07 ` Anibal Limon
2019-11-14 18:08 ` Anibal Limon
0 siblings, 1 reply; 7+ messages in thread
From: Anibal Limon @ 2019-11-14 18:07 UTC (permalink / raw)
To: Peter Kjellerstedt; +Cc: bitbake-devel
[-- Attachment #1: Type: text/plain, Size: 1327 bytes --]
On Thu, 14 Nov 2019 at 10:54, Peter Kjellerstedt <
peter.kjellerstedt@axis.com> wrote:
> > -----Original Message-----
> > From: bitbake-devel-bounces@lists.openembedded.org <bitbake-devel-
> > bounces@lists.openembedded.org> On Behalf Of Khem Raj
> > Sent: den 14 november 2019 03:55
> > To: Aníbal Limón <anibal.limon@linaro.org>
> > Cc: bitbake-devel <bitbake-devel@lists.openembedded.org>
> > Subject: Re: [bitbake-devel] [PATCH] lib/bb: Add
> > BB_SIGNATURE_LOCAL_DIRS_EXCLUDE to speed-up taskhash on directories
> >
> > On Wed, Nov 13, 2019 at 3:23 PM Aníbal Limón <anibal.limon@linaro.org>
> > wrote:
> > >
> > > The new BB_SIGNATURE_LOCAL_DIRS_EXCLUDE allows you to specify a list
> > > of directories to exclude when making taskhash, our specific case
> > > is using SRC_URI that points local VCS directory.
> > >
> > > In OE-Core this variable could be set by default to:
> > >
> > > BB_SIGNATURE_LOCAL_DIRS_EXCLUDE = ".cvs .git .svn"
>
> That should be:
>
> BB_SIGNATURE_LOCAL_DIRS_EXCLUDE = "CVS .git .svn"
>
So I will change to,
+ self.localdirsexclude = (data.getVar("BB_SIGNATURE_LOCAL_DIRS_EXCLUDE")
or "CVS .git .svn").split()
Any other directory to add it?
Cheers,
Anibal
>
> > I like the approach. perhaps a sane default would be nice.
>
> //Peter
>
>
[-- Attachment #2: Type: text/html, Size: 2427 bytes --]
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [PATCH] lib/bb: Add BB_SIGNATURE_LOCAL_DIRS_EXCLUDE to speed-up taskhash on directories
2019-11-14 18:07 ` Anibal Limon
@ 2019-11-14 18:08 ` Anibal Limon
2019-11-25 22:22 ` Anibal Limon
0 siblings, 1 reply; 7+ messages in thread
From: Anibal Limon @ 2019-11-14 18:08 UTC (permalink / raw)
To: Peter Kjellerstedt; +Cc: bitbake-devel
[-- Attachment #1: Type: text/plain, Size: 1557 bytes --]
On Thu, 14 Nov 2019 at 12:07, Anibal Limon <anibal.limon@linaro.org> wrote:
>
> On Thu, 14 Nov 2019 at 10:54, Peter Kjellerstedt <
> peter.kjellerstedt@axis.com> wrote:
>
>> > -----Original Message-----
>> > From: bitbake-devel-bounces@lists.openembedded.org <bitbake-devel-
>> > bounces@lists.openembedded.org> On Behalf Of Khem Raj
>> > Sent: den 14 november 2019 03:55
>> > To: Aníbal Limón <anibal.limon@linaro.org>
>> > Cc: bitbake-devel <bitbake-devel@lists.openembedded.org>
>> > Subject: Re: [bitbake-devel] [PATCH] lib/bb: Add
>> > BB_SIGNATURE_LOCAL_DIRS_EXCLUDE to speed-up taskhash on directories
>> >
>> > On Wed, Nov 13, 2019 at 3:23 PM Aníbal Limón <anibal.limon@linaro.org>
>> > wrote:
>> > >
>> > > The new BB_SIGNATURE_LOCAL_DIRS_EXCLUDE allows you to specify a list
>> > > of directories to exclude when making taskhash, our specific case
>> > > is using SRC_URI that points local VCS directory.
>> > >
>> > > In OE-Core this variable could be set by default to:
>> > >
>> > > BB_SIGNATURE_LOCAL_DIRS_EXCLUDE = ".cvs .git .svn"
>>
>> That should be:
>>
>> BB_SIGNATURE_LOCAL_DIRS_EXCLUDE = "CVS .git .svn"
>>
>
> So I will change to,
>
> + self.localdirsexclude = (data.getVar("BB_SIGNATURE_LOCAL_DIRS_EXCLUDE")
> or "CVS .git .svn").split()
>
> Any other directory to add it?
>
Replying myself...
What about add all VCS directories that we support in fetcher.
>
> Cheers,
> Anibal
>
>
>>
>> > I like the approach. perhaps a sane default would be nice.
>>
>> //Peter
>>
>>
[-- Attachment #2: Type: text/html, Size: 3161 bytes --]
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [PATCH] lib/bb: Add BB_SIGNATURE_LOCAL_DIRS_EXCLUDE to speed-up taskhash on directories
2019-11-14 18:08 ` Anibal Limon
@ 2019-11-25 22:22 ` Anibal Limon
0 siblings, 0 replies; 7+ messages in thread
From: Anibal Limon @ 2019-11-25 22:22 UTC (permalink / raw)
To: bitbake-devel
[-- Attachment #1: Type: text/plain, Size: 1942 bytes --]
Regarding some comments in PATCHv2 where default value was set [1], I agree
that this patch will be better and then set default value
in oe-core.
[1]
http://lists.openembedded.org/pipermail/bitbake-devel/2019-November/020519.html
On Thu, 14 Nov 2019 at 12:08, Anibal Limon <anibal.limon@linaro.org> wrote:
>
>
> On Thu, 14 Nov 2019 at 12:07, Anibal Limon <anibal.limon@linaro.org>
> wrote:
>
>>
>> On Thu, 14 Nov 2019 at 10:54, Peter Kjellerstedt <
>> peter.kjellerstedt@axis.com> wrote:
>>
>>> > -----Original Message-----
>>> > From: bitbake-devel-bounces@lists.openembedded.org <bitbake-devel-
>>> > bounces@lists.openembedded.org> On Behalf Of Khem Raj
>>> > Sent: den 14 november 2019 03:55
>>> > To: Aníbal Limón <anibal.limon@linaro.org>
>>> > Cc: bitbake-devel <bitbake-devel@lists.openembedded.org>
>>> > Subject: Re: [bitbake-devel] [PATCH] lib/bb: Add
>>> > BB_SIGNATURE_LOCAL_DIRS_EXCLUDE to speed-up taskhash on directories
>>> >
>>> > On Wed, Nov 13, 2019 at 3:23 PM Aníbal Limón <anibal.limon@linaro.org>
>>> > wrote:
>>> > >
>>> > > The new BB_SIGNATURE_LOCAL_DIRS_EXCLUDE allows you to specify a list
>>> > > of directories to exclude when making taskhash, our specific case
>>> > > is using SRC_URI that points local VCS directory.
>>> > >
>>> > > In OE-Core this variable could be set by default to:
>>> > >
>>> > > BB_SIGNATURE_LOCAL_DIRS_EXCLUDE = ".cvs .git .svn"
>>>
>>> That should be:
>>>
>>> BB_SIGNATURE_LOCAL_DIRS_EXCLUDE = "CVS .git .svn"
>>>
>>
>> So I will change to,
>>
>> + self.localdirsexclude = (data.getVar("BB_SIGNATURE_LOCAL_DIRS_EXCLUDE")
>> or "CVS .git .svn").split()
>>
>> Any other directory to add it?
>>
>
> Replying myself...
>
> What about add all VCS directories that we support in fetcher.
>
>
>>
>> Cheers,
>> Anibal
>>
>>
>>>
>>> > I like the approach. perhaps a sane default would be nice.
>>>
>>> //Peter
>>>
>>>
[-- Attachment #2: Type: text/html, Size: 3909 bytes --]
^ permalink raw reply [flat|nested] 7+ messages in thread
end of thread, other threads:[~2019-11-25 22:22 UTC | newest]
Thread overview: 7+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2019-11-13 23:23 [PATCH] lib/bb: Add BB_SIGNATURE_LOCAL_DIRS_EXCLUDE to speed-up taskhash on directories Aníbal Limón
2019-11-13 23:27 ` Richard Purdie
2019-11-14 2:54 ` Khem Raj
2019-11-14 16:54 ` Peter Kjellerstedt
2019-11-14 18:07 ` Anibal Limon
2019-11-14 18:08 ` Anibal Limon
2019-11-25 22:22 ` Anibal Limon
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.