* [PATCH] package.bbclass: allow using EXCLUDE_FROM_SHLIBS for subpackages
@ 2016-10-10 17:02 Andrii Bordunov
2016-10-19 13:58 ` Andrii Bordunov
2016-11-16 10:47 ` Richard Purdie
0 siblings, 2 replies; 10+ messages in thread
From: Andrii Bordunov @ 2016-10-10 17:02 UTC (permalink / raw)
To: openembedded-core; +Cc: Andrii Bordunov
Some packages containing shared libraries might be registered
as shlib providers when they shouldn't (for example, the lib is for
their private use and must not generate any dependency).
EXCLUDE_FROM_SHLIBS is targeted at that, but it could be set
for entire recipe only.
This patch expands EXCLUDE_FROM_SHLIBS usage, so now it's possible
to set it in a style similar with RDEPENDS. For example:
EXCLUDE_FROM_SHLIBS_${PN}-ptest = "1"
Signed-off-by: Andrii Bordunov <aborduno@cisco.com>
---
meta/classes/package.bbclass | 12 ++++++++++--
1 file changed, 10 insertions(+), 2 deletions(-)
diff --git a/meta/classes/package.bbclass b/meta/classes/package.bbclass
index a6f0a7a..9bf43dc 100644
--- a/meta/classes/package.bbclass
+++ b/meta/classes/package.bbclass
@@ -1499,6 +1499,14 @@ python package_do_shlibs() {
libdir_re = re.compile(".*/%s$" % d.getVar('baselib', True))
packages = d.getVar('PACKAGES', True)
+
+ shlib_pkgs = []
+ for pkg in packages.split():
+ if d.getVar('EXCLUDE_FROM_SHLIBS_' + pkg, 0):
+ bb.note("not generating shlibs for %s" % pkg)
+ else:
+ shlib_pkgs.append(pkg)
+
targetos = d.getVar('TARGET_OS', True)
workdir = d.getVar('WORKDIR', True)
@@ -1614,7 +1622,7 @@ python package_do_shlibs() {
needed = {}
shlib_provider = oe.package.read_shlib_providers(d)
- for pkg in packages.split():
+ for pkg in shlib_pkgs:
private_libs = d.getVar('PRIVATE_LIBS_' + pkg, True) or d.getVar('PRIVATE_LIBS', True) or ""
private_libs = private_libs.split()
needs_ldconfig = False
@@ -1684,7 +1692,7 @@ python package_do_shlibs() {
libsearchpath = [d.getVar('libdir', True), d.getVar('base_libdir', True)]
- for pkg in packages.split():
+ for pkg in shlib_pkgs:
bb.debug(2, "calculating shlib requirements for %s" % pkg)
deps = list()
--
2.7.4
^ permalink raw reply related [flat|nested] 10+ messages in thread
* Re: [PATCH] package.bbclass: allow using EXCLUDE_FROM_SHLIBS for subpackages
2016-10-10 17:02 [PATCH] package.bbclass: allow using EXCLUDE_FROM_SHLIBS for subpackages Andrii Bordunov
@ 2016-10-19 13:58 ` Andrii Bordunov
2016-11-15 15:05 ` Andrii Bordunov
2016-11-16 10:47 ` Richard Purdie
1 sibling, 1 reply; 10+ messages in thread
From: Andrii Bordunov @ 2016-10-19 13:58 UTC (permalink / raw)
To: openembedded-core; +Cc: Andrii Bordunov
Ping? Any comments?
Thank you,
Andrii
On 10.10.16 20:02, Andrii Bordunov wrote:
> Some packages containing shared libraries might be registered
> as shlib providers when they shouldn't (for example, the lib is for
> their private use and must not generate any dependency).
>
> EXCLUDE_FROM_SHLIBS is targeted at that, but it could be set
> for entire recipe only.
>
> This patch expands EXCLUDE_FROM_SHLIBS usage, so now it's possible
> to set it in a style similar with RDEPENDS. For example:
> EXCLUDE_FROM_SHLIBS_${PN}-ptest = "1"
>
> Signed-off-by: Andrii Bordunov <aborduno@cisco.com>
> ---
> meta/classes/package.bbclass | 12 ++++++++++--
> 1 file changed, 10 insertions(+), 2 deletions(-)
>
> diff --git a/meta/classes/package.bbclass b/meta/classes/package.bbclass
> index a6f0a7a..9bf43dc 100644
> --- a/meta/classes/package.bbclass
> +++ b/meta/classes/package.bbclass
> @@ -1499,6 +1499,14 @@ python package_do_shlibs() {
> libdir_re = re.compile(".*/%s$" % d.getVar('baselib', True))
>
> packages = d.getVar('PACKAGES', True)
> +
> + shlib_pkgs = []
> + for pkg in packages.split():
> + if d.getVar('EXCLUDE_FROM_SHLIBS_' + pkg, 0):
> + bb.note("not generating shlibs for %s" % pkg)
> + else:
> + shlib_pkgs.append(pkg)
> +
> targetos = d.getVar('TARGET_OS', True)
>
> workdir = d.getVar('WORKDIR', True)
> @@ -1614,7 +1622,7 @@ python package_do_shlibs() {
> needed = {}
> shlib_provider = oe.package.read_shlib_providers(d)
>
> - for pkg in packages.split():
> + for pkg in shlib_pkgs:
> private_libs = d.getVar('PRIVATE_LIBS_' + pkg, True) or d.getVar('PRIVATE_LIBS', True) or ""
> private_libs = private_libs.split()
> needs_ldconfig = False
> @@ -1684,7 +1692,7 @@ python package_do_shlibs() {
>
> libsearchpath = [d.getVar('libdir', True), d.getVar('base_libdir', True)]
>
> - for pkg in packages.split():
> + for pkg in shlib_pkgs:
> bb.debug(2, "calculating shlib requirements for %s" % pkg)
>
> deps = list()
>
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [PATCH] package.bbclass: allow using EXCLUDE_FROM_SHLIBS for subpackages
2016-10-19 13:58 ` Andrii Bordunov
@ 2016-11-15 15:05 ` Andrii Bordunov
2016-11-16 10:29 ` Burton, Ross
0 siblings, 1 reply; 10+ messages in thread
From: Andrii Bordunov @ 2016-11-15 15:05 UTC (permalink / raw)
To: openembedded-core
Ping-2. Guys? Anything?
Thank you,
Andrii
On 19.10.16 16:58, Andrii Bordunov wrote:
> Ping? Any comments?
>
>
>
> Thank you,
> Andrii
>
> On 10.10.16 20:02, Andrii Bordunov wrote:
>> Some packages containing shared libraries might be registered
>> as shlib providers when they shouldn't (for example, the lib is for
>> their private use and must not generate any dependency).
>>
>> EXCLUDE_FROM_SHLIBS is targeted at that, but it could be set
>> for entire recipe only.
>>
>> This patch expands EXCLUDE_FROM_SHLIBS usage, so now it's possible
>> to set it in a style similar with RDEPENDS. For example:
>> EXCLUDE_FROM_SHLIBS_${PN}-ptest = "1"
>>
>> Signed-off-by: Andrii Bordunov <aborduno@cisco.com>
>> ---
>> meta/classes/package.bbclass | 12 ++++++++++--
>> 1 file changed, 10 insertions(+), 2 deletions(-)
>>
>> diff --git a/meta/classes/package.bbclass b/meta/classes/package.bbclass
>> index a6f0a7a..9bf43dc 100644
>> --- a/meta/classes/package.bbclass
>> +++ b/meta/classes/package.bbclass
>> @@ -1499,6 +1499,14 @@ python package_do_shlibs() {
>> libdir_re = re.compile(".*/%s$" % d.getVar('baselib', True))
>>
>> packages = d.getVar('PACKAGES', True)
>> +
>> + shlib_pkgs = []
>> + for pkg in packages.split():
>> + if d.getVar('EXCLUDE_FROM_SHLIBS_' + pkg, 0):
>> + bb.note("not generating shlibs for %s" % pkg)
>> + else:
>> + shlib_pkgs.append(pkg)
>> +
>> targetos = d.getVar('TARGET_OS', True)
>>
>> workdir = d.getVar('WORKDIR', True)
>> @@ -1614,7 +1622,7 @@ python package_do_shlibs() {
>> needed = {}
>> shlib_provider = oe.package.read_shlib_providers(d)
>>
>> - for pkg in packages.split():
>> + for pkg in shlib_pkgs:
>> private_libs = d.getVar('PRIVATE_LIBS_' + pkg, True) or
>> d.getVar('PRIVATE_LIBS', True) or ""
>> private_libs = private_libs.split()
>> needs_ldconfig = False
>> @@ -1684,7 +1692,7 @@ python package_do_shlibs() {
>>
>> libsearchpath = [d.getVar('libdir', True),
>> d.getVar('base_libdir', True)]
>>
>> - for pkg in packages.split():
>> + for pkg in shlib_pkgs:
>> bb.debug(2, "calculating shlib requirements for %s" % pkg)
>>
>> deps = list()
>>
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [PATCH] package.bbclass: allow using EXCLUDE_FROM_SHLIBS for subpackages
2016-11-15 15:05 ` Andrii Bordunov
@ 2016-11-16 10:29 ` Burton, Ross
0 siblings, 0 replies; 10+ messages in thread
From: Burton, Ross @ 2016-11-16 10:29 UTC (permalink / raw)
To: Andrii Bordunov; +Cc: OE-core
[-- Attachment #1: Type: text/plain, Size: 242 bytes --]
On 15 November 2016 at 15:05, Andrii Bordunov <aborduno@cisco.com> wrote:
> Ping-2. Guys? Anything?
>
Sorry, missed this. It looks reasonable and doesn't introduce any changes
to existing packaging, so it's in my queue now.
Ross
[-- Attachment #2: Type: text/html, Size: 655 bytes --]
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [PATCH] package.bbclass: allow using EXCLUDE_FROM_SHLIBS for subpackages
2016-10-10 17:02 [PATCH] package.bbclass: allow using EXCLUDE_FROM_SHLIBS for subpackages Andrii Bordunov
2016-10-19 13:58 ` Andrii Bordunov
@ 2016-11-16 10:47 ` Richard Purdie
2017-03-16 15:07 ` Andrii Bordunov
1 sibling, 1 reply; 10+ messages in thread
From: Richard Purdie @ 2016-11-16 10:47 UTC (permalink / raw)
To: Andrii Bordunov, openembedded-core
On Mon, 2016-10-10 at 20:02 +0300, Andrii Bordunov wrote:
> Some packages containing shared libraries might be registered
> as shlib providers when they shouldn't (for example, the lib is for
> their private use and must not generate any dependency).
>
> EXCLUDE_FROM_SHLIBS is targeted at that, but it could be set
> for entire recipe only.
>
> This patch expands EXCLUDE_FROM_SHLIBS usage, so now it's possible
> to set it in a style similar with RDEPENDS. For example:
> EXCLUDE_FROM_SHLIBS_${PN}-ptest = "1"
>
> Signed-off-by: Andrii Bordunov <aborduno@cisco.com>
> ---
> meta/classes/package.bbclass | 12 ++++++++++--
> 1 file changed, 10 insertions(+), 2 deletions(-)
My main concern here is actually performance. For a package with many
sub packages you just added many calls to getVar when that getVar is
extremely unlikely to find any value. Whilst not hugely slow, that
operation isn't trivial and hard to fix later.
I'm wondering if we could support the syntax
EXCLUDE_FROM_SHLIBS = "${PN}-ptest"
and then have "1", used in a couple of places as the special value to
mean all packages?
Also please use False, not 0 as the parameter to getVar.
Cheers,
Richard
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [PATCH] package.bbclass: allow using EXCLUDE_FROM_SHLIBS for subpackages
2016-11-16 10:47 ` Richard Purdie
@ 2017-03-16 15:07 ` Andrii Bordunov
2018-03-13 17:25 ` [PATCH v3] " Andrii Bordunov
0 siblings, 1 reply; 10+ messages in thread
From: Andrii Bordunov @ 2017-03-16 15:07 UTC (permalink / raw)
To: Richard Purdie, openembedded-core
Hi Richard,
Sorry, my Python knowledge is quite basic. I'm sure it's possible to
have a (global?) EXCLUDE_FROM_SHLIBS list holding subpackage names, but
how it could lead to performance improvement? It would also need some
check for every subpackage.
Could you please give more details?
And what do you mean under "all packages"? The intent is like opposite,
to split package operations, not to gather.
As for False parameter to getVar, OK, I just used existing
EXCLUDE_FROM_SHLIBS example (seems fixed now).
In the meantime tried to make a quick and dirty profiling like following:
bitbake -c package glibc
bitbake -c clean glibc
python -m cProfile -s cumtime /mnt/src/oe/distro/bitbake/bin/bitbake
-c package glibc > result.txt
Result is below. If got it right, there are 3833 getVar calls with total
time 0.025 second (including subcalls).
This gives ~6.5 microsecond duration of one call (incl subcalls). Not a lot.
This is only a partial rebuild of one package of course.
Also tried to do all of this with and without my patch, but getVar calls
number stays exactly the same (w/o modifying glibc recipe, but anyway).
NOTE: Tasks Summary: Attempted 314 tasks of which 314 didn't need to
be rerun and all succeeded.
672869 function calls (667640 primitive calls) in 12.912 seconds
Ordered by: cumulative time
ncalls tottime percall cumtime percall filename:lineno(function)
1 0.001 0.001 12.913 12.913 bitbake:25(<module>)
1 0.001 0.001 12.082 12.082 main.py:320(bitbake_main)
1 0.005 0.005 11.494 11.494 knotty.py:253(main)
250 11.476 0.046 11.476 0.046 {method 'poll' of
'_multiprocessing.Connection' objects}
...
4481/4046 0.007 0.000 0.026 0.000
data_smart.py:666(getVarFlag)
...
3833/3719 0.002 0.000 0.025 0.000 data_smart.py:569(getVar)
...
1338 0.002 0.000 0.003 0.000
data_smart.py:805(getVarFlags)
Thank you,
Andrii
On 16.11.16 12:47, Richard Purdie wrote:
> On Mon, 2016-10-10 at 20:02 +0300, Andrii Bordunov wrote:
>> Some packages containing shared libraries might be registered
>> as shlib providers when they shouldn't (for example, the lib is for
>> their private use and must not generate any dependency).
>>
>> EXCLUDE_FROM_SHLIBS is targeted at that, but it could be set
>> for entire recipe only.
>>
>> This patch expands EXCLUDE_FROM_SHLIBS usage, so now it's possible
>> to set it in a style similar with RDEPENDS. For example:
>> EXCLUDE_FROM_SHLIBS_${PN}-ptest = "1"
>>
>> Signed-off-by: Andrii Bordunov <aborduno@cisco.com>
>> ---
>> meta/classes/package.bbclass | 12 ++++++++++--
>> 1 file changed, 10 insertions(+), 2 deletions(-)
>
> My main concern here is actually performance. For a package with many
> sub packages you just added many calls to getVar when that getVar is
> extremely unlikely to find any value. Whilst not hugely slow, that
> operation isn't trivial and hard to fix later.
>
> I'm wondering if we could support the syntax
>
> EXCLUDE_FROM_SHLIBS = "${PN}-ptest"
>
> and then have "1", used in a couple of places as the special value to
> mean all packages?
>
> Also please use False, not 0 as the parameter to getVar.
>
> Cheers,
>
> Richard
>
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [PATCH v3] package.bbclass: allow using EXCLUDE_FROM_SHLIBS for subpackages
2017-03-16 15:07 ` Andrii Bordunov
@ 2018-03-13 17:25 ` Andrii Bordunov
2018-04-12 14:42 ` Andrii Bordunov
0 siblings, 1 reply; 10+ messages in thread
From: Andrii Bordunov @ 2018-03-13 17:25 UTC (permalink / raw)
To: Richard Purdie, openembedded-core; +Cc: aborduno, xe-linux-external
Ping
Rebased patch to latest master.
Regarding special value for the variable, it could be done with
PRIVATE_LIBS to mean all libraries, I think. PRIVATE_LIBS could be set
per-package already.
Would it be a better option?
However, EXCLUDE_FROM_SHLIBS seems to be a more proper thing to use.
Thank you,
Andrii
On 03/16/2017 05:07 PM, Andrii Bordunov wrote:
> Hi Richard,
>
> Sorry, my Python knowledge is quite basic. I'm sure it's possible to
> have a (global?) EXCLUDE_FROM_SHLIBS list holding subpackage names, but
> how it could lead to performance improvement? It would also need some
> check for every subpackage.
> Could you please give more details?
> And what do you mean under "all packages"? The intent is like opposite,
> to split package operations, not to gather.
>
> As for False parameter to getVar, OK, I just used existing
> EXCLUDE_FROM_SHLIBS example (seems fixed now).
>
> In the meantime tried to make a quick and dirty profiling like following:
>
> bitbake -c package glibc
> bitbake -c clean glibc
> python -m cProfile -s cumtime /mnt/src/oe/distro/bitbake/bin/bitbake
> -c package glibc > result.txt
>
> Result is below. If got it right, there are 3833 getVar calls with total
> time 0.025 second (including subcalls).
> This gives ~6.5 microsecond duration of one call (incl subcalls). Not a
> lot.
> This is only a partial rebuild of one package of course.
> Also tried to do all of this with and without my patch, but getVar calls
> number stays exactly the same (w/o modifying glibc recipe, but anyway).
>
> NOTE: Tasks Summary: Attempted 314 tasks of which 314 didn't need to
> be rerun and all succeeded.
> 672869 function calls (667640 primitive calls) in 12.912
> seconds
> Ordered by: cumulative time
> ncalls tottime percall cumtime percall filename:lineno(function)
> 1 0.001 0.001 12.913 12.913 bitbake:25(<module>)
> 1 0.001 0.001 12.082 12.082 main.py:320(bitbake_main)
> 1 0.005 0.005 11.494 11.494 knotty.py:253(main)
> 250 11.476 0.046 11.476 0.046 {method 'poll' of
> '_multiprocessing.Connection' objects}
> ...
> 4481/4046 0.007 0.000 0.026 0.000
> data_smart.py:666(getVarFlag)
> ...
> 3833/3719 0.002 0.000 0.025 0.000 data_smart.py:569(getVar)
> ...
> 1338 0.002 0.000 0.003 0.000
> data_smart.py:805(getVarFlags)
>
>
>
> Thank you,
> Andrii
>
> On 16.11.16 12:47, Richard Purdie wrote:
>> On Mon, 2016-10-10 at 20:02 +0300, Andrii Bordunov wrote:
>>> Some packages containing shared libraries might be registered
>>> as shlib providers when they shouldn't (for example, the lib is for
>>> their private use and must not generate any dependency).
>>>
>>> EXCLUDE_FROM_SHLIBS is targeted at that, but it could be set
>>> for entire recipe only.
>>>
>>> This patch expands EXCLUDE_FROM_SHLIBS usage, so now it's possible
>>> to set it in a style similar with RDEPENDS. For example:
>>> EXCLUDE_FROM_SHLIBS_${PN}-ptest = "1"
>>>
>>> Signed-off-by: Andrii Bordunov <aborduno@cisco.com>
>>> ---
>>> meta/classes/package.bbclass | 12 ++++++++++--
>>> 1 file changed, 10 insertions(+), 2 deletions(-)
>>
>> My main concern here is actually performance. For a package with many
>> sub packages you just added many calls to getVar when that getVar is
>> extremely unlikely to find any value. Whilst not hugely slow, that
>> operation isn't trivial and hard to fix later.
>>
>> I'm wondering if we could support the syntax
>>
>> EXCLUDE_FROM_SHLIBS = "${PN}-ptest"
>>
>> and then have "1", used in a couple of places as the special value to
>> mean all packages?
>>
>> Also please use False, not 0 as the parameter to getVar.
>>
>> Cheers,
>>
>> Richard
--
From 4a33d384a72a351bad5a98e46a7c9c1c9755635e Mon Sep 17 00:00:00 2001
From: Andrii Bordunov <aborduno@cisco.com>
Date: Mon, 12 Mar 2018 08:13:44 -0700
Subject: [PATCH v3] Allow EXCLUDE_FROM_SHLIBS to be specified for
subpackages
Some subpackages containing shared libraries might be registered
as shlib providers when they shouldn't (the lib is for their private use
and must not generate any dependency).
EXCLUDE_FROM_SHLIBS is targeted at that, but it could be set
for entire recipe only.
This patch expands EXCLUDE_FROM_SHLIBS usage, so now it's possible
to set it like following:
EXCLUDE_FROM_SHLIBS_${PN}-ptest = "1"
Signed-off-by: Andrii Bordunov <aborduno@cisco.com>
---
meta/classes/package.bbclass | 11 +++++++++--
1 file changed, 9 insertions(+), 2 deletions(-)
diff --git a/meta/classes/package.bbclass b/meta/classes/package.bbclass
index 317c775..5d49d41 100644
--- a/meta/classes/package.bbclass
+++ b/meta/classes/package.bbclass
@@ -1571,6 +1571,13 @@ python package_do_shlibs() {
workdir = d.getVar('WORKDIR')
+ shlib_pkgs = []
+ for pkg in packages.split():
+ if d.getVar('EXCLUDE_FROM_SHLIBS_' + pkg, False):
+ bb.note("not generating shlibs for %s" % pkg)
+ else:
+ shlib_pkgs.append(pkg)
+
ver = d.getVar('PKGV')
if not ver:
msg = "PKGV not defined"
@@ -1698,7 +1705,7 @@ python package_do_shlibs() {
needed = {}
shlib_provider = oe.package.read_shlib_providers(d)
- for pkg in packages.split():
+ for pkg in shlib_pkgs:
private_libs = d.getVar('PRIVATE_LIBS_' + pkg) or
d.getVar('PRIVATE_LIBS') or ""
private_libs = private_libs.split()
needs_ldconfig = False
@@ -1770,7 +1777,7 @@ python package_do_shlibs() {
libsearchpath = [d.getVar('libdir'), d.getVar('base_libdir')]
- for pkg in packages.split():
+ for pkg in shlib_pkgs:
bb.debug(2, "calculating shlib requirements for %s" % pkg)
private_libs = d.getVar('PRIVATE_LIBS_' + pkg) or
d.getVar('PRIVATE_LIBS') or ""
--
2.10.3.dirty
^ permalink raw reply related [flat|nested] 10+ messages in thread
* Re: [PATCH v3] package.bbclass: allow using EXCLUDE_FROM_SHLIBS for subpackages
2018-03-13 17:25 ` [PATCH v3] " Andrii Bordunov
@ 2018-04-12 14:42 ` Andrii Bordunov
2018-06-11 12:20 ` Andrii Bordunov
0 siblings, 1 reply; 10+ messages in thread
From: Andrii Bordunov @ 2018-04-12 14:42 UTC (permalink / raw)
To: Richard Purdie, openembedded-core, Burton Ross; +Cc: xe-linux-external
Another ping.
Any comments? Or could it be just applied as is?
Thank you,
Andrii
On 03/13/2018 07:25 PM, Andrii Bordunov wrote:
> Ping
>
> Rebased patch to latest master.
>
> Regarding special value for the variable, it could be done with
> PRIVATE_LIBS to mean all libraries, I think. PRIVATE_LIBS could be set
> per-package already.
> Would it be a better option?
> However, EXCLUDE_FROM_SHLIBS seems to be a more proper thing to use.
>
>
> Thank you,
> Andrii
>
> On 03/16/2017 05:07 PM, Andrii Bordunov wrote:
>> Hi Richard,
>>
>> Sorry, my Python knowledge is quite basic. I'm sure it's possible to
>> have a (global?) EXCLUDE_FROM_SHLIBS list holding subpackage names,
>> but how it could lead to performance improvement? It would also need
>> some check for every subpackage.
>> Could you please give more details?
>> And what do you mean under "all packages"? The intent is like
>> opposite, to split package operations, not to gather.
>>
>> As for False parameter to getVar, OK, I just used existing
>> EXCLUDE_FROM_SHLIBS example (seems fixed now).
>>
>> In the meantime tried to make a quick and dirty profiling like following:
>>
>> bitbake -c package glibc
>> bitbake -c clean glibc
>> python -m cProfile -s cumtime
>> /mnt/src/oe/distro/bitbake/bin/bitbake -c package glibc > result.txt
>>
>> Result is below. If got it right, there are 3833 getVar calls with
>> total time 0.025 second (including subcalls).
>> This gives ~6.5 microsecond duration of one call (incl subcalls). Not
>> a lot.
>> This is only a partial rebuild of one package of course.
>> Also tried to do all of this with and without my patch, but getVar
>> calls number stays exactly the same (w/o modifying glibc recipe, but
>> anyway).
>>
>> NOTE: Tasks Summary: Attempted 314 tasks of which 314 didn't need
>> to be rerun and all succeeded.
>> 672869 function calls (667640 primitive calls) in 12.912
>> seconds
>> Ordered by: cumulative time
>> ncalls tottime percall cumtime percall
>> filename:lineno(function)
>> 1 0.001 0.001 12.913 12.913 bitbake:25(<module>)
>> 1 0.001 0.001 12.082 12.082
>> main.py:320(bitbake_main)
>> 1 0.005 0.005 11.494 11.494 knotty.py:253(main)
>> 250 11.476 0.046 11.476 0.046 {method 'poll' of
>> '_multiprocessing.Connection' objects}
>> ...
>> 4481/4046 0.007 0.000 0.026 0.000
>> data_smart.py:666(getVarFlag)
>> ...
>> 3833/3719 0.002 0.000 0.025 0.000
>> data_smart.py:569(getVar)
>> ...
>> 1338 0.002 0.000 0.003 0.000
>> data_smart.py:805(getVarFlags)
>>
>>
>>
>> Thank you,
>> Andrii
>>
>> On 16.11.16 12:47, Richard Purdie wrote:
>>> On Mon, 2016-10-10 at 20:02 +0300, Andrii Bordunov wrote:
>>>> Some packages containing shared libraries might be registered
>>>> as shlib providers when they shouldn't (for example, the lib is for
>>>> their private use and must not generate any dependency).
>>>>
>>>> EXCLUDE_FROM_SHLIBS is targeted at that, but it could be set
>>>> for entire recipe only.
>>>>
>>>> This patch expands EXCLUDE_FROM_SHLIBS usage, so now it's possible
>>>> to set it in a style similar with RDEPENDS. For example:
>>>> EXCLUDE_FROM_SHLIBS_${PN}-ptest = "1"
>>>>
>>>> Signed-off-by: Andrii Bordunov <aborduno@cisco.com>
>>>> ---
>>>> meta/classes/package.bbclass | 12 ++++++++++--
>>>> 1 file changed, 10 insertions(+), 2 deletions(-)
>>>
>>> My main concern here is actually performance. For a package with many
>>> sub packages you just added many calls to getVar when that getVar is
>>> extremely unlikely to find any value. Whilst not hugely slow, that
>>> operation isn't trivial and hard to fix later.
>>>
>>> I'm wondering if we could support the syntax
>>>
>>> EXCLUDE_FROM_SHLIBS = "${PN}-ptest"
>>>
>>> and then have "1", used in a couple of places as the special value to
>>> mean all packages?
>>>
>>> Also please use False, not 0 as the parameter to getVar.
>>>
>>> Cheers,
>>>
>>> Richard
--
From 4a33d384a72a351bad5a98e46a7c9c1c9755635e Mon Sep 17 00:00:00 2001
From: Andrii Bordunov <aborduno@cisco.com>
Date: Mon, 12 Mar 2018 08:13:44 -0700
Subject: [PATCH v3] Allow EXCLUDE_FROM_SHLIBS to be specified for
subpackages
Some subpackages containing shared libraries might be registered
as shlib providers when they shouldn't (the lib is for their private use
and must not generate any dependency).
EXCLUDE_FROM_SHLIBS is targeted at that, but it could be set
for entire recipe only.
This patch expands EXCLUDE_FROM_SHLIBS usage, so now it's possible
to set it like following:
EXCLUDE_FROM_SHLIBS_${PN}-ptest = "1"
Signed-off-by: Andrii Bordunov <aborduno@cisco.com>
---
meta/classes/package.bbclass | 11 +++++++++--
1 file changed, 9 insertions(+), 2 deletions(-)
diff --git a/meta/classes/package.bbclass b/meta/classes/package.bbclass
index 317c775..5d49d41 100644
--- a/meta/classes/package.bbclass
+++ b/meta/classes/package.bbclass
@@ -1571,6 +1571,13 @@ python package_do_shlibs() {
workdir = d.getVar('WORKDIR')
+ shlib_pkgs = []
+ for pkg in packages.split():
+ if d.getVar('EXCLUDE_FROM_SHLIBS_' + pkg, False):
+ bb.note("not generating shlibs for %s" % pkg)
+ else:
+ shlib_pkgs.append(pkg)
+
ver = d.getVar('PKGV')
if not ver:
msg = "PKGV not defined"
@@ -1698,7 +1705,7 @@ python package_do_shlibs() {
needed = {}
shlib_provider = oe.package.read_shlib_providers(d)
- for pkg in packages.split():
+ for pkg in shlib_pkgs:
private_libs = d.getVar('PRIVATE_LIBS_' + pkg) or
d.getVar('PRIVATE_LIBS') or ""
private_libs = private_libs.split()
needs_ldconfig = False
@@ -1770,7 +1777,7 @@ python package_do_shlibs() {
libsearchpath = [d.getVar('libdir'), d.getVar('base_libdir')]
- for pkg in packages.split():
+ for pkg in shlib_pkgs:
bb.debug(2, "calculating shlib requirements for %s" % pkg)
private_libs = d.getVar('PRIVATE_LIBS_' + pkg) or
d.getVar('PRIVATE_LIBS') or ""
--
2.10.3.dirty
^ permalink raw reply related [flat|nested] 10+ messages in thread
* Re: [PATCH v3] package.bbclass: allow using EXCLUDE_FROM_SHLIBS for subpackages
2018-04-12 14:42 ` Andrii Bordunov
@ 2018-06-11 12:20 ` Andrii Bordunov
2018-06-16 12:00 ` Richard Purdie
0 siblings, 1 reply; 10+ messages in thread
From: Andrii Bordunov @ 2018-06-11 12:20 UTC (permalink / raw)
To: Richard Purdie, openembedded-core, Burton Ross; +Cc: xe-linux-external
Ping again.
Thank you,
Andrii
On 04/12/2018 05:42 PM, Andrii Bordunov wrote:
> Another ping.
>
> Any comments? Or could it be just applied as is?
>
>
> Thank you,
> Andrii
>
> On 03/13/2018 07:25 PM, Andrii Bordunov wrote:
>> Ping
>>
>> Rebased patch to latest master.
>>
>> Regarding special value for the variable, it could be done with
>> PRIVATE_LIBS to mean all libraries, I think. PRIVATE_LIBS could be set
>> per-package already.
>> Would it be a better option?
>> However, EXCLUDE_FROM_SHLIBS seems to be a more proper thing to use.
>>
>>
>> Thank you,
>> Andrii
>>
>> On 03/16/2017 05:07 PM, Andrii Bordunov wrote:
>>> Hi Richard,
>>>
>>> Sorry, my Python knowledge is quite basic. I'm sure it's possible to
>>> have a (global?) EXCLUDE_FROM_SHLIBS list holding subpackage names,
>>> but how it could lead to performance improvement? It would also need
>>> some check for every subpackage.
>>> Could you please give more details?
>>> And what do you mean under "all packages"? The intent is like
>>> opposite, to split package operations, not to gather.
>>>
>>> As for False parameter to getVar, OK, I just used existing
>>> EXCLUDE_FROM_SHLIBS example (seems fixed now).
>>>
>>> In the meantime tried to make a quick and dirty profiling like
>>> following:
>>>
>>> bitbake -c package glibc
>>> bitbake -c clean glibc
>>> python -m cProfile -s cumtime
>>> /mnt/src/oe/distro/bitbake/bin/bitbake -c package glibc > result.txt
>>>
>>> Result is below. If got it right, there are 3833 getVar calls with
>>> total time 0.025 second (including subcalls).
>>> This gives ~6.5 microsecond duration of one call (incl subcalls). Not
>>> a lot.
>>> This is only a partial rebuild of one package of course.
>>> Also tried to do all of this with and without my patch, but getVar
>>> calls number stays exactly the same (w/o modifying glibc recipe, but
>>> anyway).
>>>
>>> NOTE: Tasks Summary: Attempted 314 tasks of which 314 didn't need
>>> to be rerun and all succeeded.
>>> 672869 function calls (667640 primitive calls) in 12.912
>>> seconds
>>> Ordered by: cumulative time
>>> ncalls tottime percall cumtime percall
>>> filename:lineno(function)
>>> 1 0.001 0.001 12.913 12.913 bitbake:25(<module>)
>>> 1 0.001 0.001 12.082 12.082
>>> main.py:320(bitbake_main)
>>> 1 0.005 0.005 11.494 11.494 knotty.py:253(main)
>>> 250 11.476 0.046 11.476 0.046 {method 'poll' of
>>> '_multiprocessing.Connection' objects}
>>> ...
>>> 4481/4046 0.007 0.000 0.026 0.000
>>> data_smart.py:666(getVarFlag)
>>> ...
>>> 3833/3719 0.002 0.000 0.025 0.000
>>> data_smart.py:569(getVar)
>>> ...
>>> 1338 0.002 0.000 0.003 0.000
>>> data_smart.py:805(getVarFlags)
>>>
>>>
>>>
>>> Thank you,
>>> Andrii
>>>
>>> On 16.11.16 12:47, Richard Purdie wrote:
>>>> On Mon, 2016-10-10 at 20:02 +0300, Andrii Bordunov wrote:
>>>>> Some packages containing shared libraries might be registered
>>>>> as shlib providers when they shouldn't (for example, the lib is for
>>>>> their private use and must not generate any dependency).
>>>>>
>>>>> EXCLUDE_FROM_SHLIBS is targeted at that, but it could be set
>>>>> for entire recipe only.
>>>>>
>>>>> This patch expands EXCLUDE_FROM_SHLIBS usage, so now it's possible
>>>>> to set it in a style similar with RDEPENDS. For example:
>>>>> EXCLUDE_FROM_SHLIBS_${PN}-ptest = "1"
>>>>>
>>>>> Signed-off-by: Andrii Bordunov <aborduno@cisco.com>
>>>>> ---
>>>>> meta/classes/package.bbclass | 12 ++++++++++--
>>>>> 1 file changed, 10 insertions(+), 2 deletions(-)
>>>>
>>>> My main concern here is actually performance. For a package with many
>>>> sub packages you just added many calls to getVar when that getVar is
>>>> extremely unlikely to find any value. Whilst not hugely slow, that
>>>> operation isn't trivial and hard to fix later.
>>>>
>>>> I'm wondering if we could support the syntax
>>>>
>>>> EXCLUDE_FROM_SHLIBS = "${PN}-ptest"
>>>>
>>>> and then have "1", used in a couple of places as the special value to
>>>> mean all packages?
>>>>
>>>> Also please use False, not 0 as the parameter to getVar.
>>>>
>>>> Cheers,
>>>>
>>>> Richard
>
> --
>
> From 4a33d384a72a351bad5a98e46a7c9c1c9755635e Mon Sep 17 00:00:00 2001
> From: Andrii Bordunov <aborduno@cisco.com>
> Date: Mon, 12 Mar 2018 08:13:44 -0700
> Subject: [PATCH v3] Allow EXCLUDE_FROM_SHLIBS to be specified for
> subpackages
>
> Some subpackages containing shared libraries might be registered
> as shlib providers when they shouldn't (the lib is for their private use
> and must not generate any dependency).
>
> EXCLUDE_FROM_SHLIBS is targeted at that, but it could be set
> for entire recipe only.
>
> This patch expands EXCLUDE_FROM_SHLIBS usage, so now it's possible
> to set it like following:
> EXCLUDE_FROM_SHLIBS_${PN}-ptest = "1"
>
> Signed-off-by: Andrii Bordunov <aborduno@cisco.com>
> ---
> meta/classes/package.bbclass | 11 +++++++++--
> 1 file changed, 9 insertions(+), 2 deletions(-)
>
> diff --git a/meta/classes/package.bbclass b/meta/classes/package.bbclass
> index 317c775..5d49d41 100644
> --- a/meta/classes/package.bbclass
> +++ b/meta/classes/package.bbclass
> @@ -1571,6 +1571,13 @@ python package_do_shlibs() {
>
> workdir = d.getVar('WORKDIR')
>
> + shlib_pkgs = []
> + for pkg in packages.split():
> + if d.getVar('EXCLUDE_FROM_SHLIBS_' + pkg, False):
> + bb.note("not generating shlibs for %s" % pkg)
> + else:
> + shlib_pkgs.append(pkg)
> +
> ver = d.getVar('PKGV')
> if not ver:
> msg = "PKGV not defined"
> @@ -1698,7 +1705,7 @@ python package_do_shlibs() {
> needed = {}
> shlib_provider = oe.package.read_shlib_providers(d)
>
> - for pkg in packages.split():
> + for pkg in shlib_pkgs:
> private_libs = d.getVar('PRIVATE_LIBS_' + pkg) or
> d.getVar('PRIVATE_LIBS') or ""
> private_libs = private_libs.split()
> needs_ldconfig = False
> @@ -1770,7 +1777,7 @@ python package_do_shlibs() {
>
> libsearchpath = [d.getVar('libdir'), d.getVar('base_libdir')]
>
> - for pkg in packages.split():
> + for pkg in shlib_pkgs:
> bb.debug(2, "calculating shlib requirements for %s" % pkg)
>
> private_libs = d.getVar('PRIVATE_LIBS_' + pkg) or
> d.getVar('PRIVATE_LIBS') or ""
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [PATCH v3] package.bbclass: allow using EXCLUDE_FROM_SHLIBS for subpackages
2018-06-11 12:20 ` Andrii Bordunov
@ 2018-06-16 12:00 ` Richard Purdie
0 siblings, 0 replies; 10+ messages in thread
From: Richard Purdie @ 2018-06-16 12:00 UTC (permalink / raw)
To: Andrii Bordunov, openembedded-core, Burton Ross; +Cc: xe-linux-external
Sorry about the delay on this, my concern still remains. Having thought
about this a bit, the performance issue wouldn't show up with glibc, it
would probably show up better with glibc-locales, perl (due to its
modules) or the kernel (again due to large number of modules) as these
all have significantly more packages.
Basically, regardless of the profile data, adding new API which mean
hundreds of new getVar calls for something which is not common case,
isn't something I really want to do as it limits us in the future.
I'd much prefer we have one variable which lists all packages which
need to be excluded from shlibs and have some special values which mean
"all packages", including "1" for backwards compatibility. I'd be happy
to see "1" trigger a warning and have it replaced with "*" or
something with clearer meaning.
Cheers,
Richard
On Mon, 2018-06-11 at 15:20 +0300, Andrii Bordunov wrote:
> Ping again.
>
>
> Thank you,
> Andrii
>
> On 04/12/2018 05:42 PM, Andrii Bordunov wrote:
> > Another ping.
> >
> > Any comments? Or could it be just applied as is?
> >
> >
> > Thank you,
> > Andrii
> >
> > On 03/13/2018 07:25 PM, Andrii Bordunov wrote:
> > > Ping
> > >
> > > Rebased patch to latest master.
> > >
> > > Regarding special value for the variable, it could be done with
> > > PRIVATE_LIBS to mean all libraries, I think. PRIVATE_LIBS could
> > > be set
> > > per-package already.
> > > Would it be a better option?
> > > However, EXCLUDE_FROM_SHLIBS seems to be a more proper thing to
> > > use.
> > >
> > >
> > > Thank you,
> > > Andrii
> > >
> > > On 03/16/2017 05:07 PM, Andrii Bordunov wrote:
> > > > Hi Richard,
> > > >
> > > > Sorry, my Python knowledge is quite basic. I'm sure it's
> > > > possible to
> > > > have a (global?) EXCLUDE_FROM_SHLIBS list holding subpackage
> > > > names,
> > > > but how it could lead to performance improvement? It would also
> > > > need
> > > > some check for every subpackage.
> > > > Could you please give more details?
> > > > And what do you mean under "all packages"? The intent is like
> > > > opposite, to split package operations, not to gather.
> > > >
> > > > As for False parameter to getVar, OK, I just used existing
> > > > EXCLUDE_FROM_SHLIBS example (seems fixed now).
> > > >
> > > > In the meantime tried to make a quick and dirty profiling like
> > > > following:
> > > >
> > > > bitbake -c package glibc
> > > > bitbake -c clean glibc
> > > > python -m cProfile -s cumtime
> > > > /mnt/src/oe/distro/bitbake/bin/bitbake -c package glibc >
> > > > result.txt
> > > >
> > > > Result is below. If got it right, there are 3833 getVar calls
> > > > with
> > > > total time 0.025 second (including subcalls).
> > > > This gives ~6.5 microsecond duration of one call (incl
> > > > subcalls). Not
> > > > a lot.
> > > > This is only a partial rebuild of one package of course.
> > > > Also tried to do all of this with and without my patch, but
> > > > getVar
> > > > calls number stays exactly the same (w/o modifying glibc
> > > > recipe, but
> > > > anyway).
> > > >
> > > > NOTE: Tasks Summary: Attempted 314 tasks of which 314 didn't
> > > > need
> > > > to be rerun and all succeeded.
> > > > 672869 function calls (667640 primitive calls) in
> > > > 12.912
> > > > seconds
> > > > Ordered by: cumulative time
> > > > ncalls tottime percall cumtime percall
> > > > filename:lineno(function)
> > > > 1 0.001 0.001 12.913 12.913
> > > > bitbake:25(<module>)
> > > > 1 0.001 0.001 12.082 12.082
> > > > main.py:320(bitbake_main)
> > > > 1 0.005 0.005 11.494 11.494
> > > > knotty.py:253(main)
> > > > 250 11.476 0.046 11.476 0.046 {method 'poll'
> > > > of
> > > > '_multiprocessing.Connection' objects}
> > > > ...
> > > > 4481/4046 0.007 0.000 0.026 0.000
> > > > data_smart.py:666(getVarFlag)
> > > > ...
> > > > 3833/3719 0.002 0.000 0.025 0.000
> > > > data_smart.py:569(getVar)
> > > > ...
> > > > 1338 0.002 0.000 0.003 0.000
> > > > data_smart.py:805(getVarFlags)
> > > >
> > > >
> > > >
> > > > Thank you,
> > > > Andrii
> > > >
> > > > On 16.11.16 12:47, Richard Purdie wrote:
> > > > > On Mon, 2016-10-10 at 20:02 +0300, Andrii Bordunov wrote:
> > > > > > Some packages containing shared libraries might be
> > > > > > registered
> > > > > > as shlib providers when they shouldn't (for example, the
> > > > > > lib is for
> > > > > > their private use and must not generate any dependency).
> > > > > >
> > > > > > EXCLUDE_FROM_SHLIBS is targeted at that, but it could be
> > > > > > set
> > > > > > for entire recipe only.
> > > > > >
> > > > > > This patch expands EXCLUDE_FROM_SHLIBS usage, so now it's
> > > > > > possible
> > > > > > to set it in a style similar with RDEPENDS. For example:
> > > > > > EXCLUDE_FROM_SHLIBS_${PN}-ptest = "1"
> > > > > >
> > > > > > Signed-off-by: Andrii Bordunov <aborduno@cisco.com>
> > > > > > ---
> > > > > > meta/classes/package.bbclass | 12 ++++++++++--
> > > > > > 1 file changed, 10 insertions(+), 2 deletions(-)
> > > > >
> > > > > My main concern here is actually performance. For a package
> > > > > with many
> > > > > sub packages you just added many calls to getVar when that
> > > > > getVar is
> > > > > extremely unlikely to find any value. Whilst not hugely slow,
> > > > > that
> > > > > operation isn't trivial and hard to fix later.
> > > > >
> > > > > I'm wondering if we could support the syntax
> > > > >
> > > > > EXCLUDE_FROM_SHLIBS = "${PN}-ptest"
> > > > >
> > > > > and then have "1", used in a couple of places as the special
> > > > > value to
> > > > > mean all packages?
> > > > >
> > > > > Also please use False, not 0 as the parameter to getVar.
> > > > >
> > > > > Cheers,
> > > > >
> > > > > Richard
> >
> > --
> >
> > From 4a33d384a72a351bad5a98e46a7c9c1c9755635e Mon Sep 17 00:00:00
> > 2001
> > From: Andrii Bordunov <aborduno@cisco.com>
> > Date: Mon, 12 Mar 2018 08:13:44 -0700
> > Subject: [PATCH v3] Allow EXCLUDE_FROM_SHLIBS to be specified for
> > subpackages
> >
> > Some subpackages containing shared libraries might be registered
> > as shlib providers when they shouldn't (the lib is for their
> > private use
> > and must not generate any dependency).
> >
> > EXCLUDE_FROM_SHLIBS is targeted at that, but it could be set
> > for entire recipe only.
> >
> > This patch expands EXCLUDE_FROM_SHLIBS usage, so now it's possible
> > to set it like following:
> > EXCLUDE_FROM_SHLIBS_${PN}-ptest = "1"
> >
> > Signed-off-by: Andrii Bordunov <aborduno@cisco.com>
> > ---
> > meta/classes/package.bbclass | 11 +++++++++--
> > 1 file changed, 9 insertions(+), 2 deletions(-)
> >
> > diff --git a/meta/classes/package.bbclass
> > b/meta/classes/package.bbclass
> > index 317c775..5d49d41 100644
> > --- a/meta/classes/package.bbclass
> > +++ b/meta/classes/package.bbclass
> > @@ -1571,6 +1571,13 @@ python package_do_shlibs() {
> >
> > workdir = d.getVar('WORKDIR')
> >
> > + shlib_pkgs = []
> > + for pkg in packages.split():
> > + if d.getVar('EXCLUDE_FROM_SHLIBS_' + pkg, False):
> > + bb.note("not generating shlibs for %s" % pkg)
> > + else:
> > + shlib_pkgs.append(pkg)
> > +
> > ver = d.getVar('PKGV')
> > if not ver:
> > msg = "PKGV not defined"
> > @@ -1698,7 +1705,7 @@ python package_do_shlibs() {
> > needed = {}
> > shlib_provider = oe.package.read_shlib_providers(d)
> >
> > - for pkg in packages.split():
> > + for pkg in shlib_pkgs:
> > private_libs = d.getVar('PRIVATE_LIBS_' + pkg) or
> > d.getVar('PRIVATE_LIBS') or ""
> > private_libs = private_libs.split()
> > needs_ldconfig = False
> > @@ -1770,7 +1777,7 @@ python package_do_shlibs() {
> >
> > libsearchpath = [d.getVar('libdir'),
> > d.getVar('base_libdir')]
> >
> > - for pkg in packages.split():
> > + for pkg in shlib_pkgs:
> > bb.debug(2, "calculating shlib requirements for %s" %
> > pkg)
> >
> > private_libs = d.getVar('PRIVATE_LIBS_' + pkg) or
> > d.getVar('PRIVATE_LIBS') or ""
>
>
^ permalink raw reply [flat|nested] 10+ messages in thread
end of thread, other threads:[~2018-06-16 12:01 UTC | newest]
Thread overview: 10+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2016-10-10 17:02 [PATCH] package.bbclass: allow using EXCLUDE_FROM_SHLIBS for subpackages Andrii Bordunov
2016-10-19 13:58 ` Andrii Bordunov
2016-11-15 15:05 ` Andrii Bordunov
2016-11-16 10:29 ` Burton, Ross
2016-11-16 10:47 ` Richard Purdie
2017-03-16 15:07 ` Andrii Bordunov
2018-03-13 17:25 ` [PATCH v3] " Andrii Bordunov
2018-04-12 14:42 ` Andrii Bordunov
2018-06-11 12:20 ` Andrii Bordunov
2018-06-16 12:00 ` Richard Purdie
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.