[PATCH 0/2] Extensible SDK size reduction

[PATCH 0/2] Extensible SDK size reduction

[Randy Witt]

This removes some unneeded items from the extensible sdk to reduce size by
~300MB.

The following changes since commit f8185ff9f8ac697733b41f023cb3a4b98b2f823f:

  bitbake: ast: Add filename/lineno to mapped functions (2016-01-06 15:27:35 +0000)

are available in the git repository at:

  git://git.yoctoproject.org/poky-contrib rewitt/extsdksize
  http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=rewitt/extsdksize

Randy Witt (2):
  copy_buildsystem.py: Allow substring matches for excluded_targets
  populate_sdk_ext: Don't put nativesdk and crosssdk items in ext sdk

 meta/classes/populate_sdk_ext.bbclass | 6 ++++++
 meta/lib/oe/copy_buildsystem.py       | 7 +++++--
 2 files changed, 11 insertions(+), 2 deletions(-)

-- 
2.5.0

[PATCH 1/2] copy_buildsystem.py: Allow substring matches for excluded_targets

[Randy Witt]

This is so that the excluded_targets passed to prune_lockedsigs can
take a partial form such as nativesdk to filter based on partial matches.

Signed-off-by: Randy Witt <randy.e.witt@linux.intel.com>
---
 meta/lib/oe/copy_buildsystem.py | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)

diff --git a/meta/lib/oe/copy_buildsystem.py b/meta/lib/oe/copy_buildsystem.py
index 15af4eb..9d8f3a6 100644
--- a/meta/lib/oe/copy_buildsystem.py
+++ b/meta/lib/oe/copy_buildsystem.py
@@ -83,8 +83,11 @@ def prune_lockedsigs(allowed_tasks, excluded_targets, lockedsigs, pruned_output)
             for line in infile:
                 if invalue:
                     if line.endswith('\\\n'):
-                        splitval = line.strip().split(':')
-                        if splitval[1] in allowed_tasks and not splitval[0] in excluded_targets:
+                        target, task = line.strip().split(':')[:2]
+                        badmatch = ([x for x in excluded_targets.split()
+                                     if x in target])
+
+                        if task in allowed_tasks and not badmatch:
                             f.write(line)
                     else:
                         f.write(line)
-- 
2.5.0

[PATCH 2/2] populate_sdk_ext: Don't put nativesdk and crosssdk items in ext sdk

[Randy Witt]

The nativesdk and crosssdk targets were only getting pulled into the
extensible sdk due to buildtools dependencies being in BB_TASKDEPDATA.
However the nativesdk and crosssdk targets are not needed for the
extensible sdk to be used.

This patch therefore removes them since it reduces the size of the
extensible sdks that include sstate by ~300MB.

Signed-off-by: Randy Witt <randy.e.witt@linux.intel.com>
---
 meta/classes/populate_sdk_ext.bbclass | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/meta/classes/populate_sdk_ext.bbclass b/meta/classes/populate_sdk_ext.bbclass
index c30181a..60467b9 100644
--- a/meta/classes/populate_sdk_ext.bbclass
+++ b/meta/classes/populate_sdk_ext.bbclass
@@ -173,6 +173,12 @@ python copy_buildsystem () {
     # Filter the locked signatures file to just the sstate tasks we are interested in
     allowed_tasks = ['do_populate_lic', 'do_populate_sysroot', 'do_packagedata', 'do_package_write_ipk', 'do_package_write_rpm', 'do_package_write_deb', 'do_package_qa', 'do_deploy']
     excluded_targets = d.getVar('SDK_TARGETS', True)
+
+    # Since buildtools is included in the sdk, the nativesdk items and
+    # crosssdk end up in the tasks in locked-sigs.inc. However, the nativesdk
+    # and crosssdk items aren't needed for the extensible sdk to function so
+    # remove them from the list.
+    excluded_targets += 'nativesdk crosssdk'
     lockedsigs_pruned = baseoutpath + '/conf/locked-sigs.inc'
     oe.copy_buildsystem.prune_lockedsigs(allowed_tasks,
                                          excluded_targets,
-- 
2.5.0

Re: [PATCH 1/2] copy_buildsystem.py: Allow substring matches for excluded_targets

[Paul Eggleton]

Hi Randy,

On Wed, 06 Jan 2016 14:15:49 Randy Witt wrote:
> This is so that the excluded_targets passed to prune_lockedsigs can
> take a partial form such as nativesdk to filter based on partial matches.
> 
> Signed-off-by: Randy Witt <randy.e.witt@linux.intel.com>
> ---
>  meta/lib/oe/copy_buildsystem.py | 7 +++++--
>  1 file changed, 5 insertions(+), 2 deletions(-)
> 
> diff --git a/meta/lib/oe/copy_buildsystem.py
> b/meta/lib/oe/copy_buildsystem.py index 15af4eb..9d8f3a6 100644
> --- a/meta/lib/oe/copy_buildsystem.py
> +++ b/meta/lib/oe/copy_buildsystem.py
> @@ -83,8 +83,11 @@ def prune_lockedsigs(allowed_tasks, excluded_targets,
> lockedsigs, pruned_output) for line in infile:
>                  if invalue:
>                      if line.endswith('\\\n'):
> -                        splitval = line.strip().split(':')
> -                        if splitval[1] in allowed_tasks and not splitval[0]
> in excluded_targets: +                        target, task =
> line.strip().split(':')[:2] +                        badmatch = ([x for x
> in excluded_targets.split() +                                     if x in
> target])
> +
> +                        if task in allowed_tasks and not badmatch:

Actually Mark and I have just noticed excluded_targets is being treated as a 
list in the original code here but we're passing in a string. I think maybe 
expecting the caller to supply a list would be a better approach (so we would 
call .split() on the value of SDK_TARGETS before passing it in). Could you 
take care of that in a separate patch before doing this?

Thanks,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre

[PATCH v2 0/3] Extensible SDK size reduction

[Randy Witt]

This removes some unneeded items from the extensible sdk to reduce size by
~300MB.

Includes an additional patch

The following changes since commit 0f3ad7c77aa0f8bb7641af1886e1420cdfa8419f:

  scripts/yocto-layer: Avoids duplication of "meta-" prefix (2016-01-07 13:52:21 +0000)

are available in the git repository at:

  git://git.yoctoproject.org/poky-contrib rewitt/extsdksize
  http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=rewitt/extsdksize

Randy Witt (3):
  populate_sdk_ext: Pass excluded_targets as a list to prune_lockedsigs
  copy_buildsystem.py: Allow substring matches for excluded_targets
  populate_sdk_ext: Don't put nativesdk and crosssdk items in ext sdk

 meta/classes/populate_sdk_ext.bbclass | 8 +++++++-
 meta/lib/oe/copy_buildsystem.py       | 7 +++++--
 2 files changed, 12 insertions(+), 3 deletions(-)

-- 
2.5.0

[PATCH 1/3] populate_sdk_ext: Pass excluded_targets as a list to prune_lockedsigs

[Randy Witt]

prune_lockedsigs expects excluded_targets to be a list, whereas
previously it was passed in as a string.

Signed-off-by: Randy Witt <randy.e.witt@linux.intel.com>
---
 meta/classes/populate_sdk_ext.bbclass | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/meta/classes/populate_sdk_ext.bbclass b/meta/classes/populate_sdk_ext.bbclass
index c30181a..5c99bd0 100644
--- a/meta/classes/populate_sdk_ext.bbclass
+++ b/meta/classes/populate_sdk_ext.bbclass
@@ -175,7 +175,7 @@ python copy_buildsystem () {
     excluded_targets = d.getVar('SDK_TARGETS', True)
     lockedsigs_pruned = baseoutpath + '/conf/locked-sigs.inc'
     oe.copy_buildsystem.prune_lockedsigs(allowed_tasks,
-                                         excluded_targets,
+                                         excluded_targets.split(),
                                          sigfile,
                                          lockedsigs_pruned)
 
-- 
2.5.0

[PATCH 2/3] copy_buildsystem.py: Allow substring matches for excluded_targets

[Randy Witt]

This is so that the excluded_targets passed to prune_lockedsigs can
take a partial form such as nativesdk to filter based on partial matches.

Signed-off-by: Randy Witt <randy.e.witt@linux.intel.com>
---
 meta/lib/oe/copy_buildsystem.py | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)

diff --git a/meta/lib/oe/copy_buildsystem.py b/meta/lib/oe/copy_buildsystem.py
index 15af4eb..83ab8c5 100644
--- a/meta/lib/oe/copy_buildsystem.py
+++ b/meta/lib/oe/copy_buildsystem.py
@@ -83,8 +83,11 @@ def prune_lockedsigs(allowed_tasks, excluded_targets, lockedsigs, pruned_output)
             for line in infile:
                 if invalue:
                     if line.endswith('\\\n'):
-                        splitval = line.strip().split(':')
-                        if splitval[1] in allowed_tasks and not splitval[0] in excluded_targets:
+                        target, task = line.strip().split(':')[:2]
+                        badmatch = ([x for x in excluded_targets
+                                     if x in target])
+
+                        if task in allowed_tasks and not badmatch:
                             f.write(line)
                     else:
                         f.write(line)
-- 
2.5.0

[PATCH 3/3] populate_sdk_ext: Don't put nativesdk and crosssdk items in ext sdk

[Randy Witt]

The nativesdk and crosssdk targets were only getting pulled into the
extensible sdk due to buildtools dependencies being in BB_TASKDEPDATA.
However the nativesdk and crosssdk targets are not needed for the
extensible sdk to be used.

This patch therefore removes them since it reduces the size of the
extensible sdks that include sstate by ~300MB.

Signed-off-by: Randy Witt <randy.e.witt@linux.intel.com>
---
 meta/classes/populate_sdk_ext.bbclass | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/meta/classes/populate_sdk_ext.bbclass b/meta/classes/populate_sdk_ext.bbclass
index 5c99bd0..ef014b8 100644
--- a/meta/classes/populate_sdk_ext.bbclass
+++ b/meta/classes/populate_sdk_ext.bbclass
@@ -173,6 +173,12 @@ python copy_buildsystem () {
     # Filter the locked signatures file to just the sstate tasks we are interested in
     allowed_tasks = ['do_populate_lic', 'do_populate_sysroot', 'do_packagedata', 'do_package_write_ipk', 'do_package_write_rpm', 'do_package_write_deb', 'do_package_qa', 'do_deploy']
     excluded_targets = d.getVar('SDK_TARGETS', True)
+
+    # Since buildtools is included in the sdk, the nativesdk items and
+    # crosssdk end up in the tasks in locked-sigs.inc. However, the nativesdk
+    # and crosssdk items aren't needed for the extensible sdk to function so
+    # remove them from the list.
+    excluded_targets += 'nativesdk crosssdk'
     lockedsigs_pruned = baseoutpath + '/conf/locked-sigs.inc'
     oe.copy_buildsystem.prune_lockedsigs(allowed_tasks,
                                          excluded_targets.split(),
-- 
2.5.0

Re: [PATCH 3/3] populate_sdk_ext: Don't put nativesdk and crosssdk items in ext sdk

[Paul Eggleton]

On Thu, 07 Jan 2016 11:24:27 Randy Witt wrote:
> The nativesdk and crosssdk targets were only getting pulled into the
> extensible sdk due to buildtools dependencies being in BB_TASKDEPDATA.
> However the nativesdk and crosssdk targets are not needed for the
> extensible sdk to be used.
> 
> This patch therefore removes them since it reduces the size of the
> extensible sdks that include sstate by ~300MB.
> 
> Signed-off-by: Randy Witt <randy.e.witt@linux.intel.com>
> ---
>  meta/classes/populate_sdk_ext.bbclass | 6 ++++++
>  1 file changed, 6 insertions(+)
> 
> diff --git a/meta/classes/populate_sdk_ext.bbclass
> b/meta/classes/populate_sdk_ext.bbclass index 5c99bd0..ef014b8 100644
> --- a/meta/classes/populate_sdk_ext.bbclass
> +++ b/meta/classes/populate_sdk_ext.bbclass
> @@ -173,6 +173,12 @@ python copy_buildsystem () {
>      # Filter the locked signatures file to just the sstate tasks we are
> interested in allowed_tasks = ['do_populate_lic', 'do_populate_sysroot',
> 'do_packagedata', 'do_package_write_ipk', 'do_package_write_rpm',
> 'do_package_write_deb', 'do_package_qa', 'do_deploy'] excluded_targets =
> d.getVar('SDK_TARGETS', True)
> +
> +    # Since buildtools is included in the sdk, the nativesdk items and
> +    # crosssdk end up in the tasks in locked-sigs.inc. However, the
> nativesdk 
> +    # and crosssdk items aren't needed for the extensible sdk to
> function so 
> +    # remove them from the list.
> +    excluded_targets += 'nativesdk crosssdk'

You need a leading space here to make this work (or alternatively split the 
excluded_targets value in the previous patch when you first set it, and add the 
items as list items here).

Cheers,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre

Re: [PATCH 3/3] populate_sdk_ext: Don't put nativesdk and crosssdk items in ext sdk

[Paul Eggleton]

On Mon, 11 Jan 2016 11:19:37 Paul Eggleton wrote:
> On Thu, 07 Jan 2016 11:24:27 Randy Witt wrote:
> > The nativesdk and crosssdk targets were only getting pulled into the
> > extensible sdk due to buildtools dependencies being in BB_TASKDEPDATA.
> > However the nativesdk and crosssdk targets are not needed for the
> > extensible sdk to be used.
> > 
> > This patch therefore removes them since it reduces the size of the
> > extensible sdks that include sstate by ~300MB.
> > 
> > Signed-off-by: Randy Witt <randy.e.witt@linux.intel.com>
> > ---
> > 
> >  meta/classes/populate_sdk_ext.bbclass | 6 ++++++
> >  1 file changed, 6 insertions(+)
> > 
> > diff --git a/meta/classes/populate_sdk_ext.bbclass
> > b/meta/classes/populate_sdk_ext.bbclass index 5c99bd0..ef014b8 100644
> > --- a/meta/classes/populate_sdk_ext.bbclass
> > +++ b/meta/classes/populate_sdk_ext.bbclass
> > @@ -173,6 +173,12 @@ python copy_buildsystem () {
> > 
> >      # Filter the locked signatures file to just the sstate tasks we are
> > 
> > interested in allowed_tasks = ['do_populate_lic', 'do_populate_sysroot',
> > 'do_packagedata', 'do_package_write_ipk', 'do_package_write_rpm',
> > 'do_package_write_deb', 'do_package_qa', 'do_deploy'] excluded_targets =
> > d.getVar('SDK_TARGETS', True)
> > +
> > +    # Since buildtools is included in the sdk, the nativesdk items and
> > +    # crosssdk end up in the tasks in locked-sigs.inc. However, the
> > nativesdk
> > +    # and crosssdk items aren't needed for the extensible sdk to
> > function so
> > +    # remove them from the list.
> > +    excluded_targets += 'nativesdk crosssdk'
> 
> You need a leading space here to make this work (or alternatively split the
> excluded_targets value in the previous patch when you first set it, and add
> the items as list items here).

Actually I think I have an alternative fix which we need to do anyway - create 
a separate task for the recursive dependencies, and then the issue goes away. 
I'll send out some patches to do this once I've finished testing.

Cheers,
Paul


-- 

Paul Eggleton
Intel Open Source Technology Centre