[PATCH 0/5] devtool improvements

[PATCH 0/5] devtool improvements

[Paul Eggleton]

I hope people aren't getting too bored with the steady stream of
devtool / extensible SDK patches...


The following changes since commit fc4209baa098caebf9c4cb75f9a6f2e85f43333c:

  Revert "xz: Allow to work with ASSUME_PROVIDED xz-native" (2016-01-25 10:08:25 +0000)

are available in the git repository at:

  git://git.openembedded.org/openembedded-core-contrib paule/devtool12
  http://cgit.openembedded.org/cgit.cgi/openembedded-core-contrib/log/?h=paule/devtool12

Paul Eggleton (5):
  devtool: move edit-recipe to a separate module
  devtool: add configure-help subcommand
  devtool: build-image: allow specifying packages to add to image
  devtool: add: warn if modified recipe found in attic directory
  devtool: properly handle bb.build.FuncFailed when extracting source

 scripts/lib/devtool/__init__.py    |   9 +-
 scripts/lib/devtool/build-image.py |  11 +-
 scripts/lib/devtool/standard.py    |  44 ++-----
 scripts/lib/devtool/utilcmds.py    | 240 +++++++++++++++++++++++++++++++++++++
 4 files changed, 261 insertions(+), 43 deletions(-)
 create mode 100644 scripts/lib/devtool/utilcmds.py

-- 
2.5.0

[PATCH 1/5] devtool: move edit-recipe to a separate module

[Paul Eggleton]

standard.py is getting a bit large; move the "utility" commands to
another module.

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 scripts/lib/devtool/standard.py | 36 ---------------------
 scripts/lib/devtool/utilcmds.py | 70 +++++++++++++++++++++++++++++++++++++++++
 2 files changed, 70 insertions(+), 36 deletions(-)
 create mode 100644 scripts/lib/devtool/utilcmds.py

diff --git a/scripts/lib/devtool/standard.py b/scripts/lib/devtool/standard.py
index f19de27..83ec7d8 100644
--- a/scripts/lib/devtool/standard.py
+++ b/scripts/lib/devtool/standard.py
@@ -1258,36 +1258,6 @@ def reset(args, config, basepath, workspace):
     return 0
 
 
-def edit_recipe(args, config, basepath, workspace):
-    """Entry point for the devtool 'edit-recipe' subcommand"""
-    if args.any_recipe:
-        tinfoil = setup_tinfoil(config_only=False, basepath=basepath)
-        try:
-            rd = parse_recipe(config, tinfoil, args.recipename, True)
-            if not rd:
-                return 1
-            recipefile = rd.getVar('FILE', True)
-        finally:
-            tinfoil.shutdown()
-    else:
-        check_workspace_recipe(workspace, args.recipename)
-        recipefile = workspace[args.recipename]['recipefile']
-        if not recipefile:
-            raise DevtoolError("Recipe file for %s is not under the workspace" %
-                               args.recipename)
-
-    editor = os.environ.get('EDITOR', None)
-    if not editor:
-        raise DevtoolError("EDITOR environment variable not set")
-
-    import subprocess
-    try:
-        subprocess.check_call('%s "%s"' % (editor, recipefile), shell=True)
-    except subprocess.CalledProcessError as e:
-        return e.returncode
-
-    return 0
-
 def get_default_srctree(config, recipename=''):
     """Get the default srctree path"""
     srctreeparent = config.get('General', 'default_source_parent_dir', config.workspace_path)
@@ -1365,9 +1335,3 @@ def register_commands(subparsers, context):
     parser_reset.add_argument('--all', '-a', action="store_true", help='Reset all recipes (clear workspace)')
     parser_reset.add_argument('--no-clean', '-n', action="store_true", help='Don\'t clean the sysroot to remove recipe output')
     parser_reset.set_defaults(func=reset)
-
-    parser_edit_recipe = subparsers.add_parser('edit-recipe', help='Edit a recipe file in your workspace',
-                                         description='Runs the default editor (as specified by the EDITOR variable) on the specified recipe. Note that the recipe file itself must be in the workspace (i.e. as a result of "devtool add" or "devtool upgrade"); you can override this with the -a/--any-recipe option.')
-    parser_edit_recipe.add_argument('recipename', help='Recipe to edit')
-    parser_edit_recipe.add_argument('--any-recipe', '-a', action="store_true", help='Edit any recipe, not just where the recipe file itself is in the workspace')
-    parser_edit_recipe.set_defaults(func=edit_recipe)
diff --git a/scripts/lib/devtool/utilcmds.py b/scripts/lib/devtool/utilcmds.py
new file mode 100644
index 0000000..375d7a3
--- /dev/null
+++ b/scripts/lib/devtool/utilcmds.py
@@ -0,0 +1,70 @@
+# Development tool - utility commands plugin
+#
+# Copyright (C) 2015-2016 Intel Corporation
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License version 2 as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License along
+# with this program; if not, write to the Free Software Foundation, Inc.,
+# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
+
+"""Devtool utility plugins"""
+
+import os
+import sys
+import shutil
+import tempfile
+import logging
+import argparse
+import subprocess
+from devtool import exec_build_env_command, setup_tinfoil, check_workspace_recipe, DevtoolError
+from devtool import parse_recipe
+
+logger = logging.getLogger('devtool')
+
+
+def edit_recipe(args, config, basepath, workspace):
+    """Entry point for the devtool 'edit-recipe' subcommand"""
+    if args.any_recipe:
+        tinfoil = setup_tinfoil(config_only=False, basepath=basepath)
+        try:
+            rd = parse_recipe(config, tinfoil, args.recipename, True)
+            if not rd:
+                return 1
+            recipefile = rd.getVar('FILE', True)
+        finally:
+            tinfoil.shutdown()
+    else:
+        check_workspace_recipe(workspace, args.recipename)
+        recipefile = workspace[args.recipename]['recipefile']
+        if not recipefile:
+            raise DevtoolError("Recipe file for %s is not under the workspace" %
+                               args.recipename)
+
+    editor = os.environ.get('EDITOR', None)
+    if not editor:
+        raise DevtoolError("EDITOR environment variable not set")
+
+    import subprocess
+    try:
+        subprocess.check_call('%s "%s"' % (editor, recipefile), shell=True)
+    except subprocess.CalledProcessError as e:
+        return e.returncode
+
+    return 0
+
+
+def register_commands(subparsers, context):
+    """Register devtool subcommands from this plugin"""
+    parser_edit_recipe = subparsers.add_parser('edit-recipe', help='Edit a recipe file in your workspace',
+                                         description='Runs the default editor (as specified by the EDITOR variable) on the specified recipe. Note that the recipe file itself must be in the workspace (i.e. as a result of "devtool add" or "devtool upgrade"); you can override this with the -a/--any-recipe option.')
+    parser_edit_recipe.add_argument('recipename', help='Recipe to edit')
+    parser_edit_recipe.add_argument('--any-recipe', '-a', action="store_true", help='Edit any recipe, not just where the recipe file itself is in the workspace')
+    parser_edit_recipe.set_defaults(func=edit_recipe)
-- 
2.5.0

[PATCH 2/5] devtool: add configure-help subcommand

[Paul Eggleton]

When you need to set EXTRA_OECONF for a recipe, you need to know what
options the configure script actually supports; the configure script
however is only accessible from within a devshell and (at least in the
case of autotooled software fetched from an SCM repository) may not
actually exist until do_configure has run. Thus, provide a "devtool
configure-help" subcommand that runs the configure script for a recipe
with --help and shows you the output through a pager (e.g. less),
prefaced by a header describing the current options being specified.

There is basic support for autotools, cmake and bare configure scripts.
The cmake support is a little hacky since cmake doesn't really have a
concise help option that lists user-defined knobs (without actually
running through the configure process), however that being a design
feature of cmake there's not much I can think of to do about that at
the moment.

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 scripts/lib/devtool/__init__.py |   9 ++-
 scripts/lib/devtool/utilcmds.py | 170 ++++++++++++++++++++++++++++++++++++++++
 2 files changed, 175 insertions(+), 4 deletions(-)

diff --git a/scripts/lib/devtool/__init__.py b/scripts/lib/devtool/__init__.py
index 0405d22..ff97dfc 100644
--- a/scripts/lib/devtool/__init__.py
+++ b/scripts/lib/devtool/__init__.py
@@ -129,7 +129,7 @@ def get_recipe_file(cooker, pn):
             logger.error("Unable to find any recipe file matching %s" % pn)
     return recipefile
 
-def parse_recipe(config, tinfoil, pn, appends):
+def parse_recipe(config, tinfoil, pn, appends, filter_workspace=True):
     """Parse recipe of a package"""
     import oe.recipeutils
     recipefile = get_recipe_file(tinfoil.cooker, pn)
@@ -138,9 +138,10 @@ def parse_recipe(config, tinfoil, pn, appends):
         return None
     if appends:
         append_files = tinfoil.cooker.collection.get_file_appends(recipefile)
-        # Filter out appends from the workspace
-        append_files = [path for path in append_files if
-                        not path.startswith(config.workspace_path)]
+        if filter_workspace:
+            # Filter out appends from the workspace
+            append_files = [path for path in append_files if
+                            not path.startswith(config.workspace_path)]
     else:
         append_files = None
     return oe.recipeutils.parse_recipe(recipefile, append_files,
diff --git a/scripts/lib/devtool/utilcmds.py b/scripts/lib/devtool/utilcmds.py
index 375d7a3..a8f5e97 100644
--- a/scripts/lib/devtool/utilcmds.py
+++ b/scripts/lib/devtool/utilcmds.py
@@ -61,6 +61,165 @@ def edit_recipe(args, config, basepath, workspace):
     return 0
 
 
+def configure_help(args, config, basepath, workspace):
+    """Entry point for the devtool 'configure-help' subcommand"""
+    import oe.utils
+
+    check_workspace_recipe(workspace, args.recipename)
+    tinfoil = setup_tinfoil(config_only=False, basepath=basepath)
+    try:
+        rd = parse_recipe(config, tinfoil, args.recipename, appends=True, filter_workspace=False)
+        if not rd:
+            return 1
+        b = rd.getVar('B', True)
+        s = rd.getVar('S', True)
+        configurescript = os.path.join(s, 'configure')
+        confdisabled = 'noexec' in rd.getVarFlags('do_configure') or 'do_configure' not in (rd.getVar('__BBTASKS', False) or [])
+        configureopts = oe.utils.squashspaces(rd.getVar('CONFIGUREOPTS', True) or '')
+        extra_oeconf = oe.utils.squashspaces(rd.getVar('EXTRA_OECONF', True) or '')
+        extra_oecmake = oe.utils.squashspaces(rd.getVar('EXTRA_OECMAKE', True) or '')
+        do_configure = rd.getVar('do_configure', True) or ''
+        do_configure_noexpand = rd.getVar('do_configure', False) or ''
+        packageconfig = rd.getVarFlags('PACKAGECONFIG') or []
+        autotools = bb.data.inherits_class('autotools', rd) and ('oe_runconf' in do_configure or 'autotools_do_configure' in do_configure)
+        cmake = bb.data.inherits_class('cmake', rd) and ('cmake_do_configure' in do_configure)
+        cmake_do_configure = rd.getVar('cmake_do_configure', True)
+        pn = rd.getVar('PN', True)
+    finally:
+        tinfoil.shutdown()
+
+    if 'doc' in packageconfig:
+        del packageconfig['doc']
+
+    if autotools and not os.path.exists(configurescript):
+        logger.info('Running do_configure to generate configure script')
+        try:
+            stdout, _ = exec_build_env_command(config.init_path, basepath,
+                                               'bitbake -c configure %s' % msg, args.recipename,
+                                               stderr=subprocess.STDOUT)
+        except bb.process.ExecutionError:
+            pass
+
+    if confdisabled or do_configure.strip() in ('', ':'):
+        raise DevtoolError("do_configure task has been disabled for this recipe")
+    elif args.no_pager and not os.path.exists(configurescript):
+        raise DevtoolError("No configure script found and no other information to display")
+    else:
+        configopttext = ''
+        if autotools and configureopts:
+            configopttext = '''
+Arguments currently passed to the configure script:
+
+%s
+
+Some of those are fixed.''' % (configureopts + ' ' + extra_oeconf)
+            if extra_oeconf:
+                configopttext += ''' The ones that are specified through EXTRA_OECONF (which you can change or add to easily):
+
+%s''' % extra_oeconf
+
+        elif cmake:
+            in_cmake = False
+            cmake_cmd = ''
+            for line in cmake_do_configure.splitlines():
+                if in_cmake:
+                    cmake_cmd = cmake_cmd + ' ' + line.strip().rstrip('\\')
+                    if not line.endswith('\\'):
+                        break
+                if line.lstrip().startswith('cmake '):
+                    cmake_cmd = line.strip().rstrip('\\')
+                    if line.endswith('\\'):
+                        in_cmake = True
+                    else:
+                        break
+            if cmake_cmd:
+                configopttext = '''
+The current cmake command line:
+
+%s
+
+Arguments specified through EXTRA_OECMAKE (which you can change or add to easily)
+
+%s''' % (oe.utils.squashspaces(cmake_cmd), extra_oecmake)
+            else:
+                configopttext = '''
+The current implementation of cmake_do_configure:
+
+cmake_do_configure() {
+%s
+}
+
+Arguments specified through EXTRA_OECMAKE (which you can change or add to easily)
+
+%s''' % (cmake_do_configure.rstrip(), extra_oecmake)
+
+        elif do_configure:
+            configopttext = '''
+The current implementation of do_configure:
+
+do_configure() {
+%s
+}''' % do_configure.rstrip()
+            if '${EXTRA_OECONF}' in do_configure_noexpand:
+                configopttext += '''
+
+Arguments specified through EXTRA_OECONF (which you can change or add to easily):
+
+%s''' % extra_oeconf
+
+        if packageconfig:
+            configopttext += '''
+
+Some of these options may be controlled through PACKAGECONFIG; for more details please see the recipe.'''
+
+        if args.arg:
+            helpargs = ' '.join(args.arg)
+        elif cmake:
+            helpargs = '-LH'
+        else:
+            helpargs = '--help'
+
+        msg = '''configure information for %s
+------------------------------------------
+%s''' % (pn, configopttext)
+
+        if cmake:
+            msg += '''
+
+The cmake %s output for %s follows. After "-- Cache values" you should see a list of variables you can add to EXTRA_OECMAKE (prefixed with -D and suffixed with = followed by the desired value, without any spaces).
+------------------------------------------''' % (helpargs, pn)
+        elif os.path.exists(configurescript):
+            msg += '''
+
+The ./configure %s output for %s follows.
+------------------------------------------''' % (helpargs, pn)
+
+        olddir = os.getcwd()
+        tmppath = tempfile.mkdtemp()
+        with tempfile.NamedTemporaryFile('w', delete=False) as tf:
+            if not args.no_header:
+                tf.write(msg + '\n')
+            tf.close()
+            try:
+                try:
+                    cmd = 'cat %s' % tf.name
+                    if cmake:
+                        cmd += '; cmake %s %s 2>&1' % (helpargs, s)
+                        os.chdir(b)
+                    elif os.path.exists(configurescript):
+                        cmd += '; %s %s' % (configurescript, helpargs)
+                    if sys.stdout.isatty() and not args.no_pager:
+                        pager = os.environ.get('PAGER', 'less')
+                        cmd = '(%s) | %s' % (cmd, pager)
+                    subprocess.check_call(cmd, shell=True)
+                except subprocess.CalledProcessError as e:
+                    return e.returncode
+            finally:
+                os.chdir(olddir)
+                shutil.rmtree(tmppath)
+                os.remove(tf.name)
+
+
 def register_commands(subparsers, context):
     """Register devtool subcommands from this plugin"""
     parser_edit_recipe = subparsers.add_parser('edit-recipe', help='Edit a recipe file in your workspace',
@@ -68,3 +227,14 @@ def register_commands(subparsers, context):
     parser_edit_recipe.add_argument('recipename', help='Recipe to edit')
     parser_edit_recipe.add_argument('--any-recipe', '-a', action="store_true", help='Edit any recipe, not just where the recipe file itself is in the workspace')
     parser_edit_recipe.set_defaults(func=edit_recipe)
+
+    # NOTE: Needed to override the usage string here since the default
+    # gets the order wrong - recipename must come before --arg
+    parser_configure_help = subparsers.add_parser('configure-help', help='Get help on configure script options',
+                                         usage='devtool configure-help [options] recipename [--arg ...]',
+                                         description='Displays the help for the configure script for the specified recipe (i.e. runs ./configure --help) prefaced by a header describing the current options being specified. Output is piped through less (or whatever PAGER is set to, if set) for easy browsing.')
+    parser_configure_help.add_argument('recipename', help='Recipe to show configure help for')
+    parser_configure_help.add_argument('-p', '--no-pager', help='Disable paged output', action="store_true")
+    parser_configure_help.add_argument('-n', '--no-header', help='Disable explanatory header text', action="store_true")
+    parser_configure_help.add_argument('--arg', help='Pass remaining arguments to the configure script instead of --help (useful if the script has additional help options)', nargs=argparse.REMAINDER)
+    parser_configure_help.set_defaults(func=configure_help)
-- 
2.5.0

[PATCH 3/5] devtool: build-image: allow specifying packages to add to image

[Paul Eggleton]

Provide an option to devtool build-image to specify the list of packages
instead of taking the list of packages produced by recipes in the
workspace. Sometimes you don't want all of these packages; other times
you want to add more.

This is the most immediate fix for [YOCTO #8855], though it is a little
crude so I would like to provide better means of customising the image
contents later.

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 scripts/lib/devtool/build-image.py | 11 +++++++++--
 1 file changed, 9 insertions(+), 2 deletions(-)

diff --git a/scripts/lib/devtool/build-image.py b/scripts/lib/devtool/build-image.py
index e53239d..48c3a11 100644
--- a/scripts/lib/devtool/build-image.py
+++ b/scripts/lib/devtool/build-image.py
@@ -71,8 +71,11 @@ def build_image(args, config, basepath, workspace):
             raise DevtoolError('Specified recipe %s is not an image recipe' % image)
 
     try:
-        if workspace:
-            packages = _get_packages(tinfoil, workspace, config)
+        if workspace or args.add_packages:
+            if args.add_packages:
+                packages = args.add_packages.split(',')
+            else:
+                packages = _get_packages(tinfoil, workspace, config)
             if packages:
                 with open(appendfile, 'w') as afile:
                     # include packages from workspace recipes into the image
@@ -108,4 +111,8 @@ def register_commands(subparsers, context):
                                    description='Builds an image, extending it to include '
                                    'packages from recipes in the workspace')
     parser.add_argument('imagename', help='Image recipe to build', nargs='?')
+    parser.add_argument('-p', '--add-packages', help='Instead of adding packages for the '
+                        'entire workspace, specify packages to be added to the image '
+                        '(separate multiple packages by commas)',
+                        metavar='PACKAGES')
     parser.set_defaults(func=build_image)
-- 
2.5.0

[PATCH 4/5] devtool: add: warn if modified recipe found in attic directory

[Paul Eggleton]

If a recipe generated by "devtool add" has been modified since then when
you run "devtool reset", it will be moved into the "attic" subdirectory
of the workspace in case those modifications need to be preserved. It
seems natural that if those modifications were worth preserving we
should warn the user if such a file exists when they run "devtool add"
to create the same recipe again, so they can pick up where they left off
if they want to.

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 scripts/lib/devtool/standard.py | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/scripts/lib/devtool/standard.py b/scripts/lib/devtool/standard.py
index 83ec7d8..187dff2 100644
--- a/scripts/lib/devtool/standard.py
+++ b/scripts/lib/devtool/standard.py
@@ -190,6 +190,9 @@ def add(args, config, basepath, workspace):
             shutil.move(recipes[0], recipefile)
         else:
             raise DevtoolError('Command \'%s\' did not create any recipe file:\n%s' % (e.command, e.stdout))
+        attic_recipe = os.path.join(config.workspace_path, 'attic', os.path.basename(recipefile))
+        if os.path.exists(attic_recipe):
+            logger.warn('A modified recipe from a previous invocation exists in %s - you may wish to move this over the top of the new recipe if you had changes in it that you want to continue with' % attic_recipe)
     finally:
         if tmpsrcdir and os.path.exists(tmpsrcdir):
             shutil.rmtree(tmpsrcdir)
-- 
2.5.0

[PATCH 5/5] devtool: properly handle bb.build.FuncFailed when extracting source

[Paul Eggleton]

When we run the tasks required to extract the source for a recipe (e.g.
within "devtool modify" or "devtool extract") if one of those tasks
fails you get a bb.build.FuncFailed exception; handle this properly so
you don't see a traceback.

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 scripts/lib/devtool/standard.py | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/scripts/lib/devtool/standard.py b/scripts/lib/devtool/standard.py
index 187dff2..18daf84 100644
--- a/scripts/lib/devtool/standard.py
+++ b/scripts/lib/devtool/standard.py
@@ -386,7 +386,10 @@ class BbTaskExecutor(object):
                 logger.info('Executing %s...' % func)
             fn = self.rdata.getVar('FILE', True)
             localdata = bb.build._task_data(fn, func, self.rdata)
-            bb.build.exec_func(func, localdata)
+            try:
+                bb.build.exec_func(func, localdata)
+            except bb.build.FuncFailed as e:
+                raise DevtoolError(str(e))
             self.executed.append(func)
 
 
-- 
2.5.0

Re: [PATCH 2/5] devtool: add configure-help subcommand

[Khem Raj]

[-- Attachment #1: Type: text/plain, Size: 11936 bytes --]


> On Jan 25, 2016, at 9:53 PM, Paul Eggleton <paul.eggleton@linux.intel.com> wrote:
> 
> When you need to set EXTRA_OECONF for a recipe, you need to know what
> options the configure script actually supports; the configure script
> however is only accessible from within a devshell and (at least in the
> case of autotooled software fetched from an SCM repository) may not
> actually exist until do_configure has run. Thus, provide a "devtool
> configure-help" subcommand that runs the configure script for a recipe
> with --help and shows you the output through a pager (e.g. less),
> prefaced by a header describing the current options being specified.
> 
> There is basic support for autotools, cmake and bare configure scripts.
> The cmake support is a little hacky since cmake doesn't really have a
> concise help option that lists user-defined knobs (without actually
> running through the configure process), however that being a design
> feature of cmake there's not much I can think of to do about that at
> the moment.
> 

this option is autotools specific. We need to convey this precisely. may be involve autotools
in the option name or something

> Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
> ---
> scripts/lib/devtool/__init__.py |   9 ++-
> scripts/lib/devtool/utilcmds.py | 170 ++++++++++++++++++++++++++++++++++++++++
> 2 files changed, 175 insertions(+), 4 deletions(-)
> 
> diff --git a/scripts/lib/devtool/__init__.py b/scripts/lib/devtool/__init__.py
> index 0405d22..ff97dfc 100644
> --- a/scripts/lib/devtool/__init__.py
> +++ b/scripts/lib/devtool/__init__.py
> @@ -129,7 +129,7 @@ def get_recipe_file(cooker, pn):
>             logger.error("Unable to find any recipe file matching %s" % pn)
>     return recipefile
> 
> -def parse_recipe(config, tinfoil, pn, appends):
> +def parse_recipe(config, tinfoil, pn, appends, filter_workspace=True):
>     """Parse recipe of a package"""
>     import oe.recipeutils
>     recipefile = get_recipe_file(tinfoil.cooker, pn)
> @@ -138,9 +138,10 @@ def parse_recipe(config, tinfoil, pn, appends):
>         return None
>     if appends:
>         append_files = tinfoil.cooker.collection.get_file_appends(recipefile)
> -        # Filter out appends from the workspace
> -        append_files = [path for path in append_files if
> -                        not path.startswith(config.workspace_path)]
> +        if filter_workspace:
> +            # Filter out appends from the workspace
> +            append_files = [path for path in append_files if
> +                            not path.startswith(config.workspace_path)]
>     else:
>         append_files = None
>     return oe.recipeutils.parse_recipe(recipefile, append_files,
> diff --git a/scripts/lib/devtool/utilcmds.py b/scripts/lib/devtool/utilcmds.py
> index 375d7a3..a8f5e97 100644
> --- a/scripts/lib/devtool/utilcmds.py
> +++ b/scripts/lib/devtool/utilcmds.py
> @@ -61,6 +61,165 @@ def edit_recipe(args, config, basepath, workspace):
>     return 0
> 
> 
> +def configure_help(args, config, basepath, workspace):
> +    """Entry point for the devtool 'configure-help' subcommand"""
> +    import oe.utils
> +
> +    check_workspace_recipe(workspace, args.recipename)
> +    tinfoil = setup_tinfoil(config_only=False, basepath=basepath)
> +    try:
> +        rd = parse_recipe(config, tinfoil, args.recipename, appends=True, filter_workspace=False)
> +        if not rd:
> +            return 1
> +        b = rd.getVar('B', True)
> +        s = rd.getVar('S', True)
> +        configurescript = os.path.join(s, 'configure')
> +        confdisabled = 'noexec' in rd.getVarFlags('do_configure') or 'do_configure' not in (rd.getVar('__BBTASKS', False) or [])
> +        configureopts = oe.utils.squashspaces(rd.getVar('CONFIGUREOPTS', True) or '')
> +        extra_oeconf = oe.utils.squashspaces(rd.getVar('EXTRA_OECONF', True) or '')
> +        extra_oecmake = oe.utils.squashspaces(rd.getVar('EXTRA_OECMAKE', True) or '')
> +        do_configure = rd.getVar('do_configure', True) or ''
> +        do_configure_noexpand = rd.getVar('do_configure', False) or ''
> +        packageconfig = rd.getVarFlags('PACKAGECONFIG') or []
> +        autotools = bb.data.inherits_class('autotools', rd) and ('oe_runconf' in do_configure or 'autotools_do_configure' in do_configure)
> +        cmake = bb.data.inherits_class('cmake', rd) and ('cmake_do_configure' in do_configure)
> +        cmake_do_configure = rd.getVar('cmake_do_configure', True)
> +        pn = rd.getVar('PN', True)
> +    finally:
> +        tinfoil.shutdown()
> +
> +    if 'doc' in packageconfig:
> +        del packageconfig['doc']
> +
> +    if autotools and not os.path.exists(configurescript):
> +        logger.info('Running do_configure to generate configure script')
> +        try:
> +            stdout, _ = exec_build_env_command(config.init_path, basepath,
> +                                               'bitbake -c configure %s' % msg, args.recipename,
> +                                               stderr=subprocess.STDOUT)
> +        except bb.process.ExecutionError:
> +            pass
> +
> +    if confdisabled or do_configure.strip() in ('', ':'):
> +        raise DevtoolError("do_configure task has been disabled for this recipe")
> +    elif args.no_pager and not os.path.exists(configurescript):
> +        raise DevtoolError("No configure script found and no other information to display")
> +    else:
> +        configopttext = ''
> +        if autotools and configureopts:
> +            configopttext = '''
> +Arguments currently passed to the configure script:
> +
> +%s
> +
> +Some of those are fixed.''' % (configureopts + ' ' + extra_oeconf)
> +            if extra_oeconf:
> +                configopttext += ''' The ones that are specified through EXTRA_OECONF (which you can change or add to easily):
> +
> +%s''' % extra_oeconf
> +
> +        elif cmake:
> +            in_cmake = False
> +            cmake_cmd = ''
> +            for line in cmake_do_configure.splitlines():
> +                if in_cmake:
> +                    cmake_cmd = cmake_cmd + ' ' + line.strip().rstrip('\\')
> +                    if not line.endswith('\\'):
> +                        break
> +                if line.lstrip().startswith('cmake '):
> +                    cmake_cmd = line.strip().rstrip('\\')
> +                    if line.endswith('\\'):
> +                        in_cmake = True
> +                    else:
> +                        break
> +            if cmake_cmd:
> +                configopttext = '''
> +The current cmake command line:
> +
> +%s
> +
> +Arguments specified through EXTRA_OECMAKE (which you can change or add to easily)
> +
> +%s''' % (oe.utils.squashspaces(cmake_cmd), extra_oecmake)
> +            else:
> +                configopttext = '''
> +The current implementation of cmake_do_configure:
> +
> +cmake_do_configure() {
> +%s
> +}
> +
> +Arguments specified through EXTRA_OECMAKE (which you can change or add to easily)
> +
> +%s''' % (cmake_do_configure.rstrip(), extra_oecmake)
> +
> +        elif do_configure:
> +            configopttext = '''
> +The current implementation of do_configure:
> +
> +do_configure() {
> +%s
> +}''' % do_configure.rstrip()
> +            if '${EXTRA_OECONF}' in do_configure_noexpand:
> +                configopttext += '''
> +
> +Arguments specified through EXTRA_OECONF (which you can change or add to easily):
> +
> +%s''' % extra_oeconf
> +
> +        if packageconfig:
> +            configopttext += '''
> +
> +Some of these options may be controlled through PACKAGECONFIG; for more details please see the recipe.'''
> +
> +        if args.arg:
> +            helpargs = ' '.join(args.arg)
> +        elif cmake:
> +            helpargs = '-LH'
> +        else:
> +            helpargs = '--help'
> +
> +        msg = '''configure information for %s
> +------------------------------------------
> +%s''' % (pn, configopttext)
> +
> +        if cmake:
> +            msg += '''
> +
> +The cmake %s output for %s follows. After "-- Cache values" you should see a list of variables you can add to EXTRA_OECMAKE (prefixed with -D and suffixed with = followed by the desired value, without any spaces).
> +------------------------------------------''' % (helpargs, pn)
> +        elif os.path.exists(configurescript):
> +            msg += '''
> +
> +The ./configure %s output for %s follows.
> +------------------------------------------''' % (helpargs, pn)
> +
> +        olddir = os.getcwd()
> +        tmppath = tempfile.mkdtemp()
> +        with tempfile.NamedTemporaryFile('w', delete=False) as tf:
> +            if not args.no_header:
> +                tf.write(msg + '\n')
> +            tf.close()
> +            try:
> +                try:
> +                    cmd = 'cat %s' % tf.name
> +                    if cmake:
> +                        cmd += '; cmake %s %s 2>&1' % (helpargs, s)
> +                        os.chdir(b)
> +                    elif os.path.exists(configurescript):
> +                        cmd += '; %s %s' % (configurescript, helpargs)
> +                    if sys.stdout.isatty() and not args.no_pager:
> +                        pager = os.environ.get('PAGER', 'less')
> +                        cmd = '(%s) | %s' % (cmd, pager)
> +                    subprocess.check_call(cmd, shell=True)
> +                except subprocess.CalledProcessError as e:
> +                    return e.returncode
> +            finally:
> +                os.chdir(olddir)
> +                shutil.rmtree(tmppath)
> +                os.remove(tf.name)
> +
> +
> def register_commands(subparsers, context):
>     """Register devtool subcommands from this plugin"""
>     parser_edit_recipe = subparsers.add_parser('edit-recipe', help='Edit a recipe file in your workspace',
> @@ -68,3 +227,14 @@ def register_commands(subparsers, context):
>     parser_edit_recipe.add_argument('recipename', help='Recipe to edit')
>     parser_edit_recipe.add_argument('--any-recipe', '-a', action="store_true", help='Edit any recipe, not just where the recipe file itself is in the workspace')
>     parser_edit_recipe.set_defaults(func=edit_recipe)
> +
> +    # NOTE: Needed to override the usage string here since the default
> +    # gets the order wrong - recipename must come before --arg
> +    parser_configure_help = subparsers.add_parser('configure-help', help='Get help on configure script options',
> +                                         usage='devtool configure-help [options] recipename [--arg ...]',
> +                                         description='Displays the help for the configure script for the specified recipe (i.e. runs ./configure --help) prefaced by a header describing the current options being specified. Output is piped through less (or whatever PAGER is set to, if set) for easy browsing.')
> +    parser_configure_help.add_argument('recipename', help='Recipe to show configure help for')
> +    parser_configure_help.add_argument('-p', '--no-pager', help='Disable paged output', action="store_true")
> +    parser_configure_help.add_argument('-n', '--no-header', help='Disable explanatory header text', action="store_true")
> +    parser_configure_help.add_argument('--arg', help='Pass remaining arguments to the configure script instead of --help (useful if the script has additional help options)', nargs=argparse.REMAINDER)
> +    parser_configure_help.set_defaults(func=configure_help)
> --
> 2.5.0
> 
> --
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-core


[-- Attachment #2: Message signed with OpenPGP using GPGMail --]
[-- Type: application/pgp-signature, Size: 211 bytes --]

Re: [PATCH 2/5] devtool: add configure-help subcommand

[Paul Eggleton]

Hi Khem,

On Mon, 25 Jan 2016 22:14:09 Khem Raj wrote:
> > On Jan 25, 2016, at 9:53 PM, Paul Eggleton <paul.eggleton@linux.intel.com>
> > wrote:
> > 
> > When you need to set EXTRA_OECONF for a recipe, you need to know what
> > options the configure script actually supports; the configure script
> > however is only accessible from within a devshell and (at least in the
> > case of autotooled software fetched from an SCM repository) may not
> > actually exist until do_configure has run. Thus, provide a "devtool
> > configure-help" subcommand that runs the configure script for a recipe
> > with --help and shows you the output through a pager (e.g. less),
> > prefaced by a header describing the current options being specified.
> > 
> > There is basic support for autotools, cmake and bare configure scripts.
> > The cmake support is a little hacky since cmake doesn't really have a
> > concise help option that lists user-defined knobs (without actually
> > running through the configure process), however that being a design
> > feature of cmake there's not much I can think of to do about that at
> > the moment.
> 
> this option is autotools specific. We need to convey this precisely. may be
> involve autotools in the option name or something

Whilst not all non-autoconf configure scripts provide a --help option, some do. 
ffmpeg/libav is one example (and one where you really do need to see the output 
since there are a lot of options...)

Cheers,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre

Re: [PATCH 2/5] devtool: add configure-help subcommand

[Paul Eggleton]

On Tue, 26 Jan 2016 16:17:39 Paul Eggleton wrote:
> Hi Khem,
> 
> On Mon, 25 Jan 2016 22:14:09 Khem Raj wrote:
> > > On Jan 25, 2016, at 9:53 PM, Paul Eggleton
> > > <paul.eggleton@linux.intel.com>
> > > wrote:
> > > 
> > > When you need to set EXTRA_OECONF for a recipe, you need to know what
> > > options the configure script actually supports; the configure script
> > > however is only accessible from within a devshell and (at least in the
> > > case of autotooled software fetched from an SCM repository) may not
> > > actually exist until do_configure has run. Thus, provide a "devtool
> > > configure-help" subcommand that runs the configure script for a recipe
> > > with --help and shows you the output through a pager (e.g. less),
> > > prefaced by a header describing the current options being specified.
> > > 
> > > There is basic support for autotools, cmake and bare configure scripts.
> > > The cmake support is a little hacky since cmake doesn't really have a
> > > concise help option that lists user-defined knobs (without actually
> > > running through the configure process), however that being a design
> > > feature of cmake there's not much I can think of to do about that at
> > > the moment.
> > 
> > this option is autotools specific. We need to convey this precisely. may
> > be
> > involve autotools in the option name or something
> 
> Whilst not all non-autoconf configure scripts provide a --help option, some
> do. ffmpeg/libav is one example (and one where you really do need to see
> the output since there are a lot of options...)

I meant to add, even if configure --help doesn't work (or there is no configure 
script at all) the header this subcommand prints will work with any recipe 
provided that do_configure is actually implemented and contains some actual 
commands as opposed to calls to some other function.

Cheers,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre

Re: [PATCH 2/5] devtool: add configure-help subcommand

[Khem Raj]

On Mon, Jan 25, 2016 at 10:21 PM, Paul Eggleton
<paul.eggleton@linux.intel.com> wrote:
> On Tue, 26 Jan 2016 16:17:39 Paul Eggleton wrote:
>> Hi Khem,
>>
>> On Mon, 25 Jan 2016 22:14:09 Khem Raj wrote:
>> > > On Jan 25, 2016, at 9:53 PM, Paul Eggleton
>> > > <paul.eggleton@linux.intel.com>
>> > > wrote:
>> > >
>> > > When you need to set EXTRA_OECONF for a recipe, you need to know what
>> > > options the configure script actually supports; the configure script
>> > > however is only accessible from within a devshell and (at least in the
>> > > case of autotooled software fetched from an SCM repository) may not
>> > > actually exist until do_configure has run. Thus, provide a "devtool
>> > > configure-help" subcommand that runs the configure script for a recipe
>> > > with --help and shows you the output through a pager (e.g. less),
>> > > prefaced by a header describing the current options being specified.
>> > >
>> > > There is basic support for autotools, cmake and bare configure scripts.
>> > > The cmake support is a little hacky since cmake doesn't really have a
>> > > concise help option that lists user-defined knobs (without actually
>> > > running through the configure process), however that being a design
>> > > feature of cmake there's not much I can think of to do about that at
>> > > the moment.
>> >
>> > this option is autotools specific. We need to convey this precisely. may
>> > be
>> > involve autotools in the option name or something
>>
>> Whilst not all non-autoconf configure scripts provide a --help option, some
>> do. ffmpeg/libav is one example (and one where you really do need to see
>> the output since there are a lot of options...)
>
> I meant to add, even if configure --help doesn't work (or there is no configure
> script at all) the header this subcommand prints will work with any recipe
> provided that do_configure is actually implemented and contains some actual
> commands as opposed to calls to some other function.

my worry is it being associated to do_configure, and we know that only
a subset ( although a large one) is only using autotools but not all.
So think about avoiding that confusion.
>
> Cheers,
> Paul
>
> --
>
> Paul Eggleton
> Intel Open Source Technology Centre

Re: [PATCH 2/5] devtool: add configure-help subcommand

[Paul Eggleton]

On Mon, 25 Jan 2016 22:27:27 Khem Raj wrote:
> On Mon, Jan 25, 2016 at 10:21 PM, Paul Eggleton
> 
> <paul.eggleton@linux.intel.com> wrote:
> > On Tue, 26 Jan 2016 16:17:39 Paul Eggleton wrote:
> >> Hi Khem,
> >> 
> >> On Mon, 25 Jan 2016 22:14:09 Khem Raj wrote:
> >> > > On Jan 25, 2016, at 9:53 PM, Paul Eggleton
> >> > > <paul.eggleton@linux.intel.com>
> >> > > wrote:
> >> > > 
> >> > > When you need to set EXTRA_OECONF for a recipe, you need to know what
> >> > > options the configure script actually supports; the configure script
> >> > > however is only accessible from within a devshell and (at least in
> >> > > the
> >> > > case of autotooled software fetched from an SCM repository) may not
> >> > > actually exist until do_configure has run. Thus, provide a "devtool
> >> > > configure-help" subcommand that runs the configure script for a
> >> > > recipe
> >> > > with --help and shows you the output through a pager (e.g. less),
> >> > > prefaced by a header describing the current options being specified.
> >> > > 
> >> > > There is basic support for autotools, cmake and bare configure
> >> > > scripts.
> >> > > The cmake support is a little hacky since cmake doesn't really have a
> >> > > concise help option that lists user-defined knobs (without actually
> >> > > running through the configure process), however that being a design
> >> > > feature of cmake there's not much I can think of to do about that at
> >> > > the moment.
> >> > 
> >> > this option is autotools specific. We need to convey this precisely.
> >> > may
> >> > be
> >> > involve autotools in the option name or something
> >> 
> >> Whilst not all non-autoconf configure scripts provide a --help option,
> >> some
> >> do. ffmpeg/libav is one example (and one where you really do need to see
> >> the output since there are a lot of options...)
> > 
> > I meant to add, even if configure --help doesn't work (or there is no
> > configure script at all) the header this subcommand prints will work with
> > any recipe provided that do_configure is actually implemented and
> > contains some actual commands as opposed to calls to some other function.
> 
> my worry is it being associated to do_configure, and we know that only
> a subset ( although a large one) is only using autotools but not all.
> So think about avoiding that confusion.

I'm not terribly convinced there could be much confusion. FWIW, I just looked 
through the other recipes we have in OE-Core that have a call to a configure 
script  that don't use autoconf (man, zlib, x264, v86d, ed, libpostproc); the 
configure scripts for all of these at least produce some useful output when 
called with --help. I then took a look at meta-oe (the layer), and checked the 
following recipes:

* concurrencykit - works
* ddrescue - works
* fio - works
* libvpx - works
* lzip - works
* nodejs - works
* mg - works (no options, but message is sensible)
* terminus-font - configure script is supplied without the execute bit set but
  does support --help
* iscsi-initiator-utils - configure script is in a subdir so isn't found, but
  does support --help
* netkit-rsh - works
* netkit-rpc - works
* netkit-telnet - works
* netkit-tftp - works
* netkit-rusers - works
* netkit-rwho - works
* netmap - configure script is in a subdir so isn't found, but does support 
  --help
* ntimed - configure script is dumb and takes no options at all
* tvheadend - works
* mozjs - seems to break devtool modify, but configure script does support 
  --help

So there are some minor bugs to fix, but of those recipes I tested there is a 
grand total of 1 script that doesn't support --help. Honestly I think if a 
configure script exists it's reasonable to assume that 
--help is going to print something useful; even if it doesn't there shouldn't 
be any major harm and you're still going to get a printout of the do_configure 
value in which you'll be able to see what options are currently passed.

Cheers,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre

Re: [PATCH 2/5] devtool: add configure-help subcommand

[Paul Eggleton]

On Tue, 26 Jan 2016 17:59:53 Paul Eggleton wrote:
> On Mon, 25 Jan 2016 22:27:27 Khem Raj wrote:
> > On Mon, Jan 25, 2016 at 10:21 PM, Paul Eggleton
> > 
> > <paul.eggleton@linux.intel.com> wrote:
> > > On Tue, 26 Jan 2016 16:17:39 Paul Eggleton wrote:
> > >> Hi Khem,
> > >> 
> > >> On Mon, 25 Jan 2016 22:14:09 Khem Raj wrote:
> > >> > > On Jan 25, 2016, at 9:53 PM, Paul Eggleton
> > >> > > <paul.eggleton@linux.intel.com>
> > >> > > wrote:
> > >> > > 
> > >> > > When you need to set EXTRA_OECONF for a recipe, you need to know
> > >> > > what
> > >> > > options the configure script actually supports; the configure
> > >> > > script
> > >> > > however is only accessible from within a devshell and (at least in
> > >> > > the
> > >> > > case of autotooled software fetched from an SCM repository) may not
> > >> > > actually exist until do_configure has run. Thus, provide a "devtool
> > >> > > configure-help" subcommand that runs the configure script for a
> > >> > > recipe
> > >> > > with --help and shows you the output through a pager (e.g. less),
> > >> > > prefaced by a header describing the current options being
> > >> > > specified.
> > >> > > 
> > >> > > There is basic support for autotools, cmake and bare configure
> > >> > > scripts.
> > >> > > The cmake support is a little hacky since cmake doesn't really have
> > >> > > a
> > >> > > concise help option that lists user-defined knobs (without actually
> > >> > > running through the configure process), however that being a design
> > >> > > feature of cmake there's not much I can think of to do about that
> > >> > > at
> > >> > > the moment.
> > >> > 
> > >> > this option is autotools specific. We need to convey this precisely.
> > >> > may
> > >> > be
> > >> > involve autotools in the option name or something
> > >> 
> > >> Whilst not all non-autoconf configure scripts provide a --help option,
> > >> some
> > >> do. ffmpeg/libav is one example (and one where you really do need to
> > >> see
> > >> the output since there are a lot of options...)
> > > 
> > > I meant to add, even if configure --help doesn't work (or there is no
> > > configure script at all) the header this subcommand prints will work
> > > with
> > > any recipe provided that do_configure is actually implemented and
> > > contains some actual commands as opposed to calls to some other
> > > function.
> > 
> > my worry is it being associated to do_configure, and we know that only
> > a subset ( although a large one) is only using autotools but not all.
> > So think about avoiding that confusion.
> 
> I'm not terribly convinced there could be much confusion. FWIW, I just
> looked through the other recipes we have in OE-Core that have a call to a
> configure script  that don't use autoconf (man, zlib, x264, v86d, ed,
> libpostproc); the configure scripts for all of these at least produce some
> useful output when called with --help. I then took a look at meta-oe (the
> layer)

Actually this ended up being meta-oe, meta-networking, and meta-multimedia by 
the time I was finished.

> , and checked the following recipes:
> 
> * concurrencykit - works
> * ddrescue - works
> * fio - works
> * libvpx - works
> * lzip - works
> * nodejs - works
> * mg - works (no options, but message is sensible)
> * terminus-font - configure script is supplied without the execute bit set
> but does support --help
> * iscsi-initiator-utils - configure script is in a subdir so isn't found,
> but does support --help
> * netkit-rsh - works
> * netkit-rpc - works
> * netkit-telnet - works
> * netkit-tftp - works
> * netkit-rusers - works
> * netkit-rwho - works
> * netmap - configure script is in a subdir so isn't found, but does support
>   --help
> * ntimed - configure script is dumb and takes no options at all
> * tvheadend - works
> * mozjs - seems to break devtool modify, but configure script does support
>   --help
> 
> So there are some minor bugs to fix, but of those recipes I tested there is
> a grand total of 1 script that doesn't support --help. Honestly I think if
> a configure script exists it's reasonable to assume that
> --help is going to print something useful; even if it doesn't there
> shouldn't be any major harm and you're still going to get a printout of the
> do_configure value in which you'll be able to see what options are
> currently passed.
> 
> Cheers,
> Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre