* [PATCH 00/26] devtool/recipetool improvements
@ 2015-12-22 4:02 Paul Eggleton
2015-12-22 4:02 ` [PATCH 01/26] scripts: print usage in argparse-using scripts when a command-line error occurs Paul Eggleton
` (26 more replies)
0 siblings, 27 replies; 29+ messages in thread
From: Paul Eggleton @ 2015-12-22 4:02 UTC (permalink / raw)
To: openembedded-core
Apologies for the flood of patches today, this is the last one.
There are quite a few changes here, but the common theme is making
devtool (and recipetool) a bit easier to use, as well as making
"devtool add" and "recipetool create" more effective at picking up
things from the specified source tree.
I also extended the argparse fix to show the help text on error to all
of the command-line utilities that use it.
The following changes since commit 2a1edfd9cfa16ec334c0758b47677d4fee5e79a8:
bitbake.conf: Add filename and lineno to BB_SIGNATURE_EXCLUDE_FLAGS (2015-12-22 00:01:31 +0000)
are available in the git repository at:
git://git.openembedded.org/openembedded-core-contrib paule/devtool11
http://cgit.openembedded.org/cgit.cgi/openembedded-core-contrib/log/?h=paule/devtool11
Paul Eggleton (26):
scripts: print usage in argparse-using scripts when a command-line error occurs
recipetool: create: handle https://....git URLs
recipetool: create: avoid traceback on fetch error
recipetool: create: fix do_install handling for makefile-only software
recipetool: create: minor fix for potential issue in python handling
recipetool: create: prevent attempting to unpack entire DL_DIR
recipetool: create: detect when specified URL returns a web page
recipetool: create: set up priority system for recipe handlers
recipetool: create: support extracting name and version from build scripts
recipetool: create: basic extraction of name/version from filename
devtool: edit-recipe: add new subcommand
devtool: add: tweak help text
devtool: split out function for naming bbappend
devtool: update-recipe: use correct method to get bbappend filename
scripts/lib/argparse_oe: handle intermixing of optional positional arguments
devtool: add: allow source tree to be omitted
devtool: add: figure out recipe name from recipetool
devtool: add: allow specifying URL as positional argument
devtool: modify: default source tree path
devtool: status: list recipe file within workspace if one exists
devtool: reset: print message about leaving source tree behind
devtool: sync: tweak help / messages
recipetool: create: improve autotools support
recipetool: create: fix error when extracting source to a specified directory
oe-selftest: add tests for simple devtool add / recipetool create URL case
oe-selftest: devtool: add more explicit check for ls output
meta/lib/oeqa/selftest/devtool.py | 82 +++++-
meta/lib/oeqa/selftest/recipetool.py | 21 +-
scripts/contrib/devtool-stress.py | 5 +-
scripts/devtool | 13 +-
scripts/lib/argparse_oe.py | 66 +++++
scripts/lib/devtool/__init__.py | 15 ++
scripts/lib/devtool/standard.py | 315 +++++++++++++++++------
scripts/lib/recipetool/create.py | 206 ++++++++++++---
scripts/lib/recipetool/create_buildsys.py | 296 +++++++++++++++++----
scripts/lib/recipetool/create_buildsys_python.py | 7 +-
scripts/lib/scriptutils.py | 5 +
scripts/oe-pkgdata-util | 5 +-
scripts/oe-publish-sdk | 5 +-
scripts/oe-selftest | 3 +-
scripts/recipetool | 7 +-
scripts/send-error-report | 6 +-
scripts/test-remote-image | 3 +-
17 files changed, 872 insertions(+), 188 deletions(-)
create mode 100644 scripts/lib/argparse_oe.py
--
2.5.0
^ permalink raw reply [flat|nested] 29+ messages in thread
* [PATCH 01/26] scripts: print usage in argparse-using scripts when a command-line error occurs
2015-12-22 4:02 [PATCH 00/26] devtool/recipetool improvements Paul Eggleton
@ 2015-12-22 4:02 ` Paul Eggleton
2015-12-22 4:02 ` [PATCH 02/26] recipetool: create: handle https://....git URLs Paul Eggleton
` (25 subsequent siblings)
26 siblings, 0 replies; 29+ messages in thread
From: Paul Eggleton @ 2015-12-22 4:02 UTC (permalink / raw)
To: openembedded-core
For scripts that use Python's standard argparse module to parse
command-line arguments, create a subclass which will show the usage
the usage information when a command-line parsing error occurs. The most
common case would be when the script is run with no arguments; at least
then the user immediately gets to see what arguments they might need to
pass instead of just an error message.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
scripts/contrib/devtool-stress.py | 5 +++--
scripts/devtool | 7 ++++---
scripts/lib/argparse_oe.py | 11 +++++++++++
scripts/lib/scriptutils.py | 1 +
scripts/oe-pkgdata-util | 5 +++--
scripts/oe-publish-sdk | 5 +++--
scripts/oe-selftest | 3 ++-
scripts/recipetool | 7 ++++---
scripts/send-error-report | 6 +++++-
scripts/test-remote-image | 3 ++-
10 files changed, 38 insertions(+), 15 deletions(-)
create mode 100644 scripts/lib/argparse_oe.py
diff --git a/scripts/contrib/devtool-stress.py b/scripts/contrib/devtool-stress.py
index 4b35fc9..8cf92ca 100755
--- a/scripts/contrib/devtool-stress.py
+++ b/scripts/contrib/devtool-stress.py
@@ -35,6 +35,7 @@ import fnmatch
scripts_lib_path = os.path.abspath(os.path.join(os.path.dirname(os.path.realpath(__file__)), '..', 'lib'))
sys.path.insert(0, scripts_lib_path)
import scriptutils
+import argparse_oe
logger = scriptutils.logger_create('devtool-stress')
def select_recipes(args):
@@ -204,8 +205,8 @@ def stress_modify(args):
def main():
- parser = argparse.ArgumentParser(description="devtool stress tester",
- epilog="Use %(prog)s <subcommand> --help to get help on a specific command")
+ parser = argparse_oe.ArgumentParser(description="devtool stress tester",
+ epilog="Use %(prog)s <subcommand> --help to get help on a specific command")
parser.add_argument('-d', '--debug', help='Enable debug output', action='store_true')
parser.add_argument('-r', '--resume-from', help='Resume from specified recipe', metavar='PN')
parser.add_argument('-o', '--only', help='Only test specified recipes (comma-separated without spaces, wildcards allowed)', metavar='PNLIST')
diff --git a/scripts/devtool b/scripts/devtool
index 9d3287c..955495e 100755
--- a/scripts/devtool
+++ b/scripts/devtool
@@ -37,6 +37,7 @@ lib_path = scripts_path + '/lib'
sys.path = sys.path + [lib_path]
from devtool import DevtoolError, setup_tinfoil
import scriptutils
+import argparse_oe
logger = scriptutils.logger_create('devtool')
plugins = []
@@ -185,9 +186,9 @@ def main():
break
pth = os.path.dirname(pth)
- parser = argparse.ArgumentParser(description="OpenEmbedded development tool",
- add_help=False,
- epilog="Use %(prog)s <subcommand> --help to get help on a specific command")
+ parser = argparse_oe.ArgumentParser(description="OpenEmbedded development tool",
+ add_help=False,
+ epilog="Use %(prog)s <subcommand> --help to get help on a specific command")
parser.add_argument('--basepath', help='Base directory of SDK / build directory')
parser.add_argument('--bbpath', help='Explicitly specify the BBPATH, rather than getting it from the metadata')
parser.add_argument('-d', '--debug', help='Enable debug output', action='store_true')
diff --git a/scripts/lib/argparse_oe.py b/scripts/lib/argparse_oe.py
new file mode 100644
index 0000000..c2fee6d
--- /dev/null
+++ b/scripts/lib/argparse_oe.py
@@ -0,0 +1,11 @@
+import sys
+import argparse
+
+class ArgumentParser(argparse.ArgumentParser):
+ """Our own version of argparse's ArgumentParser"""
+
+ def error(self, message):
+ sys.stderr.write('ERROR: %s\n' % message)
+ self.print_help()
+ sys.exit(2)
+
diff --git a/scripts/lib/scriptutils.py b/scripts/lib/scriptutils.py
index 3366882..4dd7ef2 100644
--- a/scripts/lib/scriptutils.py
+++ b/scripts/lib/scriptutils.py
@@ -19,6 +19,7 @@ import sys
import os
import logging
import glob
+import argparse
def logger_create(name):
logger = logging.getLogger(name)
diff --git a/scripts/oe-pkgdata-util b/scripts/oe-pkgdata-util
index afdceaa..8e22e02 100755
--- a/scripts/oe-pkgdata-util
+++ b/scripts/oe-pkgdata-util
@@ -33,6 +33,7 @@ scripts_path = os.path.dirname(os.path.realpath(__file__))
lib_path = scripts_path + '/lib'
sys.path = sys.path + [lib_path]
import scriptutils
+import argparse_oe
logger = scriptutils.logger_create('pkgdatautil')
def tinfoil_init():
@@ -417,8 +418,8 @@ def find_path(args):
def main():
- parser = argparse.ArgumentParser(description="OpenEmbedded pkgdata tool - queries the pkgdata files written out during do_package",
- epilog="Use %(prog)s <subcommand> --help to get help on a specific command")
+ parser = argparse_oe.ArgumentParser(description="OpenEmbedded pkgdata tool - queries the pkgdata files written out during do_package",
+ epilog="Use %(prog)s <subcommand> --help to get help on a specific command")
parser.add_argument('-d', '--debug', help='Enable debug output', action='store_true')
parser.add_argument('-p', '--pkgdata-dir', help='Path to pkgdata directory (determined automatically if not specified)')
subparsers = parser.add_subparsers(title='subcommands', metavar='<subcommand>')
diff --git a/scripts/oe-publish-sdk b/scripts/oe-publish-sdk
index c4c35bd..c8c79c2 100755
--- a/scripts/oe-publish-sdk
+++ b/scripts/oe-publish-sdk
@@ -24,6 +24,7 @@ scripts_path = os.path.dirname(os.path.realpath(__file__))
lib_path = scripts_path + '/lib'
sys.path = sys.path + [lib_path]
import scriptutils
+import argparse_oe
logger = scriptutils.logger_create('sdktool')
def mkdir(d):
@@ -113,8 +114,8 @@ def publish(args):
def main():
- parser = argparse.ArgumentParser(description="OpenEmbedded development tool",
- epilog="Use %(prog)s <subcommand> --help to get help on a specific command")
+ parser = argparse_oe.ArgumentParser(description="OpenEmbedded development tool",
+ epilog="Use %(prog)s <subcommand> --help to get help on a specific command")
parser.add_argument('-d', '--debug', help='Enable debug output', action='store_true')
parser.add_argument('-q', '--quiet', help='Print only errors', action='store_true')
diff --git a/scripts/oe-selftest b/scripts/oe-selftest
index bc50b2a..f989e87 100755
--- a/scripts/oe-selftest
+++ b/scripts/oe-selftest
@@ -36,6 +36,7 @@ sys.path.insert(0, os.path.dirname(os.path.realpath(__file__)) + '/lib')
import scriptpath
scriptpath.add_bitbake_lib_path()
scriptpath.add_oe_lib_path()
+import argparse_oe
import oeqa.selftest
import oeqa.utils.ftools as ftools
@@ -65,7 +66,7 @@ log = logger_create()
def get_args_parser():
description = "Script that runs unit tests agains bitbake and other Yocto related tools. The goal is to validate tools functionality and metadata integrity. Refer to https://wiki.yoctoproject.org/wiki/Oe-selftest for more information."
- parser = argparse.ArgumentParser(description=description)
+ parser = argparse_oe.ArgumentParser(description=description)
group = parser.add_mutually_exclusive_group(required=True)
group.add_argument('--run-tests', required=False, action='store', nargs='*', dest="run_tests", default=None, help='Select what tests to run (modules, classes or test methods). Format should be: <module>.<class>.<test_method>')
group.add_argument('--run-all-tests', required=False, action="store_true", dest="run_all_tests", default=False, help='Run all (unhidden) tests')
diff --git a/scripts/recipetool b/scripts/recipetool
index 791a66a..1198cc2 100755
--- a/scripts/recipetool
+++ b/scripts/recipetool
@@ -27,6 +27,7 @@ scripts_path = os.path.dirname(os.path.realpath(__file__))
lib_path = scripts_path + '/lib'
sys.path = sys.path + [lib_path]
import scriptutils
+import argparse_oe
logger = scriptutils.logger_create('recipetool')
plugins = []
@@ -45,9 +46,9 @@ def main():
logger.error("This script can only be run after initialising the build environment (e.g. by using oe-init-build-env)")
sys.exit(1)
- parser = argparse.ArgumentParser(description="OpenEmbedded recipe tool",
- add_help=False,
- epilog="Use %(prog)s <subcommand> --help to get help on a specific command")
+ parser = argparse_oe.ArgumentParser(description="OpenEmbedded recipe tool",
+ add_help=False,
+ epilog="Use %(prog)s <subcommand> --help to get help on a specific command")
parser.add_argument('-d', '--debug', help='Enable debug output', action='store_true')
parser.add_argument('-q', '--quiet', help='Print only errors', action='store_true')
parser.add_argument('--color', choices=['auto', 'always', 'never'], default='auto', help='Colorize output (where %(metavar)s is %(choices)s)', metavar='COLOR')
diff --git a/scripts/send-error-report b/scripts/send-error-report
index 1a1b965..a29feff 100755
--- a/scripts/send-error-report
+++ b/scripts/send-error-report
@@ -15,6 +15,10 @@ import subprocess
import argparse
import logging
+scripts_lib_path = os.path.join(os.path.dirname(os.path.realpath(__file__)), 'lib')
+sys.path.insert(0, scripts_lib_path)
+import argparse_oe
+
version = "0.3"
log = logging.getLogger("send-error-report")
@@ -143,7 +147,7 @@ def send_data(data, args):
if __name__ == '__main__':
- arg_parse = argparse.ArgumentParser(description="This scripts will send an error report to your specified error-report-web server.")
+ arg_parse = argparse_oe.ArgumentParser(description="This scripts will send an error report to your specified error-report-web server.")
arg_parse.add_argument("error_file",
help="Generated error report file location",
diff --git a/scripts/test-remote-image b/scripts/test-remote-image
index f3a44eb..97d03d7 100755
--- a/scripts/test-remote-image
+++ b/scripts/test-remote-image
@@ -38,6 +38,7 @@ lib_path = scripts_path + '/lib'
sys.path = sys.path + [lib_path]
import scriptpath
+import argparse_oe
# Add meta/lib to sys.path
scriptpath.add_oe_lib_path()
@@ -82,7 +83,7 @@ log = logger_create()
# Define and return the arguments parser for the script
def get_args_parser():
description = "This script is used to run automated runtime tests using remotely published image files. You should prepare the build environment just like building local images and running the tests."
- parser = argparse.ArgumentParser(description=description)
+ parser = argparse_oe.ArgumentParser(description=description)
parser.add_argument('--image-types', required=True, action="store", nargs='*', dest="image_types", default=None, help='The image types to test(ex: core-image-minimal).')
parser.add_argument('--repo-link', required=True, action="store", type=str, dest="repo_link", default=None, help='The link to the remote images repository.')
parser.add_argument('--required-packages', required=False, action="store", nargs='*', dest="required_packages", default=None, help='Required packages for the tests. They will be built before the testing begins.')
--
2.5.0
^ permalink raw reply related [flat|nested] 29+ messages in thread
* [PATCH 02/26] recipetool: create: handle https://....git URLs
2015-12-22 4:02 [PATCH 00/26] devtool/recipetool improvements Paul Eggleton
2015-12-22 4:02 ` [PATCH 01/26] scripts: print usage in argparse-using scripts when a command-line error occurs Paul Eggleton
@ 2015-12-22 4:02 ` Paul Eggleton
2015-12-22 4:02 ` [PATCH 03/26] recipetool: create: avoid traceback on fetch error Paul Eggleton
` (24 subsequent siblings)
26 siblings, 0 replies; 29+ messages in thread
From: Paul Eggleton @ 2015-12-22 4:02 UTC (permalink / raw)
To: openembedded-core
When you grab a URL for a github repository you'll almost certainly find
it in https://github.com/path/to/repository.git format; but bitbake's
fetcher can't handle that because it'll see https:// at the start and
assume it should use wget to fetch it. If the URL starts with http:// or
https:// and the path part ends with .git then assume it's a git
repository and adjust it accordingly.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
scripts/lib/recipetool/create.py | 5 +++++
1 file changed, 5 insertions(+)
diff --git a/scripts/lib/recipetool/create.py b/scripts/lib/recipetool/create.py
index 2d75046..7987fbb 100644
--- a/scripts/lib/recipetool/create.py
+++ b/scripts/lib/recipetool/create.py
@@ -108,6 +108,11 @@ def create_recipe(args):
# Assume the archive contains the directory structure verbatim
# so we need to extract to a subdirectory
fetchuri += ';subdir=%s' % os.path.splitext(os.path.basename(urlparse.urlsplit(fetchuri).path))[0]
+ git_re = re.compile('(https?)://([^;]+\.git)(;.*)?')
+ res = git_re.match(fetchuri)
+ if res:
+ # Need to switch the URI around so that the git fetcher is used
+ fetchuri = 'git://%s;protocol=%s%s' % (res.group(2), res.group(1), res.group(3) or '')
srcuri = fetchuri
rev_re = re.compile(';rev=([^;]+)')
res = rev_re.search(srcuri)
--
2.5.0
^ permalink raw reply related [flat|nested] 29+ messages in thread
* [PATCH 03/26] recipetool: create: avoid traceback on fetch error
2015-12-22 4:02 [PATCH 00/26] devtool/recipetool improvements Paul Eggleton
2015-12-22 4:02 ` [PATCH 01/26] scripts: print usage in argparse-using scripts when a command-line error occurs Paul Eggleton
2015-12-22 4:02 ` [PATCH 02/26] recipetool: create: handle https://....git URLs Paul Eggleton
@ 2015-12-22 4:02 ` Paul Eggleton
2015-12-22 4:02 ` [PATCH 04/26] recipetool: create: fix do_install handling for makefile-only software Paul Eggleton
` (23 subsequent siblings)
26 siblings, 0 replies; 29+ messages in thread
From: Paul Eggleton @ 2015-12-22 4:02 UTC (permalink / raw)
To: openembedded-core
If a fetch error occurs, the fetcher already prints a reasonable error -
we don't need the traceback as well, so catch that and exit if it
occurs.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
scripts/lib/recipetool/create.py | 6 +++++-
1 file changed, 5 insertions(+), 1 deletion(-)
diff --git a/scripts/lib/recipetool/create.py b/scripts/lib/recipetool/create.py
index 7987fbb..f98f525 100644
--- a/scripts/lib/recipetool/create.py
+++ b/scripts/lib/recipetool/create.py
@@ -122,7 +122,11 @@ def create_recipe(args):
tempsrc = tempfile.mkdtemp(prefix='recipetool-')
srctree = tempsrc
logger.info('Fetching %s...' % srcuri)
- checksums = scriptutils.fetch_uri(tinfoil.config_data, fetchuri, srctree, srcrev)
+ try:
+ checksums = scriptutils.fetch_uri(tinfoil.config_data, fetchuri, srctree, srcrev)
+ except bb.fetch2.FetchError:
+ # Error already printed
+ sys.exit(1)
dirlist = os.listdir(srctree)
if 'git.indirectionsymlink' in dirlist:
dirlist.remove('git.indirectionsymlink')
--
2.5.0
^ permalink raw reply related [flat|nested] 29+ messages in thread
* [PATCH 04/26] recipetool: create: fix do_install handling for makefile-only software
2015-12-22 4:02 [PATCH 00/26] devtool/recipetool improvements Paul Eggleton
` (2 preceding siblings ...)
2015-12-22 4:02 ` [PATCH 03/26] recipetool: create: avoid traceback on fetch error Paul Eggleton
@ 2015-12-22 4:02 ` Paul Eggleton
2015-12-22 4:02 ` [PATCH 05/26] recipetool: create: minor fix for potential issue in python handling Paul Eggleton
` (22 subsequent siblings)
26 siblings, 0 replies; 29+ messages in thread
From: Paul Eggleton @ 2015-12-22 4:02 UTC (permalink / raw)
To: openembedded-core
In my testing here it appears make -qn returns an error (exit code 2)
whereas make -n doesn't; I can't immediately tell why based on the
documentation. We don't actually care for it to be quiet since we're
capturing the output, so let's just leave -q off and have this work
properly as a result.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
scripts/lib/recipetool/create_buildsys.py | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/scripts/lib/recipetool/create_buildsys.py b/scripts/lib/recipetool/create_buildsys.py
index ed14a53..931ef3b 100644
--- a/scripts/lib/recipetool/create_buildsys.py
+++ b/scripts/lib/recipetool/create_buildsys.py
@@ -279,7 +279,7 @@ class MakefileRecipeHandler(RecipeHandler):
installtarget = True
try:
- stdout, stderr = bb.process.run('make -qn install', cwd=srctree, shell=True)
+ stdout, stderr = bb.process.run('make -n install', cwd=srctree, shell=True)
except bb.process.ExecutionError as e:
if e.exitcode != 1:
installtarget = False
--
2.5.0
^ permalink raw reply related [flat|nested] 29+ messages in thread
* [PATCH 05/26] recipetool: create: minor fix for potential issue in python handling
2015-12-22 4:02 [PATCH 00/26] devtool/recipetool improvements Paul Eggleton
` (3 preceding siblings ...)
2015-12-22 4:02 ` [PATCH 04/26] recipetool: create: fix do_install handling for makefile-only software Paul Eggleton
@ 2015-12-22 4:02 ` Paul Eggleton
2015-12-22 4:02 ` [PATCH 06/26] recipetool: create: prevent attempting to unpack entire DL_DIR Paul Eggleton
` (21 subsequent siblings)
26 siblings, 0 replies; 29+ messages in thread
From: Paul Eggleton @ 2015-12-22 4:02 UTC (permalink / raw)
To: openembedded-core
If SRC_URI happened not to be in the pre-generated lines then this code
would error out. This is unlikely to happen with the way the create code
is structured at the moment, but handle it just in case.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
scripts/lib/recipetool/create_buildsys_python.py | 5 ++++-
1 file changed, 4 insertions(+), 1 deletion(-)
diff --git a/scripts/lib/recipetool/create_buildsys_python.py b/scripts/lib/recipetool/create_buildsys_python.py
index e0af2a0..266c423 100644
--- a/scripts/lib/recipetool/create_buildsys_python.py
+++ b/scripts/lib/recipetool/create_buildsys_python.py
@@ -278,7 +278,10 @@ class PythonRecipeHandler(RecipeHandler):
for k in sorted(bbinfo):
v = bbinfo[k]
mdinfo.append('{} = "{}"'.format(k, v))
- lines_before[src_uri_line-1:src_uri_line-1] = mdinfo
+ if src_uri_line:
+ lines_before[src_uri_line-1:src_uri_line-1] = mdinfo
+ else:
+ lines_before.extend(mdinfo)
mapped_deps, unmapped_deps = self.scan_setup_python_deps(srctree, setup_info, setup_non_literals)
--
2.5.0
^ permalink raw reply related [flat|nested] 29+ messages in thread
* [PATCH 06/26] recipetool: create: prevent attempting to unpack entire DL_DIR
2015-12-22 4:02 [PATCH 00/26] devtool/recipetool improvements Paul Eggleton
` (4 preceding siblings ...)
2015-12-22 4:02 ` [PATCH 05/26] recipetool: create: minor fix for potential issue in python handling Paul Eggleton
@ 2015-12-22 4:02 ` Paul Eggleton
2015-12-22 4:03 ` [PATCH 07/26] recipetool: create: detect when specified URL returns a web page Paul Eggleton
` (20 subsequent siblings)
26 siblings, 0 replies; 29+ messages in thread
From: Paul Eggleton @ 2015-12-22 4:02 UTC (permalink / raw)
To: openembedded-core
If you specify a URL ending in /, BitBake's fetcher returns a localpath
of ${DL_DIR}, and if you then try to unpack that it will attempt to copy
the entire DL_DIR contents to the destination - which at least on my
system filled my entire /tmp. Obviously we should fix the fetcher, but
at least detect and stop that from happening here for now.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
scripts/lib/scriptutils.py | 4 ++++
1 file changed, 4 insertions(+)
diff --git a/scripts/lib/scriptutils.py b/scripts/lib/scriptutils.py
index 4dd7ef2..69e76d8 100644
--- a/scripts/lib/scriptutils.py
+++ b/scripts/lib/scriptutils.py
@@ -86,6 +86,10 @@ def fetch_uri(d, uri, destdir, srcrev=None):
ud = fetcher.ud[u]
ud.ignore_checksums = True
fetcher.download()
+ for u in fetcher.ud:
+ ud = fetcher.ud[u]
+ if ud.localpath.rstrip(os.sep) == localdata.getVar('DL_DIR', True).rstrip(os.sep):
+ raise Exception('Local path is download directory - please check that the URI "%s" is correct' % uri)
fetcher.unpack(destdir)
for u in fetcher.ud:
ud = fetcher.ud[u]
--
2.5.0
^ permalink raw reply related [flat|nested] 29+ messages in thread
* [PATCH 07/26] recipetool: create: detect when specified URL returns a web page
2015-12-22 4:02 [PATCH 00/26] devtool/recipetool improvements Paul Eggleton
` (5 preceding siblings ...)
2015-12-22 4:02 ` [PATCH 06/26] recipetool: create: prevent attempting to unpack entire DL_DIR Paul Eggleton
@ 2015-12-22 4:03 ` Paul Eggleton
2015-12-22 4:03 ` [PATCH 08/26] recipetool: create: set up priority system for recipe handlers Paul Eggleton
` (19 subsequent siblings)
26 siblings, 0 replies; 29+ messages in thread
From: Paul Eggleton @ 2015-12-22 4:03 UTC (permalink / raw)
To: openembedded-core
If the user specifies a URL that just returns a web page, then it's
probably incorrect (or broken); attempt to detect this and show an error
if it's the case.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
scripts/lib/recipetool/create.py | 15 +++++++++++----
1 file changed, 11 insertions(+), 4 deletions(-)
diff --git a/scripts/lib/recipetool/create.py b/scripts/lib/recipetool/create.py
index f98f525..3d5a373 100644
--- a/scripts/lib/recipetool/create.py
+++ b/scripts/lib/recipetool/create.py
@@ -130,10 +130,17 @@ def create_recipe(args):
dirlist = os.listdir(srctree)
if 'git.indirectionsymlink' in dirlist:
dirlist.remove('git.indirectionsymlink')
- if len(dirlist) == 1 and os.path.isdir(os.path.join(srctree, dirlist[0])):
- # We unpacked a single directory, so we should use that
- srcsubdir = dirlist[0]
- srctree = os.path.join(srctree, srcsubdir)
+ if len(dirlist) == 1:
+ singleitem = os.path.join(srctree, dirlist[0])
+ if os.path.isdir(singleitem):
+ # We unpacked a single directory, so we should use that
+ srcsubdir = dirlist[0]
+ srctree = os.path.join(srctree, srcsubdir)
+ else:
+ with open(singleitem, 'r') as f:
+ if '<html' in f.read(100).lower():
+ logger.error('Fetching "%s" returned a single HTML page - check the URL is correct and functional' % fetchuri)
+ sys.exit(1)
else:
# Assume we're pointing to an existing source tree
if args.extract_to:
--
2.5.0
^ permalink raw reply related [flat|nested] 29+ messages in thread
* [PATCH 08/26] recipetool: create: set up priority system for recipe handlers
2015-12-22 4:02 [PATCH 00/26] devtool/recipetool improvements Paul Eggleton
` (6 preceding siblings ...)
2015-12-22 4:03 ` [PATCH 07/26] recipetool: create: detect when specified URL returns a web page Paul Eggleton
@ 2015-12-22 4:03 ` Paul Eggleton
2015-12-22 4:03 ` [PATCH 09/26] recipetool: create: support extracting name and version from build scripts Paul Eggleton
` (18 subsequent siblings)
26 siblings, 0 replies; 29+ messages in thread
From: Paul Eggleton @ 2015-12-22 4:03 UTC (permalink / raw)
To: openembedded-core
Sometimes we want to force one handler to run before another; if the two
handlers are in different plugins that's difficult without some kind of
priority number, so add one and sort by it.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
scripts/lib/recipetool/create.py | 16 ++++++++++++++--
1 file changed, 14 insertions(+), 2 deletions(-)
diff --git a/scripts/lib/recipetool/create.py b/scripts/lib/recipetool/create.py
index 3d5a373..5c249ab 100644
--- a/scripts/lib/recipetool/create.py
+++ b/scripts/lib/recipetool/create.py
@@ -251,10 +251,22 @@ def create_recipe(args):
lines_after.append('')
# Find all plugins that want to register handlers
- handlers = []
+ logger.debug('Loading recipe handlers')
+ raw_handlers = []
for plugin in plugins:
if hasattr(plugin, 'register_recipe_handlers'):
- plugin.register_recipe_handlers(handlers)
+ plugin.register_recipe_handlers(raw_handlers)
+ # Sort handlers by priority
+ handlers = []
+ for i, handler in enumerate(raw_handlers):
+ if isinstance(handler, tuple):
+ handlers.append((handler[0], handler[1], i))
+ else:
+ handlers.append((handler, 0, i))
+ handlers.sort(key=lambda item: (item[1], -item[2]), reverse=True)
+ for handler, priority, _ in handlers:
+ logger.debug('Handler: %s (priority %d)' % (handler.__class__.__name__, priority))
+ handlers = [item[0] for item in handlers]
# Apply the handlers
classes = []
--
2.5.0
^ permalink raw reply related [flat|nested] 29+ messages in thread
* [PATCH 09/26] recipetool: create: support extracting name and version from build scripts
2015-12-22 4:02 [PATCH 00/26] devtool/recipetool improvements Paul Eggleton
` (7 preceding siblings ...)
2015-12-22 4:03 ` [PATCH 08/26] recipetool: create: set up priority system for recipe handlers Paul Eggleton
@ 2015-12-22 4:03 ` Paul Eggleton
2015-12-22 4:03 ` [PATCH 10/26] recipetool: create: basic extraction of name/version from filename Paul Eggleton
` (17 subsequent siblings)
26 siblings, 0 replies; 29+ messages in thread
From: Paul Eggleton @ 2015-12-22 4:03 UTC (permalink / raw)
To: openembedded-core
Some build systems (notably autotools) support declaring the name and
version of the program being built; since we need those for the recipe
we can attempt to extract them. It's a little fuzzy as they are often
omitted or may not be appropriately formatted for our purposes, but it
does work on a reasonable number of software packages to be useful.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
meta/lib/oeqa/selftest/devtool.py | 2 +-
meta/lib/oeqa/selftest/recipetool.py | 2 +-
scripts/lib/recipetool/create.py | 130 ++++++++---
scripts/lib/recipetool/create_buildsys.py | 282 +++++++++++++++++++----
scripts/lib/recipetool/create_buildsys_python.py | 2 +-
5 files changed, 344 insertions(+), 74 deletions(-)
diff --git a/meta/lib/oeqa/selftest/devtool.py b/meta/lib/oeqa/selftest/devtool.py
index 0a44ae7..8faea93 100644
--- a/meta/lib/oeqa/selftest/devtool.py
+++ b/meta/lib/oeqa/selftest/devtool.py
@@ -279,7 +279,7 @@ class DevtoolTests(DevtoolBase):
self.assertIn('_git.bb', recipefile, 'Recipe file incorrectly named')
checkvars = {}
checkvars['S'] = '${WORKDIR}/git'
- checkvars['PV'] = '1.0+git${SRCPV}'
+ checkvars['PV'] = '1.11+git${SRCPV}'
checkvars['SRC_URI'] = url
checkvars['SRCREV'] = '${AUTOREV}'
self._test_recipe_contents(recipefile, checkvars, [])
diff --git a/meta/lib/oeqa/selftest/recipetool.py b/meta/lib/oeqa/selftest/recipetool.py
index b1f1d2ab..bbfce7c 100644
--- a/meta/lib/oeqa/selftest/recipetool.py
+++ b/meta/lib/oeqa/selftest/recipetool.py
@@ -395,7 +395,7 @@ class RecipetoolTests(RecipetoolBase):
checkvars['LICENSE'] = 'LGPLv2.1'
checkvars['LIC_FILES_CHKSUM'] = 'file://COPYING;md5=7fbc338309ac38fefcd64b04bb903e34'
checkvars['S'] = '${WORKDIR}/git'
- checkvars['PV'] = '1.0+git${SRCPV}'
+ checkvars['PV'] = '1.11+git${SRCPV}'
checkvars['SRC_URI'] = srcuri
checkvars['DEPENDS'] = 'libpng pango libx11 libxext jpeg'
inherits = ['autotools', 'pkgconfig']
diff --git a/scripts/lib/recipetool/create.py b/scripts/lib/recipetool/create.py
index 5c249ab..6c7b9fd 100644
--- a/scripts/lib/recipetool/create.py
+++ b/scripts/lib/recipetool/create.py
@@ -41,10 +41,17 @@ def tinfoil_init(instance):
class RecipeHandler():
@staticmethod
- def checkfiles(path, speclist):
+ def checkfiles(path, speclist, recursive=False):
results = []
- for spec in speclist:
- results.extend(glob.glob(os.path.join(path, spec)))
+ if recursive:
+ for root, _, files in os.walk(path):
+ for fn in files:
+ for spec in speclist:
+ if fnmatch.fnmatch(fn, spec):
+ results.append(os.path.join(root, fn))
+ else:
+ for spec in speclist:
+ results.extend(glob.glob(os.path.join(path, spec)))
return results
def genfunction(self, outlines, funcname, content, python=False, forcespace=False):
@@ -70,10 +77,14 @@ class RecipeHandler():
outlines.append('}')
outlines.append('')
- def process(self, srctree, classes, lines_before, lines_after, handled):
+ def process(self, srctree, classes, lines_before, lines_after, handled, extravalues):
return False
+def validate_pv(pv):
+ if not pv or '_version' in pv.lower() or pv[0] not in '0123456789':
+ return False
+ return True
def supports_srcrev(uri):
localdata = bb.data.createCopy(tinfoil.config_data)
@@ -152,7 +163,12 @@ def create_recipe(args):
srcuri = ''
srctree = args.source
- outfile = args.outfile
+ if args.outfile and os.path.isdir(args.outfile):
+ outfile = None
+ outdir = args.outfile
+ else:
+ outfile = args.outfile
+ outdir = None
if outfile and outfile != '-':
if os.path.exists(outfile):
logger.error('Output file %s already exists' % outfile)
@@ -196,28 +212,29 @@ def create_recipe(args):
lines_before.append('')
# FIXME This is kind of a hack, we probably ought to be using bitbake to do this
- # we'd also want a way to automatically set outfile based upon auto-detecting these values from the source if possible
- recipefn = os.path.splitext(os.path.basename(outfile))[0]
- fnsplit = recipefn.split('_')
- if len(fnsplit) > 1:
- pn = fnsplit[0]
- pv = fnsplit[1]
- else:
- pn = recipefn
- pv = None
+ pn = None
+ pv = None
+ if outfile:
+ recipefn = os.path.splitext(os.path.basename(outfile))[0]
+ fnsplit = recipefn.split('_')
+ if len(fnsplit) > 1:
+ pn = fnsplit[0]
+ pv = fnsplit[1]
+ else:
+ pn = recipefn
if args.version:
pv = args.version
+ if args.name:
+ pn = args.name
+
if pv and pv not in 'git svn hg'.split():
realpv = pv
else:
realpv = None
- if srcuri:
- if realpv:
- srcuri = srcuri.replace(realpv, '${PV}')
- else:
+ if not srcuri:
lines_before.append('# No information for SRC_URI yet (only an external source tree was specified)')
lines_before.append('SRC_URI = "%s"' % srcuri)
(md5value, sha256value) = checksums
@@ -232,13 +249,7 @@ def create_recipe(args):
lines_before.append('SRCREV = "%s"' % srcrev)
lines_before.append('')
- if srcsubdir and pv:
- if srcsubdir == "%s-%s" % (pn, pv):
- # This would be the default, so we don't need to set S in the recipe
- srcsubdir = ''
if srcsubdir:
- if pv and pv not in 'git svn hg'.split():
- srcsubdir = srcsubdir.replace(pv, '${PV}')
lines_before.append('S = "${WORKDIR}/%s"' % srcsubdir)
lines_before.append('')
@@ -276,8 +287,74 @@ def create_recipe(args):
classes.append('bin_package')
handled.append('buildsystem')
+ extravalues = {}
for handler in handlers:
- handler.process(srctree, classes, lines_before, lines_after, handled)
+ handler.process(srctree, classes, lines_before, lines_after, handled, extravalues)
+
+ if not realpv:
+ realpv = extravalues.get('PV', None)
+ if realpv:
+ if not validate_pv(realpv):
+ realpv = None
+ else:
+ realpv = realpv.lower().split()[0]
+ if '_' in realpv:
+ realpv = realpv.replace('_', '-')
+ if not pn:
+ pn = extravalues.get('PN', None)
+ if pn:
+ if pn.startswith('GNU '):
+ pn = pn[4:]
+ if ' ' in pn:
+ # Probably a descriptive identifier rather than a proper name
+ pn = None
+ else:
+ pn = pn.lower()
+ if '_' in pn:
+ pn = pn.replace('_', '-')
+
+ if not outfile:
+ if not pn:
+ logger.error('Unable to determine short program name from source tree - please specify name with -N/--name or output file name with -o/--outfile')
+ # devtool looks for this specific exit code, so don't change it
+ sys.exit(15)
+ else:
+ if srcuri and srcuri.startswith(('git://', 'hg://', 'svn://')):
+ outfile = '%s_%s.bb' % (pn, srcuri.split(':', 1)[0])
+ elif realpv:
+ outfile = '%s_%s.bb' % (pn, realpv)
+ else:
+ outfile = '%s.bb' % pn
+ if outdir:
+ outfile = os.path.join(outdir, outfile)
+ # We need to check this again
+ if os.path.exists(outfile):
+ logger.error('Output file %s already exists' % outfile)
+ sys.exit(1)
+
+ lines = lines_before
+ lines_before = []
+ skipblank = True
+ for line in lines:
+ if skipblank:
+ skipblank = False
+ if not line:
+ continue
+ if line.startswith('S = '):
+ if realpv and pv not in 'git svn hg'.split():
+ line = line.replace(realpv, '${PV}')
+ if pn:
+ line = line.replace(pn, '${BPN}')
+ if line == 'S = "${WORKDIR}/${BPN}-${PV}"':
+ skipblank = True
+ continue
+ elif line.startswith('SRC_URI = '):
+ if realpv:
+ line = line.replace(realpv, '${PV}')
+ elif line.startswith('PV = '):
+ if realpv:
+ line = re.sub('"[^+]*\+', '"%s+' % realpv, line)
+ lines_before.append(line)
outlines = []
outlines.extend(lines_before)
@@ -469,9 +546,10 @@ def register_commands(subparsers):
help='Create a new recipe',
description='Creates a new recipe from a source tree')
parser_create.add_argument('source', help='Path or URL to source')
- parser_create.add_argument('-o', '--outfile', help='Specify filename for recipe to create', required=True)
+ parser_create.add_argument('-o', '--outfile', help='Specify filename for recipe to create')
parser_create.add_argument('-m', '--machine', help='Make recipe machine-specific as opposed to architecture-specific', action='store_true')
parser_create.add_argument('-x', '--extract-to', metavar='EXTRACTPATH', help='Assuming source is a URL, fetch it and extract it to the directory specified as %(metavar)s')
+ parser_create.add_argument('-N', '--name', help='Name to use within recipe (PN)')
parser_create.add_argument('-V', '--version', help='Version to use within recipe (PV)')
parser_create.add_argument('-b', '--binary', help='Treat the source tree as something that should be installed verbatim (no compilation, same directory structure)', action='store_true')
parser_create.set_defaults(func=create_recipe)
diff --git a/scripts/lib/recipetool/create_buildsys.py b/scripts/lib/recipetool/create_buildsys.py
index 931ef3b..0aff59e 100644
--- a/scripts/lib/recipetool/create_buildsys.py
+++ b/scripts/lib/recipetool/create_buildsys.py
@@ -17,7 +17,7 @@
import re
import logging
-from recipetool.create import RecipeHandler, read_pkgconfig_provides
+from recipetool.create import RecipeHandler, read_pkgconfig_provides, validate_pv
logger = logging.getLogger('recipetool')
@@ -27,13 +27,17 @@ def tinfoil_init(instance):
global tinfoil
tinfoil = instance
+
class CmakeRecipeHandler(RecipeHandler):
- def process(self, srctree, classes, lines_before, lines_after, handled):
+ def process(self, srctree, classes, lines_before, lines_after, handled, extravalues):
if 'buildsystem' in handled:
return False
if RecipeHandler.checkfiles(srctree, ['CMakeLists.txt']):
classes.append('cmake')
+ values = CmakeRecipeHandler.extract_cmake_deps(lines_before, srctree, extravalues)
+ for var, value in values.iteritems():
+ lines_before.append('%s = "%s"' % (var, value))
lines_after.append('# Specify any options you want to pass to cmake using EXTRA_OECMAKE:')
lines_after.append('EXTRA_OECMAKE = ""')
lines_after.append('')
@@ -41,8 +45,26 @@ class CmakeRecipeHandler(RecipeHandler):
return True
return False
+ @staticmethod
+ def extract_cmake_deps(outlines, srctree, extravalues, cmakelistsfile=None):
+ values = {}
+
+ if cmakelistsfile:
+ srcfiles = [cmakelistsfile]
+ else:
+ srcfiles = RecipeHandler.checkfiles(srctree, ['CMakeLists.txt'])
+
+ proj_re = re.compile('project\(([^)]*)\)', re.IGNORECASE)
+ with open(srcfiles[0], 'r') as f:
+ for line in f:
+ res = proj_re.match(line.strip())
+ if res:
+ extravalues['PN'] = res.group(1).split()[0]
+
+ return values
+
class SconsRecipeHandler(RecipeHandler):
- def process(self, srctree, classes, lines_before, lines_after, handled):
+ def process(self, srctree, classes, lines_before, lines_after, handled, extravalues):
if 'buildsystem' in handled:
return False
@@ -56,7 +78,7 @@ class SconsRecipeHandler(RecipeHandler):
return False
class QmakeRecipeHandler(RecipeHandler):
- def process(self, srctree, classes, lines_before, lines_after, handled):
+ def process(self, srctree, classes, lines_before, lines_after, handled, extravalues):
if 'buildsystem' in handled:
return False
@@ -67,14 +89,14 @@ class QmakeRecipeHandler(RecipeHandler):
return False
class AutotoolsRecipeHandler(RecipeHandler):
- def process(self, srctree, classes, lines_before, lines_after, handled):
+ def process(self, srctree, classes, lines_before, lines_after, handled, extravalues):
if 'buildsystem' in handled:
return False
autoconf = False
if RecipeHandler.checkfiles(srctree, ['configure.ac', 'configure.in']):
autoconf = True
- values = AutotoolsRecipeHandler.extract_autotools_deps(lines_before, srctree)
+ values = AutotoolsRecipeHandler.extract_autotools_deps(lines_before, srctree, extravalues)
classes.extend(values.pop('inherit', '').split())
for var, value in values.iteritems():
lines_before.append('%s = "%s"' % (var, value))
@@ -88,6 +110,22 @@ class AutotoolsRecipeHandler(RecipeHandler):
autoconf = True
break
+ if autoconf and not ('PV' in extravalues and 'PN' in extravalues):
+ # Last resort
+ conffile = RecipeHandler.checkfiles(srctree, ['configure'])
+ if conffile:
+ with open(conffile[0], 'r') as f:
+ for line in f:
+ line = line.strip()
+ if line.startswith('VERSION=') or line.startswith('PACKAGE_VERSION='):
+ pv = line.split('=')[1].strip('"\'')
+ if pv and not 'PV' in extravalues and validate_pv(pv):
+ extravalues['PV'] = pv
+ elif line.startswith('PACKAGE_NAME=') or line.startswith('PACKAGE='):
+ pn = line.split('=')[1].strip('"\'')
+ if pn and not 'PN' in extravalues:
+ extravalues['PN'] = pn
+
if autoconf:
lines_before.append('# NOTE: if this software is not capable of being built in a separate build directory')
lines_before.append('# from the source, you should replace autotools with autotools-brokensep in the')
@@ -102,7 +140,7 @@ class AutotoolsRecipeHandler(RecipeHandler):
return False
@staticmethod
- def extract_autotools_deps(outlines, srctree, acfile=None):
+ def extract_autotools_deps(outlines, srctree, extravalues=None, acfile=None):
import shlex
import oe.package
@@ -122,6 +160,9 @@ class AutotoolsRecipeHandler(RecipeHandler):
lib_re = re.compile('AC_CHECK_LIB\(\[?([a-zA-Z0-9]*)\]?, .*')
progs_re = re.compile('_PROGS?\(\[?[a-zA-Z0-9]*\]?, \[?([^,\]]*)\]?[),].*')
dep_re = re.compile('([^ ><=]+)( [<>=]+ [^ ><=]+)?')
+ ac_init_re = re.compile('AC_INIT\(([^,]+), *([^,]+)[,)].*')
+ am_init_re = re.compile('AM_INIT_AUTOMAKE\(([^,]+), *([^,]+)[,)].*')
+ define_re = re.compile(' *(m4_)?define\(([^,]+), *([^,]+)\)')
# Build up lib library->package mapping
shlib_providers = oe.package.read_shlib_providers(tinfoil.config_data)
@@ -157,55 +198,157 @@ class AutotoolsRecipeHandler(RecipeHandler):
else:
raise
+ defines = {}
+ def subst_defines(value):
+ newvalue = value
+ for define, defval in defines.iteritems():
+ newvalue = newvalue.replace(define, defval)
+ if newvalue != value:
+ return subst_defines(newvalue)
+ return value
+
+ def process_value(value):
+ value = value.replace('[', '').replace(']', '')
+ if value.startswith('m4_esyscmd(') or value.startswith('m4_esyscmd_s('):
+ cmd = subst_defines(value[value.index('(')+1:-1])
+ try:
+ if '|' in cmd:
+ cmd = 'set -o pipefail; ' + cmd
+ stdout, _ = bb.process.run(cmd, cwd=srctree, shell=True)
+ ret = stdout.rstrip()
+ except bb.process.ExecutionError as e:
+ ret = ''
+ elif value.startswith('m4_'):
+ return None
+ ret = subst_defines(value)
+ if ret:
+ ret = ret.strip('"\'')
+ return ret
+
# Since a configure.ac file is essentially a program, this is only ever going to be
# a hack unfortunately; but it ought to be enough of an approximation
if acfile:
srcfiles = [acfile]
else:
- srcfiles = RecipeHandler.checkfiles(srctree, ['configure.ac', 'configure.in'])
+ srcfiles = RecipeHandler.checkfiles(srctree, ['acinclude.m4', 'configure.ac', 'configure.in'])
+
pcdeps = []
deps = []
unmapped = []
unmappedlibs = []
- with open(srcfiles[0], 'r') as f:
- for line in f:
- if 'PKG_CHECK_MODULES' in line:
- res = pkg_re.search(line)
- if res:
- res = dep_re.findall(res.group(1))
- if res:
- pcdeps.extend([x[0] for x in res])
- inherits.append('pkgconfig')
- if line.lstrip().startswith('AM_GNU_GETTEXT'):
- inherits.append('gettext')
- elif 'AC_CHECK_PROG' in line or 'AC_PATH_PROG' in line:
- res = progs_re.search(line)
+
+ def process_macro(keyword, value):
+ if keyword == 'PKG_CHECK_MODULES':
+ res = pkg_re.search(value)
+ if res:
+ res = dep_re.findall(res.group(1))
if res:
- for prog in shlex.split(res.group(1)):
- prog = prog.split()[0]
- progclass = progclassmap.get(prog, None)
- if progclass:
- inherits.append(progclass)
+ pcdeps.extend([x[0] for x in res])
+ inherits.append('pkgconfig')
+ elif keyword == 'AM_GNU_GETTEXT':
+ inherits.append('gettext')
+ elif keyword == 'AC_CHECK_PROG' or keyword == 'AC_PATH_PROG':
+ res = progs_re.search(value)
+ if res:
+ for prog in shlex.split(res.group(1)):
+ prog = prog.split()[0]
+ progclass = progclassmap.get(prog, None)
+ if progclass:
+ inherits.append(progclass)
+ else:
+ progdep = progmap.get(prog, None)
+ if progdep:
+ deps.append(progdep)
else:
- progdep = progmap.get(prog, None)
- if progdep:
- deps.append(progdep)
- else:
- if not prog.startswith('$'):
- unmapped.append(prog)
- elif 'AC_CHECK_LIB' in line:
- res = lib_re.search(line)
+ if not prog.startswith('$'):
+ unmapped.append(prog)
+ elif keyword == 'AC_CHECK_LIB':
+ res = lib_re.search(value)
+ if res:
+ lib = res.group(1)
+ libdep = recipelibmap.get(lib, None)
+ if libdep:
+ deps.append(libdep)
+ else:
+ if libdep is None:
+ if not lib.startswith('$'):
+ unmappedlibs.append(lib)
+ elif keyword == 'AC_PATH_X':
+ deps.append('libx11')
+ elif keyword == 'AC_INIT':
+ if extravalues is not None:
+ res = ac_init_re.match(value)
if res:
- lib = res.group(1)
- libdep = recipelibmap.get(lib, None)
- if libdep:
- deps.append(libdep)
- else:
- if libdep is None:
- if not lib.startswith('$'):
- unmappedlibs.append(lib)
- elif 'AC_PATH_X' in line:
- deps.append('libx11')
+ extravalues['PN'] = process_value(res.group(1))
+ pv = process_value(res.group(2))
+ if validate_pv(pv):
+ extravalues['PV'] = pv
+ elif keyword == 'AM_INIT_AUTOMAKE':
+ if extravalues is not None:
+ if 'PN' not in extravalues:
+ res = am_init_re.match(value)
+ if res:
+ if res.group(1) != 'AC_PACKAGE_NAME':
+ extravalues['PN'] = process_value(res.group(1))
+ pv = process_value(res.group(2))
+ if validate_pv(pv):
+ extravalues['PV'] = pv
+ elif keyword == 'define(':
+ res = define_re.match(value)
+ if res:
+ key = res.group(2).strip('[]')
+ value = process_value(res.group(3))
+ if value is not None:
+ defines[key] = value
+
+ keywords = ['PKG_CHECK_MODULES',
+ 'AM_GNU_GETTEXT',
+ 'AC_CHECK_PROG',
+ 'AC_PATH_PROG',
+ 'AC_CHECK_LIB',
+ 'AC_PATH_X',
+ 'AC_INIT',
+ 'AM_INIT_AUTOMAKE',
+ 'define(',
+ ]
+ for srcfile in srcfiles:
+ nesting = 0
+ in_keyword = ''
+ partial = ''
+ with open(srcfile, 'r') as f:
+ for line in f:
+ if in_keyword:
+ partial += ' ' + line.strip()
+ if partial.endswith('\\'):
+ partial = partial[:-1]
+ nesting = nesting + line.count('(') - line.count(')')
+ if nesting == 0:
+ process_macro(in_keyword, partial)
+ partial = ''
+ in_keyword = ''
+ else:
+ for keyword in keywords:
+ if keyword in line:
+ nesting = line.count('(') - line.count(')')
+ if nesting > 0:
+ partial = line.strip()
+ if partial.endswith('\\'):
+ partial = partial[:-1]
+ in_keyword = keyword
+ else:
+ process_macro(keyword, line.strip())
+ break
+
+ if in_keyword:
+ process_macro(in_keyword, partial)
+
+ if extravalues:
+ for k,v in extravalues.items():
+ if v:
+ if v.startswith('$') or v.startswith('@') or v.startswith('%'):
+ del extravalues[k]
+ else:
+ extravalues[k] = v.strip('"\'').rstrip('()')
if unmapped:
outlines.append('# NOTE: the following prog dependencies are unknown, ignoring: %s' % ' '.join(unmapped))
@@ -240,7 +383,7 @@ class AutotoolsRecipeHandler(RecipeHandler):
class MakefileRecipeHandler(RecipeHandler):
- def process(self, srctree, classes, lines_before, lines_after, handled):
+ def process(self, srctree, classes, lines_before, lines_after, handled, extravalues):
if 'buildsystem' in handled:
return False
@@ -307,6 +450,53 @@ class MakefileRecipeHandler(RecipeHandler):
self.genfunction(lines_after, 'do_install', ['# Specify install commands here'])
+class VersionFileRecipeHandler(RecipeHandler):
+ def process(self, srctree, classes, lines_before, lines_after, handled, extravalues):
+ if 'PV' not in extravalues:
+ # Look for a VERSION or version file containing a single line consisting
+ # only of a version number
+ filelist = RecipeHandler.checkfiles(srctree, ['VERSION', 'version'])
+ version = None
+ for fileitem in filelist:
+ linecount = 0
+ with open(fileitem, 'r') as f:
+ for line in f:
+ line = line.rstrip().strip('"\'')
+ linecount += 1
+ if line:
+ if linecount > 1:
+ version = None
+ break
+ else:
+ if validate_pv(line):
+ version = line
+ if version:
+ extravalues['PV'] = version
+ break
+
+
+class SpecFileRecipeHandler(RecipeHandler):
+ def process(self, srctree, classes, lines_before, lines_after, handled, extravalues):
+ if 'PV' in extravalues and 'PN' in extravalues:
+ return
+ filelist = RecipeHandler.checkfiles(srctree, ['*.spec'], recursive=True)
+ pn = None
+ pv = None
+ for fileitem in filelist:
+ linecount = 0
+ with open(fileitem, 'r') as f:
+ for line in f:
+ if line.startswith('Name:') and not pn:
+ pn = line.split(':')[1].strip()
+ if line.startswith('Version:') and not pv:
+ pv = line.split(':')[1].strip()
+ if pv or pn:
+ if pv and not 'PV' in extravalues and validate_pv(pv):
+ extravalues['PV'] = pv
+ if pn and not 'PN' in extravalues:
+ extravalues['PN'] = pn
+ break
+
def register_recipe_handlers(handlers):
# These are in a specific order so that the right one is detected first
handlers.append(CmakeRecipeHandler())
@@ -314,3 +504,5 @@ def register_recipe_handlers(handlers):
handlers.append(SconsRecipeHandler())
handlers.append(QmakeRecipeHandler())
handlers.append(MakefileRecipeHandler())
+ handlers.append((VersionFileRecipeHandler(), -1))
+ handlers.append((SpecFileRecipeHandler(), -1))
diff --git a/scripts/lib/recipetool/create_buildsys_python.py b/scripts/lib/recipetool/create_buildsys_python.py
index 266c423..4d65c96 100644
--- a/scripts/lib/recipetool/create_buildsys_python.py
+++ b/scripts/lib/recipetool/create_buildsys_python.py
@@ -159,7 +159,7 @@ class PythonRecipeHandler(RecipeHandler):
def __init__(self):
pass
- def process(self, srctree, classes, lines_before, lines_after, handled):
+ def process(self, srctree, classes, lines_before, lines_after, handled, extravalues):
if 'buildsystem' in handled:
return False
--
2.5.0
^ permalink raw reply related [flat|nested] 29+ messages in thread
* [PATCH 10/26] recipetool: create: basic extraction of name/version from filename
2015-12-22 4:02 [PATCH 00/26] devtool/recipetool improvements Paul Eggleton
` (8 preceding siblings ...)
2015-12-22 4:03 ` [PATCH 09/26] recipetool: create: support extracting name and version from build scripts Paul Eggleton
@ 2015-12-22 4:03 ` Paul Eggleton
2015-12-22 4:03 ` [PATCH 11/26] devtool: edit-recipe: add new subcommand Paul Eggleton
` (16 subsequent siblings)
26 siblings, 0 replies; 29+ messages in thread
From: Paul Eggleton @ 2015-12-22 4:03 UTC (permalink / raw)
To: openembedded-core
Often the filename (e.g. source tarball) contains the name and version
of the software it contains.
(This isn't intended to be exhaustive, just to catch the common case.)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
meta/lib/oeqa/selftest/devtool.py | 15 ++++++++-------
scripts/lib/recipetool/create.py | 32 ++++++++++++++++++++++++++++++++
2 files changed, 40 insertions(+), 7 deletions(-)
diff --git a/meta/lib/oeqa/selftest/devtool.py b/meta/lib/oeqa/selftest/devtool.py
index 8faea93..7af82df 100644
--- a/meta/lib/oeqa/selftest/devtool.py
+++ b/meta/lib/oeqa/selftest/devtool.py
@@ -232,15 +232,16 @@ class DevtoolTests(DevtoolBase):
self.assertIn(srcdir, result.output)
# Check recipe
recipefile = get_bb_var('FILE', testrecipe)
- self.assertIn('%s.bb' % testrecipe, recipefile, 'Recipe file incorrectly named')
+ self.assertIn('%s_%s.bb' % (testrecipe, testver), recipefile, 'Recipe file incorrectly named')
checkvars = {}
- checkvars['S'] = '${WORKDIR}/MarkupSafe-%s' % testver
- checkvars['SRC_URI'] = url
+ checkvars['S'] = '${WORKDIR}/MarkupSafe-${PV}'
+ checkvars['SRC_URI'] = url.replace(testver, '${PV}')
self._test_recipe_contents(recipefile, checkvars, [])
# Try with version specified
result = runCmd('devtool reset -n %s' % testrecipe)
shutil.rmtree(srcdir)
- result = runCmd('devtool add %s %s -f %s -V %s' % (testrecipe, srcdir, url, testver))
+ fakever = '1.9'
+ result = runCmd('devtool add %s %s -f %s -V %s' % (testrecipe, srcdir, url, fakever))
self.assertTrue(os.path.isfile(os.path.join(srcdir, 'setup.py')), 'Unable to find setup.py in source directory')
# Test devtool status
result = runCmd('devtool status')
@@ -248,10 +249,10 @@ class DevtoolTests(DevtoolBase):
self.assertIn(srcdir, result.output)
# Check recipe
recipefile = get_bb_var('FILE', testrecipe)
- self.assertIn('%s_%s.bb' % (testrecipe, testver), recipefile, 'Recipe file incorrectly named')
+ self.assertIn('%s_%s.bb' % (testrecipe, fakever), recipefile, 'Recipe file incorrectly named')
checkvars = {}
- checkvars['S'] = '${WORKDIR}/MarkupSafe-${PV}'
- checkvars['SRC_URI'] = url.replace(testver, '${PV}')
+ checkvars['S'] = '${WORKDIR}/MarkupSafe-%s' % testver
+ checkvars['SRC_URI'] = url
self._test_recipe_contents(recipefile, checkvars, [])
@testcase(1161)
diff --git a/scripts/lib/recipetool/create.py b/scripts/lib/recipetool/create.py
index 6c7b9fd..4887604 100644
--- a/scripts/lib/recipetool/create.py
+++ b/scripts/lib/recipetool/create.py
@@ -86,6 +86,27 @@ def validate_pv(pv):
return False
return True
+def determine_from_filename(srcfile):
+ """Determine name and version from a filename"""
+ part = ''
+ if '.tar.' in srcfile:
+ namepart = srcfile.split('.tar.')[0]
+ else:
+ namepart = os.path.splitext(srcfile)[0]
+ splitval = namepart.rsplit('_', 1)
+ if len(splitval) == 1:
+ splitval = namepart.rsplit('-', 1)
+ pn = splitval[0].replace('_', '-')
+ if len(splitval) > 1:
+ if splitval[1][0] in '0123456789':
+ pv = splitval[1]
+ else:
+ pn = '-'.join(splitval).replace('_', '-')
+ pv = None
+ else:
+ pv = None
+ return (pn, pv)
+
def supports_srcrev(uri):
localdata = bb.data.createCopy(tinfoil.config_data)
# This is a bit sad, but if you don't have this set there can be some
@@ -234,6 +255,17 @@ def create_recipe(args):
else:
realpv = None
+ if srcuri and not realpv or not pn:
+ parseres = urlparse.urlparse(srcuri)
+ if parseres.path:
+ srcfile = os.path.basename(parseres.path)
+ name_pn, name_pv = determine_from_filename(srcfile)
+ logger.debug('Determined from filename: name = "%s", version = "%s"' % (name_pn, name_pv))
+ if name_pn and not pn:
+ pn = name_pn
+ if name_pv and not realpv:
+ realpv = name_pv
+
if not srcuri:
lines_before.append('# No information for SRC_URI yet (only an external source tree was specified)')
lines_before.append('SRC_URI = "%s"' % srcuri)
--
2.5.0
^ permalink raw reply related [flat|nested] 29+ messages in thread
* [PATCH 11/26] devtool: edit-recipe: add new subcommand
2015-12-22 4:02 [PATCH 00/26] devtool/recipetool improvements Paul Eggleton
` (9 preceding siblings ...)
2015-12-22 4:03 ` [PATCH 10/26] recipetool: create: basic extraction of name/version from filename Paul Eggleton
@ 2015-12-22 4:03 ` Paul Eggleton
2015-12-22 4:03 ` [PATCH 12/26] devtool: add: tweak help text Paul Eggleton
` (15 subsequent siblings)
26 siblings, 0 replies; 29+ messages in thread
From: Paul Eggleton @ 2015-12-22 4:03 UTC (permalink / raw)
To: openembedded-core
Add an "edit-recipe" subcommand that runs your default editor (as
specified by the EDITOR environment variable) on the specified recipe in
the workspace. Note that by default the recipe file itself must be in
the workspace - i.e. as a result of "devtool add" or "devtool upgrade";
however there is a -a/--any-recipe option to override this.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
scripts/lib/devtool/standard.py | 44 +++++++++++++++++++++++++++++++++++++++++
1 file changed, 44 insertions(+)
diff --git a/scripts/lib/devtool/standard.py b/scripts/lib/devtool/standard.py
index a5e81f3..c11de9d 100644
--- a/scripts/lib/devtool/standard.py
+++ b/scripts/lib/devtool/standard.py
@@ -1140,6 +1140,44 @@ def reset(args, config, basepath, workspace):
return 0
+def edit_recipe(args, config, basepath, workspace):
+ """Entry point for the devtool 'edit-recipe' subcommand"""
+ if args.any_recipe:
+ tinfoil = setup_tinfoil(config_only=False, basepath=basepath)
+ try:
+ rd = parse_recipe(config, tinfoil, args.recipename, True)
+ if not rd:
+ return 1
+ recipefile = rd.getVar('FILE', True)
+ finally:
+ tinfoil.shutdown()
+ else:
+ check_workspace_recipe(workspace, args.recipename)
+ bbappend = workspace[args.recipename]['bbappend']
+ bbfile = os.path.basename(bbappend).replace('.bbappend', '.bb').replace('%', '*')
+ recipefile = glob.glob(os.path.join(config.workspace_path,
+ 'recipes',
+ args.recipename,
+ bbfile))
+ if recipefile:
+ recipefile = recipefile[0]
+ else:
+ raise DevtoolError("Recipe file for %s is not under the workspace" %
+ args.recipename)
+
+ editor = os.environ.get('EDITOR', None)
+ if not editor:
+ raise DevtoolError("EDITOR environment variable not set")
+
+ import subprocess
+ try:
+ subprocess.check_call('%s "%s"' % (editor, recipefile), shell=True)
+ except subprocess.CalledProcessError as e:
+ return e.returncode
+
+ return 0
+
+
def register_commands(subparsers, context):
"""Register devtool subcommands from this plugin"""
parser_add = subparsers.add_parser('add', help='Add a new recipe',
@@ -1204,3 +1242,9 @@ def register_commands(subparsers, context):
parser_reset.add_argument('--all', '-a', action="store_true", help='Reset all recipes (clear workspace)')
parser_reset.add_argument('--no-clean', '-n', action="store_true", help='Don\'t clean the sysroot to remove recipe output')
parser_reset.set_defaults(func=reset)
+
+ parser_edit_recipe = subparsers.add_parser('edit-recipe', help='Edit a recipe file in your workspace',
+ description='Runs the default editor (as specified by the EDITOR variable) on the specified recipe. Note that the recipe file itself must be in the workspace (i.e. as a result of "devtool add" or "devtool upgrade"); you can override this with the -a/--any-recipe option.')
+ parser_edit_recipe.add_argument('recipename', help='Recipe to edit')
+ parser_edit_recipe.add_argument('--any-recipe', '-a', action="store_true", help='Edit any recipe, not just where the recipe file itself is in the workspace')
+ parser_edit_recipe.set_defaults(func=edit_recipe)
--
2.5.0
^ permalink raw reply related [flat|nested] 29+ messages in thread
* [PATCH 12/26] devtool: add: tweak help text
2015-12-22 4:02 [PATCH 00/26] devtool/recipetool improvements Paul Eggleton
` (10 preceding siblings ...)
2015-12-22 4:03 ` [PATCH 11/26] devtool: edit-recipe: add new subcommand Paul Eggleton
@ 2015-12-22 4:03 ` Paul Eggleton
2015-12-22 4:03 ` [PATCH 13/26] devtool: split out function for naming bbappend Paul Eggleton
` (14 subsequent siblings)
26 siblings, 0 replies; 29+ messages in thread
From: Paul Eggleton @ 2015-12-22 4:03 UTC (permalink / raw)
To: openembedded-core
Add a few clarifying words.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
scripts/lib/devtool/standard.py | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/scripts/lib/devtool/standard.py b/scripts/lib/devtool/standard.py
index c11de9d..57175a4 100644
--- a/scripts/lib/devtool/standard.py
+++ b/scripts/lib/devtool/standard.py
@@ -1181,7 +1181,7 @@ def edit_recipe(args, config, basepath, workspace):
def register_commands(subparsers, context):
"""Register devtool subcommands from this plugin"""
parser_add = subparsers.add_parser('add', help='Add a new recipe',
- description='Adds a new recipe to the workspace to build a specified source tree')
+ description='Adds a new recipe to the workspace to build a specified source tree. Can optionally fetch a remote URI and unpack it to create the source tree.')
parser_add.add_argument('recipename', help='Name for new recipe to add (just name - no version, path or extension)')
parser_add.add_argument('srctree', help='Path to external source tree')
group = parser_add.add_mutually_exclusive_group()
@@ -1190,7 +1190,7 @@ def register_commands(subparsers, context):
parser_add.add_argument('--fetch', '-f', help='Fetch the specified URI and extract it to create the source tree', metavar='URI')
parser_add.add_argument('--version', '-V', help='Version to use within recipe (PV)')
parser_add.add_argument('--no-git', '-g', help='If -f/--fetch is specified, do not set up source tree as a git repository', action="store_true")
- parser_add.add_argument('--binary', '-b', help='Treat the source tree as something that should be installed verbatim (no compilation, same directory structure)', action='store_true')
+ parser_add.add_argument('--binary', '-b', help='Treat the source tree as something that should be installed verbatim (no compilation, same directory structure). Useful with binary packages e.g. RPMs.', action='store_true')
parser_add.set_defaults(func=add)
parser_modify = subparsers.add_parser('modify', help='Modify the source for an existing recipe',
--
2.5.0
^ permalink raw reply related [flat|nested] 29+ messages in thread
* [PATCH 13/26] devtool: split out function for naming bbappend
2015-12-22 4:02 [PATCH 00/26] devtool/recipetool improvements Paul Eggleton
` (11 preceding siblings ...)
2015-12-22 4:03 ` [PATCH 12/26] devtool: add: tweak help text Paul Eggleton
@ 2015-12-22 4:03 ` Paul Eggleton
2015-12-22 4:03 ` [PATCH 14/26] devtool: update-recipe: use correct method to get bbappend filename Paul Eggleton
` (13 subsequent siblings)
26 siblings, 0 replies; 29+ messages in thread
From: Paul Eggleton @ 2015-12-22 4:03 UTC (permalink / raw)
To: openembedded-core
We're repeating this in a couple of places, so we might as well have a
function to do it.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
scripts/lib/devtool/__init__.py | 15 +++++++++++++++
scripts/lib/devtool/standard.py | 18 +++++-------------
2 files changed, 20 insertions(+), 13 deletions(-)
diff --git a/scripts/lib/devtool/__init__.py b/scripts/lib/devtool/__init__.py
index e617d60..7f16e17 100644
--- a/scripts/lib/devtool/__init__.py
+++ b/scripts/lib/devtool/__init__.py
@@ -22,6 +22,7 @@ import os
import sys
import subprocess
import logging
+import re
logger = logging.getLogger('devtool')
@@ -199,3 +200,17 @@ def setup_git_repo(repodir, version, devbranch, basetag='devtool-base'):
bb.process.run('git checkout -b %s' % devbranch, cwd=repodir)
bb.process.run('git tag -f %s' % basetag, cwd=repodir)
+
+def recipe_to_append(recipefile, config, wildcard=False):
+ """
+ Convert a recipe file to a bbappend file path within the workspace.
+ NOTE: if the bbappend already exists, you should be using
+ workspace[args.recipename]['bbappend'] instead of calling this
+ function.
+ """
+ appendname = os.path.splitext(os.path.basename(recipefile))[0]
+ if wildcard:
+ appendname = re.sub(r'_.*', '_%', appendname)
+ appendpath = os.path.join(config.workspace_path, 'appends')
+ appendfile = os.path.join(appendpath, appendname + '.bbappend')
+ return appendfile
diff --git a/scripts/lib/devtool/standard.py b/scripts/lib/devtool/standard.py
index 57175a4..0103d93 100644
--- a/scripts/lib/devtool/standard.py
+++ b/scripts/lib/devtool/standard.py
@@ -26,7 +26,7 @@ import argparse
import scriptutils
import errno
from collections import OrderedDict
-from devtool import exec_build_env_command, setup_tinfoil, check_workspace_recipe, use_external_build, setup_git_repo, DevtoolError
+from devtool import exec_build_env_command, setup_tinfoil, check_workspace_recipe, use_external_build, setup_git_repo, recipe_to_append, DevtoolError
from devtool import parse_recipe
logger = logging.getLogger('devtool')
@@ -64,10 +64,6 @@ def add(args, config, basepath, workspace):
raise DevtoolError("Specified source tree %s could not be found" %
srctree)
- appendpath = os.path.join(config.workspace_path, 'appends')
- if not os.path.exists(appendpath):
- os.makedirs(appendpath)
-
recipedir = os.path.join(config.workspace_path, 'recipes', args.recipename)
bb.utils.mkdirhier(recipedir)
rfv = None
@@ -121,7 +117,8 @@ def add(args, config, basepath, workspace):
if not rd:
return 1
- appendfile = os.path.join(appendpath, '%s.bbappend' % bp)
+ appendfile = recipe_to_append(recipefile, config)
+ bb.utils.mkdirhier(os.path.dirname(appendfile))
with open(appendfile, 'w') as f:
f.write('inherit externalsrc\n')
f.write('EXTERNALSRC = "%s"\n' % srctree)
@@ -584,11 +581,7 @@ def modify(args, config, basepath, workspace):
pn)
recipefile = rd.getVar('FILE', True)
- appendname = os.path.splitext(os.path.basename(recipefile))[0]
- if args.wildcard:
- appendname = re.sub(r'_.*', '_%', appendname)
- appendpath = os.path.join(config.workspace_path, 'appends')
- appendfile = os.path.join(appendpath, appendname + '.bbappend')
+ appendfile = recipe_to_append(recipefile, config, args.wildcard)
if os.path.exists(appendfile):
raise DevtoolError("Another variant of recipe %s is already in your "
"workspace (only one variant of a recipe can "
@@ -632,8 +625,7 @@ def modify(args, config, basepath, workspace):
srcsubdir = os.path.relpath(s, workdir).split(os.sep, 1)[1]
srctree = os.path.join(srctree, srcsubdir)
- if not os.path.exists(appendpath):
- os.makedirs(appendpath)
+ bb.utils.mkdirhier(os.path.dirname(appendfile))
with open(appendfile, 'w') as f:
f.write('FILESEXTRAPATHS_prepend := "${THISDIR}/${PN}:"\n')
# Local files can be modified/tracked in separate subdir under srctree
--
2.5.0
^ permalink raw reply related [flat|nested] 29+ messages in thread
* [PATCH 14/26] devtool: update-recipe: use correct method to get bbappend filename
2015-12-22 4:02 [PATCH 00/26] devtool/recipetool improvements Paul Eggleton
` (12 preceding siblings ...)
2015-12-22 4:03 ` [PATCH 13/26] devtool: split out function for naming bbappend Paul Eggleton
@ 2015-12-22 4:03 ` Paul Eggleton
2015-12-22 4:03 ` [PATCH 15/26] scripts/lib/argparse_oe: handle intermixing of optional positional arguments Paul Eggleton
` (12 subsequent siblings)
26 siblings, 0 replies; 29+ messages in thread
From: Paul Eggleton @ 2015-12-22 4:03 UTC (permalink / raw)
To: openembedded-core
The bbappend already exists at this point, so we know what its path is -
there's no need to figure it out from scratch here.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
scripts/lib/devtool/standard.py | 7 +++----
1 file changed, 3 insertions(+), 4 deletions(-)
diff --git a/scripts/lib/devtool/standard.py b/scripts/lib/devtool/standard.py
index 0103d93..195da1a 100644
--- a/scripts/lib/devtool/standard.py
+++ b/scripts/lib/devtool/standard.py
@@ -908,14 +908,13 @@ def _update_recipe_srcrev(args, srctree, rd, config_data):
_remove_source_files(args, remove_files, destpath)
-def _update_recipe_patch(args, config, srctree, rd, config_data):
+def _update_recipe_patch(args, config, workspace, srctree, rd, config_data):
"""Implement the 'patch' mode of update-recipe"""
import bb
import oe.recipeutils
recipefile = rd.getVar('FILE', True)
- append = os.path.join(config.workspace_path, 'appends', '%s.bbappend' %
- os.path.splitext(os.path.basename(recipefile))[0])
+ append = workspace[args.recipename]['bbappend']
if not os.path.exists(append):
raise DevtoolError('unable to find workspace bbappend for recipe %s' %
args.recipename)
@@ -1063,7 +1062,7 @@ def update_recipe(args, config, basepath, workspace):
if mode == 'srcrev':
_update_recipe_srcrev(args, srctree, rd, tinfoil.config_data)
elif mode == 'patch':
- _update_recipe_patch(args, config, srctree, rd, tinfoil.config_data)
+ _update_recipe_patch(args, config, workspace, srctree, rd, tinfoil.config_data)
else:
raise DevtoolError('update_recipe: invalid mode %s' % mode)
--
2.5.0
^ permalink raw reply related [flat|nested] 29+ messages in thread
* [PATCH 15/26] scripts/lib/argparse_oe: handle intermixing of optional positional arguments
2015-12-22 4:02 [PATCH 00/26] devtool/recipetool improvements Paul Eggleton
` (13 preceding siblings ...)
2015-12-22 4:03 ` [PATCH 14/26] devtool: update-recipe: use correct method to get bbappend filename Paul Eggleton
@ 2015-12-22 4:03 ` Paul Eggleton
2015-12-22 4:03 ` [PATCH 16/26] devtool: add: allow source tree to be omitted Paul Eggleton
` (11 subsequent siblings)
26 siblings, 0 replies; 29+ messages in thread
From: Paul Eggleton @ 2015-12-22 4:03 UTC (permalink / raw)
To: openembedded-core
Python's argparse module can't handle when several optional positional
arguments (set with nargs='?') are intermixed with other options. If the
positional arguments aren't optional then this isn't an issue; thus when
changing positional arguments to optional (as we are doing with devtool)
we need this workaround.
This is a pretty horrible hack, but we don't want this flexibility of
ordering to disappear simply because we made some arguments optional.
Unfortunately the corresponding bug remains unresolved upstream even in
Python 3, and argparse is not really designed to be subclassed so it
doesn't make things like this easy.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
scripts/lib/argparse_oe.py | 39 +++++++++++++++++++++++++++++++++++++++
1 file changed, 39 insertions(+)
diff --git a/scripts/lib/argparse_oe.py b/scripts/lib/argparse_oe.py
index c2fee6d..ec7eb8d 100644
--- a/scripts/lib/argparse_oe.py
+++ b/scripts/lib/argparse_oe.py
@@ -9,3 +9,42 @@ class ArgumentParser(argparse.ArgumentParser):
self.print_help()
sys.exit(2)
+ def add_subparsers(self, *args, **kwargs):
+ ret = super(ArgumentParser, self).add_subparsers(*args, **kwargs)
+ ret._parser_class = ArgumentSubParser
+ return ret
+
+class ArgumentSubParser(ArgumentParser):
+ def parse_known_args(self, args=None, namespace=None):
+ # This works around argparse not handling optional positional arguments being
+ # intermixed with other options. A pretty horrible hack, but we're not left
+ # with much choice given that the bug in argparse exists and it's difficult
+ # to subclass.
+ # Borrowed from http://stackoverflow.com/questions/20165843/argparse-how-to-handle-variable-number-of-arguments-nargs
+ # with an extra workaround (in format_help() below) for the positional
+ # arguments disappearing from the --help output, as well as structural tweaks.
+ # Originally simplified from http://bugs.python.org/file30204/test_intermixed.py
+ positionals = self._get_positional_actions()
+ for action in positionals:
+ # deactivate positionals
+ action.save_nargs = action.nargs
+ action.nargs = 0
+
+ namespace, remaining_args = super(ArgumentSubParser, self).parse_known_args(args, namespace)
+ for action in positionals:
+ # remove the empty positional values from namespace
+ if hasattr(namespace, action.dest):
+ delattr(namespace, action.dest)
+ for action in positionals:
+ action.nargs = action.save_nargs
+ # parse positionals
+ namespace, extras = super(ArgumentSubParser, self).parse_known_args(remaining_args, namespace)
+ return namespace, extras
+
+ def format_help(self):
+ # Quick, restore the positionals!
+ positionals = self._get_positional_actions()
+ for action in positionals:
+ if hasattr(action, 'save_nargs'):
+ action.nargs = action.save_nargs
+ return super(ArgumentParser, self).format_help()
--
2.5.0
^ permalink raw reply related [flat|nested] 29+ messages in thread
* [PATCH 16/26] devtool: add: allow source tree to be omitted
2015-12-22 4:02 [PATCH 00/26] devtool/recipetool improvements Paul Eggleton
` (14 preceding siblings ...)
2015-12-22 4:03 ` [PATCH 15/26] scripts/lib/argparse_oe: handle intermixing of optional positional arguments Paul Eggleton
@ 2015-12-22 4:03 ` Paul Eggleton
2015-12-22 4:03 ` [PATCH 17/26] devtool: add: figure out recipe name from recipetool Paul Eggleton
` (10 subsequent siblings)
26 siblings, 0 replies; 29+ messages in thread
From: Paul Eggleton @ 2015-12-22 4:03 UTC (permalink / raw)
To: openembedded-core
Assuming we're fetching source remotely (from a URI) we can default the
source tree that will be extracted from it to a "sources" directory
under the workspace in order to save the user specifying it if they
don't have a preferred location.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
scripts/lib/devtool/standard.py | 30 ++++++++++++++++++++++++++----
1 file changed, 26 insertions(+), 4 deletions(-)
diff --git a/scripts/lib/devtool/standard.py b/scripts/lib/devtool/standard.py
index 195da1a..b129db3 100644
--- a/scripts/lib/devtool/standard.py
+++ b/scripts/lib/devtool/standard.py
@@ -49,7 +49,12 @@ def add(args, config, basepath, workspace):
if '/' in args.recipename:
raise DevtoolError('"/" is not a valid character in recipe names')
- srctree = os.path.abspath(args.srctree)
+ if args.srctree:
+ srctree = os.path.abspath(args.srctree)
+ else:
+ srctree = get_default_srctree(config, args.recipename)
+ logger.info('Using default source tree path %s' % srctree)
+
if os.path.exists(srctree):
if args.fetch:
if not os.path.isdir(srctree):
@@ -61,8 +66,16 @@ def add(args, config, basepath, workspace):
"it already exists and is non-empty" %
srctree)
elif not args.fetch:
- raise DevtoolError("Specified source tree %s could not be found" %
- srctree)
+ if args.srctree:
+ raise DevtoolError("Specified source tree %s could not be found" %
+ args.srctree)
+ else:
+ raise DevtoolError("No source tree exists at default path %s - "
+ "either create and populate this directory, "
+ "or specify a path to a source tree, or use "
+ "the -f/--fetch option to fetch source code "
+ "and extract it to the source directory" %
+ srctree)
recipedir = os.path.join(config.workspace_path, 'recipes', args.recipename)
bb.utils.mkdirhier(recipedir)
@@ -1168,13 +1181,22 @@ def edit_recipe(args, config, basepath, workspace):
return 0
+def get_default_srctree(config, recipename=''):
+ """Get the default srctree path"""
+ srctreeparent = config.get('General', 'default_source_parent_dir', config.workspace_path)
+ if recipename:
+ return os.path.join(srctreeparent, 'sources', recipename)
+ else:
+ return os.path.join(srctreeparent, 'sources')
def register_commands(subparsers, context):
"""Register devtool subcommands from this plugin"""
+
+ defsrctree = get_default_srctree(context.config)
parser_add = subparsers.add_parser('add', help='Add a new recipe',
description='Adds a new recipe to the workspace to build a specified source tree. Can optionally fetch a remote URI and unpack it to create the source tree.')
parser_add.add_argument('recipename', help='Name for new recipe to add (just name - no version, path or extension)')
- parser_add.add_argument('srctree', help='Path to external source tree')
+ parser_add.add_argument('srctree', nargs='?', help='Path to external source tree. If not specified, a subdirectory of %s will be used.' % defsrctree)
group = parser_add.add_mutually_exclusive_group()
group.add_argument('--same-dir', '-s', help='Build in same directory as source', action="store_true")
group.add_argument('--no-same-dir', help='Force build in a separate build directory', action="store_true")
--
2.5.0
^ permalink raw reply related [flat|nested] 29+ messages in thread
* [PATCH 17/26] devtool: add: figure out recipe name from recipetool
2015-12-22 4:02 [PATCH 00/26] devtool/recipetool improvements Paul Eggleton
` (15 preceding siblings ...)
2015-12-22 4:03 ` [PATCH 16/26] devtool: add: allow source tree to be omitted Paul Eggleton
@ 2015-12-22 4:03 ` Paul Eggleton
2015-12-22 4:03 ` [PATCH 18/26] devtool: add: allow specifying URL as positional argument Paul Eggleton
` (9 subsequent siblings)
26 siblings, 0 replies; 29+ messages in thread
From: Paul Eggleton @ 2015-12-22 4:03 UTC (permalink / raw)
To: openembedded-core
recipetool create now has all the logic in it for auto-detecting the
name and version, and using those in the file name - so we can make the
name an optional parameter for devtool add and we pick up the file name
that recipetool has used after the fact.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
scripts/lib/devtool/standard.py | 127 ++++++++++++++++++++++++++++------------
1 file changed, 90 insertions(+), 37 deletions(-)
diff --git a/scripts/lib/devtool/standard.py b/scripts/lib/devtool/standard.py
index b129db3..6cdd44c 100644
--- a/scripts/lib/devtool/standard.py
+++ b/scripts/lib/devtool/standard.py
@@ -37,25 +37,39 @@ def add(args, config, basepath, workspace):
import bb
import oe.recipeutils
- if args.recipename in workspace:
- raise DevtoolError("recipe %s is already in your workspace" %
- args.recipename)
+ if args.recipename and not args.srctree:
+ # These are positional arguments, but because we're nice, allow
+ # specifying source tree without name if we can detect that that
+ # is what the user meant
+ if os.sep in args.recipename:
+ args.srctree = args.recipename
+ args.recipename = None
+ elif os.path.isdir(args.recipename):
+ logger.warn('Ambiguous argument %s - assuming you mean it to be the recipe name')
- reason = oe.recipeutils.validate_pn(args.recipename)
- if reason:
- raise DevtoolError(reason)
+ if args.recipename:
+ if args.recipename in workspace:
+ raise DevtoolError("recipe %s is already in your workspace" %
+ args.recipename)
+ reason = oe.recipeutils.validate_pn(args.recipename)
+ if reason:
+ raise DevtoolError(reason)
- # FIXME this ought to be in validate_pn but we're using that in other contexts
- if '/' in args.recipename:
- raise DevtoolError('"/" is not a valid character in recipe names')
+ # FIXME this ought to be in validate_pn but we're using that in other contexts
+ if '/' in args.recipename:
+ raise DevtoolError('"/" is not a valid character in recipe names')
if args.srctree:
srctree = os.path.abspath(args.srctree)
+ srctreeparent = None
+ tmpsrcdir = None
else:
- srctree = get_default_srctree(config, args.recipename)
- logger.info('Using default source tree path %s' % srctree)
+ srctree = None
+ srctreeparent = get_default_srctree(config)
+ bb.utils.mkdirhier(srctreeparent)
+ tmpsrcdir = tempfile.mkdtemp(prefix='devtoolsrc', dir=srctreeparent)
- if os.path.exists(srctree):
+ if srctree and os.path.exists(srctree):
if args.fetch:
if not os.path.isdir(srctree):
raise DevtoolError("Cannot fetch into source tree path %s as "
@@ -69,33 +83,21 @@ def add(args, config, basepath, workspace):
if args.srctree:
raise DevtoolError("Specified source tree %s could not be found" %
args.srctree)
- else:
+ elif srctree:
raise DevtoolError("No source tree exists at default path %s - "
"either create and populate this directory, "
"or specify a path to a source tree, or use "
"the -f/--fetch option to fetch source code "
"and extract it to the source directory" %
srctree)
+ else:
+ raise DevtoolError("You must either specify a source tree "
+ "or -f to fetch source")
- recipedir = os.path.join(config.workspace_path, 'recipes', args.recipename)
- bb.utils.mkdirhier(recipedir)
- rfv = None
if args.version:
if '_' in args.version or ' ' in args.version:
raise DevtoolError('Invalid version string "%s"' % args.version)
- rfv = args.version
- if args.fetch:
- if args.fetch.startswith('git://'):
- rfv = 'git'
- elif args.fetch.startswith('svn://'):
- rfv = 'svn'
- elif args.fetch.startswith('hg://'):
- rfv = 'hg'
- if rfv:
- bp = "%s_%s" % (args.recipename, rfv)
- else:
- bp = args.recipename
- recipefile = os.path.join(recipedir, "%s.bb" % bp)
+
if args.color == 'auto' and sys.stdout.isatty():
color = 'always'
else:
@@ -103,19 +105,71 @@ def add(args, config, basepath, workspace):
extracmdopts = ''
if args.fetch:
source = args.fetch
- extracmdopts = '-x %s' % srctree
+ if srctree:
+ extracmdopts += ' -x %s' % srctree
+ else:
+ extracmdopts += ' -x %s' % tmpsrcdir
else:
source = srctree
+ if args.recipename:
+ extracmdopts += ' -N %s' % args.recipename
if args.version:
extracmdopts += ' -V %s' % args.version
if args.binary:
extracmdopts += ' -b'
+
+ tempdir = tempfile.mkdtemp(prefix='devtool')
try:
- stdout, _ = exec_build_env_command(config.init_path, basepath, 'recipetool --color=%s create -o %s "%s" %s' % (color, recipefile, source, extracmdopts))
- except bb.process.ExecutionError as e:
- raise DevtoolError('Command \'%s\' failed:\n%s' % (e.command, e.stdout))
+ try:
+ stdout, _ = exec_build_env_command(config.init_path, basepath, 'recipetool --color=%s create -o %s "%s" %s' % (color, tempdir, source, extracmdopts))
+ except bb.process.ExecutionError as e:
+ if e.exitcode == 15:
+ raise DevtoolError('Unable to auto-determine name from source tree, please specify it with -N/--name')
+ else:
+ raise DevtoolError('Command \'%s\' failed:\n%s' % (e.command, e.stdout))
+
+ recipes = glob.glob(os.path.join(tempdir, '*.bb'))
+ if recipes:
+ recipename = os.path.splitext(os.path.basename(recipes[0]))[0].split('_')[0]
+ if recipename in workspace:
+ raise DevtoolError('A recipe with the same name as the one being created (%s) already exists in your workspace' % recipename)
+ recipedir = os.path.join(config.workspace_path, 'recipes', recipename)
+ bb.utils.mkdirhier(recipedir)
+ recipefile = os.path.join(recipedir, os.path.basename(recipes[0]))
+ appendfile = recipe_to_append(recipefile, config)
+ if os.path.exists(appendfile):
+ # This shouldn't be possible, but just in case
+ raise DevtoolError('A recipe with the same name as the one being created already exists in your workspace')
+ if os.path.exists(recipefile):
+ raise DevtoolError('A recipe file %s already exists in your workspace; this shouldn\'t be there - please delete it before continuing' % recipefile)
+ if tmpsrcdir:
+ srctree = os.path.join(srctreeparent, recipename)
+ if os.path.exists(tmpsrcdir):
+ if os.path.exists(srctree):
+ if os.path.isdir(srctree):
+ try:
+ os.rmdir(srctree)
+ except OSError as e:
+ if e.errno == errno.ENOTEMPTY:
+ raise DevtoolError('Source tree path %s already exists and is not empty' % srctree)
+ else:
+ raise
+ else:
+ raise DevtoolError('Source tree path %s already exists and is not a directory' % srctree)
+ logger.info('Using default source tree path %s' % srctree)
+ shutil.move(tmpsrcdir, srctree)
+ else:
+ raise DevtoolError('Couldn\'t find source tree created by recipetool')
+ bb.utils.mkdirhier(recipedir)
+ shutil.move(recipes[0], recipefile)
+ else:
+ raise DevtoolError('Command \'%s\' did not create any recipe file:\n%s' % (e.command, e.stdout))
+ finally:
+ if tmpsrcdir and os.path.exists(tmpsrcdir):
+ shutil.rmtree(tmpsrcdir)
+ shutil.rmtree(tempdir)
- _add_md5(config, args.recipename, recipefile)
+ _add_md5(config, recipename, recipefile)
if args.fetch and not args.no_git:
setup_git_repo(srctree, args.version, 'devtool')
@@ -130,7 +184,6 @@ def add(args, config, basepath, workspace):
if not rd:
return 1
- appendfile = recipe_to_append(recipefile, config)
bb.utils.mkdirhier(os.path.dirname(appendfile))
with open(appendfile, 'w') as f:
f.write('inherit externalsrc\n')
@@ -148,7 +201,7 @@ def add(args, config, basepath, workspace):
f.write(' rm -f ${D}/singletask.lock\n')
f.write('}\n')
- _add_md5(config, args.recipename, appendfile)
+ _add_md5(config, recipename, appendfile)
logger.info('Recipe %s has been automatically created; further editing may be required to make it fully functional' % recipefile)
@@ -1195,7 +1248,7 @@ def register_commands(subparsers, context):
defsrctree = get_default_srctree(context.config)
parser_add = subparsers.add_parser('add', help='Add a new recipe',
description='Adds a new recipe to the workspace to build a specified source tree. Can optionally fetch a remote URI and unpack it to create the source tree.')
- parser_add.add_argument('recipename', help='Name for new recipe to add (just name - no version, path or extension)')
+ parser_add.add_argument('recipename', nargs='?', help='Name for new recipe to add (just name - no version, path or extension). If not specified, will attempt to auto-detect it.')
parser_add.add_argument('srctree', nargs='?', help='Path to external source tree. If not specified, a subdirectory of %s will be used.' % defsrctree)
group = parser_add.add_mutually_exclusive_group()
group.add_argument('--same-dir', '-s', help='Build in same directory as source', action="store_true")
--
2.5.0
^ permalink raw reply related [flat|nested] 29+ messages in thread
* [PATCH 18/26] devtool: add: allow specifying URL as positional argument
2015-12-22 4:02 [PATCH 00/26] devtool/recipetool improvements Paul Eggleton
` (16 preceding siblings ...)
2015-12-22 4:03 ` [PATCH 17/26] devtool: add: figure out recipe name from recipetool Paul Eggleton
@ 2015-12-22 4:03 ` Paul Eggleton
2015-12-22 4:03 ` [PATCH 19/26] devtool: modify: default source tree path Paul Eggleton
` (8 subsequent siblings)
26 siblings, 0 replies; 29+ messages in thread
From: Paul Eggleton @ 2015-12-22 4:03 UTC (permalink / raw)
To: openembedded-core
Having to specify -f is a little bit ugly when a URI is distinctive
enough to recognise amongst the other positional parameters, so take it
as an optional positional parameter. -f/--fetch is still supported, but
deprecated.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
scripts/devtool | 2 ++
scripts/lib/argparse_oe.py | 16 +++++++++++++
scripts/lib/devtool/standard.py | 52 +++++++++++++++++++++++++++++------------
3 files changed, 55 insertions(+), 15 deletions(-)
diff --git a/scripts/devtool b/scripts/devtool
index 955495e..93ba58e 100755
--- a/scripts/devtool
+++ b/scripts/devtool
@@ -280,6 +280,8 @@ def main():
if str(err):
logger.error(str(err))
ret = 1
+ except argparse_oe.ArgumentUsageError as ae:
+ parser.error_subcommand(ae.message, ae.subcommand)
return ret
diff --git a/scripts/lib/argparse_oe.py b/scripts/lib/argparse_oe.py
index ec7eb8d..fd86692 100644
--- a/scripts/lib/argparse_oe.py
+++ b/scripts/lib/argparse_oe.py
@@ -1,6 +1,12 @@
import sys
import argparse
+class ArgumentUsageError(Exception):
+ """Exception class you can raise (and catch) in order to show the help"""
+ def __init__(self, message, subcommand=None):
+ self.message = message
+ self.subcommand = subcommand
+
class ArgumentParser(argparse.ArgumentParser):
"""Our own version of argparse's ArgumentParser"""
@@ -9,6 +15,16 @@ class ArgumentParser(argparse.ArgumentParser):
self.print_help()
sys.exit(2)
+ def error_subcommand(self, message, subcommand):
+ if subcommand:
+ for action in self._actions:
+ if isinstance(action, argparse._SubParsersAction):
+ for choice, subparser in action.choices.items():
+ if choice == subcommand:
+ subparser.error(message)
+ return
+ self.error(message)
+
def add_subparsers(self, *args, **kwargs):
ret = super(ArgumentParser, self).add_subparsers(*args, **kwargs)
ret._parser_class = ArgumentSubParser
diff --git a/scripts/lib/devtool/standard.py b/scripts/lib/devtool/standard.py
index 6cdd44c..e249661 100644
--- a/scripts/lib/devtool/standard.py
+++ b/scripts/lib/devtool/standard.py
@@ -23,6 +23,7 @@ import shutil
import tempfile
import logging
import argparse
+import argparse_oe
import scriptutils
import errno
from collections import OrderedDict
@@ -37,16 +38,38 @@ def add(args, config, basepath, workspace):
import bb
import oe.recipeutils
- if args.recipename and not args.srctree:
- # These are positional arguments, but because we're nice, allow
- # specifying source tree without name if we can detect that that
- # is what the user meant
+ if not args.recipename and not args.srctree and not args.fetch and not args.fetchuri:
+ raise argparse_oe.ArgumentUsageError('At least one of recipename, srctree, fetchuri or -f/--fetch must be specified', 'add')
+
+ # These are positional arguments, but because we're nice, allow
+ # specifying e.g. source tree without name, or fetch URI without name or
+ # source tree (if we can detect that that is what the user meant)
+ if '://' in args.recipename:
+ if not args.fetchuri:
+ if args.fetch:
+ raise DevtoolError('URI specified as positional argument as well as -f/--fetch')
+ args.fetchuri = args.recipename
+ args.recipename = ''
+ elif '://' in args.srctree:
+ if not args.fetchuri:
+ if args.fetch:
+ raise DevtoolError('URI specified as positional argument as well as -f/--fetch')
+ args.fetchuri = args.srctree
+ args.srctree = ''
+ elif args.recipename and not args.srctree:
if os.sep in args.recipename:
args.srctree = args.recipename
args.recipename = None
elif os.path.isdir(args.recipename):
logger.warn('Ambiguous argument %s - assuming you mean it to be the recipe name')
+ if args.fetch:
+ if args.fetchuri:
+ raise DevtoolError('URI specified as positional argument as well as -f/--fetch')
+ else:
+ # FIXME should show a warning that -f/--fetch is deprecated here
+ args.fetchuri = args.fetch
+
if args.recipename:
if args.recipename in workspace:
raise DevtoolError("recipe %s is already in your workspace" %
@@ -70,7 +93,7 @@ def add(args, config, basepath, workspace):
tmpsrcdir = tempfile.mkdtemp(prefix='devtoolsrc', dir=srctreeparent)
if srctree and os.path.exists(srctree):
- if args.fetch:
+ if args.fetchuri:
if not os.path.isdir(srctree):
raise DevtoolError("Cannot fetch into source tree path %s as "
"it exists and is not a directory" %
@@ -79,20 +102,18 @@ def add(args, config, basepath, workspace):
raise DevtoolError("Cannot fetch into source tree path %s as "
"it already exists and is non-empty" %
srctree)
- elif not args.fetch:
+ elif not args.fetchuri:
if args.srctree:
raise DevtoolError("Specified source tree %s could not be found" %
args.srctree)
elif srctree:
raise DevtoolError("No source tree exists at default path %s - "
"either create and populate this directory, "
- "or specify a path to a source tree, or use "
- "the -f/--fetch option to fetch source code "
- "and extract it to the source directory" %
- srctree)
+ "or specify a path to a source tree, or a "
+ "URI to fetch source from" % srctree)
else:
raise DevtoolError("You must either specify a source tree "
- "or -f to fetch source")
+ "or a URI to fetch source from")
if args.version:
if '_' in args.version or ' ' in args.version:
@@ -103,8 +124,8 @@ def add(args, config, basepath, workspace):
else:
color = args.color
extracmdopts = ''
- if args.fetch:
- source = args.fetch
+ if args.fetchuri:
+ source = args.fetchuri
if srctree:
extracmdopts += ' -x %s' % srctree
else:
@@ -1250,12 +1271,13 @@ def register_commands(subparsers, context):
description='Adds a new recipe to the workspace to build a specified source tree. Can optionally fetch a remote URI and unpack it to create the source tree.')
parser_add.add_argument('recipename', nargs='?', help='Name for new recipe to add (just name - no version, path or extension). If not specified, will attempt to auto-detect it.')
parser_add.add_argument('srctree', nargs='?', help='Path to external source tree. If not specified, a subdirectory of %s will be used.' % defsrctree)
+ parser_add.add_argument('fetchuri', nargs='?', help='Fetch the specified URI and extract it to create the source tree')
group = parser_add.add_mutually_exclusive_group()
group.add_argument('--same-dir', '-s', help='Build in same directory as source', action="store_true")
group.add_argument('--no-same-dir', help='Force build in a separate build directory', action="store_true")
- parser_add.add_argument('--fetch', '-f', help='Fetch the specified URI and extract it to create the source tree', metavar='URI')
+ parser_add.add_argument('--fetch', '-f', help='Fetch the specified URI and extract it to create the source tree (deprecated - pass as positional argument instead)', metavar='URI')
parser_add.add_argument('--version', '-V', help='Version to use within recipe (PV)')
- parser_add.add_argument('--no-git', '-g', help='If -f/--fetch is specified, do not set up source tree as a git repository', action="store_true")
+ parser_add.add_argument('--no-git', '-g', help='If fetching source, do not set up source tree as a git repository', action="store_true")
parser_add.add_argument('--binary', '-b', help='Treat the source tree as something that should be installed verbatim (no compilation, same directory structure). Useful with binary packages e.g. RPMs.', action='store_true')
parser_add.set_defaults(func=add)
--
2.5.0
^ permalink raw reply related [flat|nested] 29+ messages in thread
* [PATCH 19/26] devtool: modify: default source tree path
2015-12-22 4:02 [PATCH 00/26] devtool/recipetool improvements Paul Eggleton
` (17 preceding siblings ...)
2015-12-22 4:03 ` [PATCH 18/26] devtool: add: allow specifying URL as positional argument Paul Eggleton
@ 2015-12-22 4:03 ` Paul Eggleton
2015-12-22 4:03 ` [PATCH 20/26] devtool: status: list recipe file within workspace if one exists Paul Eggleton
` (7 subsequent siblings)
26 siblings, 0 replies; 29+ messages in thread
From: Paul Eggleton @ 2015-12-22 4:03 UTC (permalink / raw)
To: openembedded-core
As per the changes to "devtool add", make the source tree path optional
and use the default path if none is specified.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
scripts/devtool | 4 ++++
scripts/lib/devtool/standard.py | 26 +++++++++++++++-----------
2 files changed, 19 insertions(+), 11 deletions(-)
diff --git a/scripts/devtool b/scripts/devtool
index 93ba58e..bda05e1 100755
--- a/scripts/devtool
+++ b/scripts/devtool
@@ -152,6 +152,10 @@ def _create_workspace(workspacedir, config, basepath):
f.write('\nIf you no longer need to use devtool you can remove the path to this\n')
f.write('workspace layer from your conf/bblayers.conf file (and then delete the\n')
f.write('layer, if you wish).\n')
+ f.write('\nNote that by default, if devtool fetches and unpacks source code, it\n')
+ f.write('will place it in a subdirectory of a "sources" subdirectory of the\n')
+ f.write('layer. If you prefer it to be elsewhere you can specify the source\n')
+ f.write('tree path on the command line.\n')
def _enable_workspace_layer(workspacedir, config, basepath):
"""Ensure the workspace layer is in bblayers.conf"""
diff --git a/scripts/lib/devtool/standard.py b/scripts/lib/devtool/standard.py
index e249661..c710c16 100644
--- a/scripts/lib/devtool/standard.py
+++ b/scripts/lib/devtool/standard.py
@@ -644,10 +644,15 @@ def modify(args, config, basepath, workspace):
raise DevtoolError("recipe %s is already in your workspace" %
args.recipename)
- if not args.extract and not os.path.isdir(args.srctree):
+ if args.srctree:
+ srctree = os.path.abspath(args.srctree)
+ else:
+ srctree = get_default_srctree(config, args.recipename)
+
+ if not args.extract and not os.path.isdir(srctree):
raise DevtoolError("directory %s does not exist or not a directory "
"(specify -x to extract source from recipe)" %
- args.srctree)
+ srctree)
if args.extract:
tinfoil = _prep_extract_operation(config, basepath, args.recipename)
if not tinfoil:
@@ -679,29 +684,28 @@ def modify(args, config, basepath, workspace):
initial_rev = None
commits = []
- srctree = os.path.abspath(args.srctree)
if args.extract:
- initial_rev = _extract_source(args.srctree, False, args.branch, False, rd)
+ initial_rev = _extract_source(srctree, False, args.branch, False, rd)
if not initial_rev:
return 1
logger.info('Source tree extracted to %s' % srctree)
# Get list of commits since this revision
- (stdout, _) = bb.process.run('git rev-list --reverse %s..HEAD' % initial_rev, cwd=args.srctree)
+ (stdout, _) = bb.process.run('git rev-list --reverse %s..HEAD' % initial_rev, cwd=srctree)
commits = stdout.split()
else:
- if os.path.exists(os.path.join(args.srctree, '.git')):
+ if os.path.exists(os.path.join(srctree, '.git')):
# Check if it's a tree previously extracted by us
try:
- (stdout, _) = bb.process.run('git branch --contains devtool-base', cwd=args.srctree)
+ (stdout, _) = bb.process.run('git branch --contains devtool-base', cwd=srctree)
except bb.process.ExecutionError:
stdout = ''
for line in stdout.splitlines():
if line.startswith('*'):
- (stdout, _) = bb.process.run('git rev-parse devtool-base', cwd=args.srctree)
+ (stdout, _) = bb.process.run('git rev-parse devtool-base', cwd=srctree)
initial_rev = stdout.rstrip()
if not initial_rev:
# Otherwise, just grab the head revision
- (stdout, _) = bb.process.run('git rev-parse HEAD', cwd=args.srctree)
+ (stdout, _) = bb.process.run('git rev-parse HEAD', cwd=srctree)
initial_rev = stdout.rstrip()
# Check that recipe isn't using a shared workdir
@@ -1282,9 +1286,9 @@ def register_commands(subparsers, context):
parser_add.set_defaults(func=add)
parser_modify = subparsers.add_parser('modify', help='Modify the source for an existing recipe',
- description='Enables modifying the source for an existing recipe')
+ description='Enables modifying the source for an existing recipe. You can either provide your own pre-prepared source tree, or specify -x/--extract to extract the source being fetched by the recipe.')
parser_modify.add_argument('recipename', help='Name of existing recipe to edit (just name - no version, path or extension)')
- parser_modify.add_argument('srctree', help='Path to external source tree')
+ parser_modify.add_argument('srctree', nargs='?', help='Path to external source tree. If not specified, a subdirectory of %s will be used.' % defsrctree)
parser_modify.add_argument('--wildcard', '-w', action="store_true", help='Use wildcard for unversioned bbappend')
parser_modify.add_argument('--extract', '-x', action="store_true", help='Extract source as well')
group = parser_modify.add_mutually_exclusive_group()
--
2.5.0
^ permalink raw reply related [flat|nested] 29+ messages in thread
* [PATCH 20/26] devtool: status: list recipe file within workspace if one exists
2015-12-22 4:02 [PATCH 00/26] devtool/recipetool improvements Paul Eggleton
` (18 preceding siblings ...)
2015-12-22 4:03 ` [PATCH 19/26] devtool: modify: default source tree path Paul Eggleton
@ 2015-12-22 4:03 ` Paul Eggleton
2015-12-22 4:03 ` [PATCH 21/26] devtool: reset: print message about leaving source tree behind Paul Eggleton
` (6 subsequent siblings)
26 siblings, 0 replies; 29+ messages in thread
From: Paul Eggleton @ 2015-12-22 4:03 UTC (permalink / raw)
To: openembedded-core
If a recipe in the workspace actually exists as a file within the
workspace (e.g. after doing "devtool add" or "devtool upgrade") then
show the path to the recipe file on the status line for the recipe.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
scripts/lib/devtool/standard.py | 12 +++++++++++-
1 file changed, 11 insertions(+), 1 deletion(-)
diff --git a/scripts/lib/devtool/standard.py b/scripts/lib/devtool/standard.py
index c710c16..ed9e793 100644
--- a/scripts/lib/devtool/standard.py
+++ b/scripts/lib/devtool/standard.py
@@ -26,6 +26,7 @@ import argparse
import argparse_oe
import scriptutils
import errno
+import glob
from collections import OrderedDict
from devtool import exec_build_env_command, setup_tinfoil, check_workspace_recipe, use_external_build, setup_git_repo, recipe_to_append, DevtoolError
from devtool import parse_recipe
@@ -1168,7 +1169,16 @@ def status(args, config, basepath, workspace):
"""Entry point for the devtool 'status' subcommand"""
if workspace:
for recipe, value in workspace.iteritems():
- print("%s: %s" % (recipe, value['srctree']))
+ bbfile = os.path.basename(value['bbappend']).replace('.bbappend', '.bb').replace('%', '*')
+ recipefile = glob.glob(os.path.join(config.workspace_path,
+ 'recipes',
+ recipe,
+ bbfile))
+ if recipefile:
+ recipestr = ' (%s)' % recipefile[0]
+ else:
+ recipestr = ''
+ print("%s: %s%s" % (recipe, value['srctree'], recipestr))
else:
logger.info('No recipes currently in your workspace - you can use "devtool modify" to work on an existing recipe or "devtool add" to add a new one')
return 0
--
2.5.0
^ permalink raw reply related [flat|nested] 29+ messages in thread
* [PATCH 21/26] devtool: reset: print message about leaving source tree behind
2015-12-22 4:02 [PATCH 00/26] devtool/recipetool improvements Paul Eggleton
` (19 preceding siblings ...)
2015-12-22 4:03 ` [PATCH 20/26] devtool: status: list recipe file within workspace if one exists Paul Eggleton
@ 2015-12-22 4:03 ` Paul Eggleton
2015-12-22 4:03 ` [PATCH 22/26] devtool: sync: tweak help / messages Paul Eggleton
` (5 subsequent siblings)
26 siblings, 0 replies; 29+ messages in thread
From: Paul Eggleton @ 2015-12-22 4:03 UTC (permalink / raw)
To: openembedded-core
We deliberately leave the source tree alone when resetting in case it
contains any work in progress belonging to the user; tell them that
we're doing this so they aren't surprised about it still existing later
on.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
scripts/lib/devtool/standard.py | 11 +++++++++++
1 file changed, 11 insertions(+)
diff --git a/scripts/lib/devtool/standard.py b/scripts/lib/devtool/standard.py
index ed9e793..a9c4cf8 100644
--- a/scripts/lib/devtool/standard.py
+++ b/scripts/lib/devtool/standard.py
@@ -1229,6 +1229,17 @@ def reset(args, config, basepath, workspace):
# We don't automatically create this dir next to appends, but the user can
preservedir(os.path.join(config.workspace_path, 'appends', pn))
+ srctree = workspace[pn]['srctree']
+ if os.path.isdir(srctree):
+ if os.listdir(srctree):
+ # We don't want to risk wiping out any work in progress
+ logger.info('Leaving source tree %s as-is; if you no '
+ 'longer need it then please delete it manually'
+ % srctree)
+ else:
+ # This is unlikely, but if it's empty we can just remove it
+ os.rmdir(srctree)
+
return 0
--
2.5.0
^ permalink raw reply related [flat|nested] 29+ messages in thread
* [PATCH 22/26] devtool: sync: tweak help / messages
2015-12-22 4:02 [PATCH 00/26] devtool/recipetool improvements Paul Eggleton
` (20 preceding siblings ...)
2015-12-22 4:03 ` [PATCH 21/26] devtool: reset: print message about leaving source tree behind Paul Eggleton
@ 2015-12-22 4:03 ` Paul Eggleton
2015-12-22 4:03 ` [PATCH 23/26] recipetool: create: improve autotools support Paul Eggleton
` (4 subsequent siblings)
26 siblings, 0 replies; 29+ messages in thread
From: Paul Eggleton @ 2015-12-22 4:03 UTC (permalink / raw)
To: openembedded-core
Much of this was copy/pasted from the extract subcommand code; make it
specific to sync.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
scripts/lib/devtool/standard.py | 10 +++++-----
1 file changed, 5 insertions(+), 5 deletions(-)
diff --git a/scripts/lib/devtool/standard.py b/scripts/lib/devtool/standard.py
index a9c4cf8..fbc49a9 100644
--- a/scripts/lib/devtool/standard.py
+++ b/scripts/lib/devtool/standard.py
@@ -346,7 +346,7 @@ def sync(args, config, basepath, workspace):
srctree = os.path.abspath(args.srctree)
initial_rev = _extract_source(srctree, args.keep_temp, args.branch, True, rd)
- logger.info('Source tree extracted to %s' % srctree)
+ logger.info('Source tree %s synchronized' % srctree)
if initial_rev:
return 0
@@ -1326,11 +1326,11 @@ def register_commands(subparsers, context):
parser_extract.add_argument('--keep-temp', action="store_true", help='Keep temporary directory (for debugging)')
parser_extract.set_defaults(func=extract, no_workspace=True)
- parser_sync = subparsers.add_parser('sync', help='Synchronize the source for an existing recipe',
- description='Synchronize the source for an existing recipe',
+ parser_sync = subparsers.add_parser('sync', help='Synchronize the source tree for an existing recipe',
+ description='Synchronize the previously extracted source tree for an existing recipe',
formatter_class=argparse.ArgumentDefaultsHelpFormatter)
- parser_sync.add_argument('recipename', help='Name for recipe to sync the source for')
- parser_sync.add_argument('srctree', help='Path to where to sync the source tree')
+ parser_sync.add_argument('recipename', help='Name of recipe to sync the source for')
+ parser_sync.add_argument('srctree', help='Path to the source tree')
parser_sync.add_argument('--branch', '-b', default="devtool", help='Name for development branch to checkout')
parser_sync.add_argument('--keep-temp', action="store_true", help='Keep temporary directory (for debugging)')
parser_sync.set_defaults(func=sync)
--
2.5.0
^ permalink raw reply related [flat|nested] 29+ messages in thread
* [PATCH 23/26] recipetool: create: improve autotools support
2015-12-22 4:02 [PATCH 00/26] devtool/recipetool improvements Paul Eggleton
` (21 preceding siblings ...)
2015-12-22 4:03 ` [PATCH 22/26] devtool: sync: tweak help / messages Paul Eggleton
@ 2015-12-22 4:03 ` Paul Eggleton
2015-12-22 4:03 ` [PATCH 24/26] recipetool: create: fix error when extracting source to a specified directory Paul Eggleton
` (3 subsequent siblings)
26 siblings, 0 replies; 29+ messages in thread
From: Paul Eggleton @ 2015-12-22 4:03 UTC (permalink / raw)
To: openembedded-core
* tar and binutils we can assume are there
* libsocket is only relevant on BSD systems, so we can ignore it.
* Detect more things implying gettext/intltool is needed
* Detect glib-2.0 requirement.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
scripts/lib/recipetool/create_buildsys.py | 38 ++++++++++++++++++++++---------
1 file changed, 27 insertions(+), 11 deletions(-)
diff --git a/scripts/lib/recipetool/create_buildsys.py b/scripts/lib/recipetool/create_buildsys.py
index 0aff59e..1674aba 100644
--- a/scripts/lib/recipetool/create_buildsys.py
+++ b/scripts/lib/recipetool/create_buildsys.py
@@ -127,6 +127,7 @@ class AutotoolsRecipeHandler(RecipeHandler):
extravalues['PN'] = pn
if autoconf:
+ lines_before.append('')
lines_before.append('# NOTE: if this software is not capable of being built in a separate build directory')
lines_before.append('# from the source, you should replace autotools with autotools-brokensep in the')
lines_before.append('# inherit line')
@@ -150,11 +151,14 @@ class AutotoolsRecipeHandler(RecipeHandler):
# FIXME this mapping is very thin
progmap = {'flex': 'flex-native',
'bison': 'bison-native',
- 'm4': 'm4-native'}
+ 'm4': 'm4-native',
+ 'tar': 'tar-native',
+ 'ar': 'binutils-native'}
progclassmap = {'gconftool-2': 'gconf',
'pkg-config': 'pkgconfig'}
- ignoredeps = ['gcc-runtime', 'glibc', 'uclibc']
+ ignoredeps = ['gcc-runtime', 'glibc', 'uclibc', 'tar-native', 'binutils-native']
+ ignorelibs = ['socket']
pkg_re = re.compile('PKG_CHECK_MODULES\(\[?[a-zA-Z0-9]*\]?, \[?([^,\]]*)[),].*')
lib_re = re.compile('AC_CHECK_LIB\(\[?([a-zA-Z0-9]*)\]?, .*')
@@ -245,8 +249,12 @@ class AutotoolsRecipeHandler(RecipeHandler):
if res:
pcdeps.extend([x[0] for x in res])
inherits.append('pkgconfig')
- elif keyword == 'AM_GNU_GETTEXT':
+ elif keyword in ('AM_GNU_GETTEXT', 'AM_GLIB_GNU_GETTEXT', 'GETTEXT_PACKAGE'):
inherits.append('gettext')
+ elif keyword in ('AC_PROG_INTLTOOL', 'IT_PROG_INTLTOOL'):
+ deps.append('intltool-native')
+ elif keyword == 'AM_PATH_GLIB_2_0':
+ deps.append('glib-2.0')
elif keyword == 'AC_CHECK_PROG' or keyword == 'AC_PATH_PROG':
res = progs_re.search(value)
if res:
@@ -266,13 +274,16 @@ class AutotoolsRecipeHandler(RecipeHandler):
res = lib_re.search(value)
if res:
lib = res.group(1)
- libdep = recipelibmap.get(lib, None)
- if libdep:
- deps.append(libdep)
+ if lib in ignorelibs:
+ logger.debug('Ignoring library dependency %s' % lib)
else:
- if libdep is None:
- if not lib.startswith('$'):
- unmappedlibs.append(lib)
+ libdep = recipelibmap.get(lib, None)
+ if libdep:
+ deps.append(libdep)
+ else:
+ if libdep is None:
+ if not lib.startswith('$'):
+ unmappedlibs.append(lib)
elif keyword == 'AC_PATH_X':
deps.append('libx11')
elif keyword == 'AC_INIT':
@@ -303,6 +314,11 @@ class AutotoolsRecipeHandler(RecipeHandler):
keywords = ['PKG_CHECK_MODULES',
'AM_GNU_GETTEXT',
+ 'AM_GLIB_GNU_GETTEXT',
+ 'GETTEXT_PACKAGE',
+ 'AC_PROG_INTLTOOL',
+ 'IT_PROG_INTLTOOL',
+ 'AM_PATH_GLIB_2_0',
'AC_CHECK_PROG',
'AC_PATH_PROG',
'AC_CHECK_LIB',
@@ -351,10 +367,10 @@ class AutotoolsRecipeHandler(RecipeHandler):
extravalues[k] = v.strip('"\'').rstrip('()')
if unmapped:
- outlines.append('# NOTE: the following prog dependencies are unknown, ignoring: %s' % ' '.join(unmapped))
+ outlines.append('# NOTE: the following prog dependencies are unknown, ignoring: %s' % ' '.join(list(set(unmapped))))
if unmappedlibs:
- outlines.append('# NOTE: the following library dependencies are unknown, ignoring: %s' % ' '.join(unmappedlibs))
+ outlines.append('# NOTE: the following library dependencies are unknown, ignoring: %s' % ' '.join(list(set(unmappedlibs))))
outlines.append('# (this is based on recipes that have previously been built and packaged)')
recipemap = read_pkgconfig_provides(tinfoil.config_data)
--
2.5.0
^ permalink raw reply related [flat|nested] 29+ messages in thread
* [PATCH 24/26] recipetool: create: fix error when extracting source to a specified directory
2015-12-22 4:02 [PATCH 00/26] devtool/recipetool improvements Paul Eggleton
` (22 preceding siblings ...)
2015-12-22 4:03 ` [PATCH 23/26] recipetool: create: improve autotools support Paul Eggleton
@ 2015-12-22 4:03 ` Paul Eggleton
2015-12-22 4:03 ` [PATCH 25/26] oe-selftest: add tests for simple devtool add / recipetool create URL case Paul Eggleton
` (2 subsequent siblings)
26 siblings, 0 replies; 29+ messages in thread
From: Paul Eggleton @ 2015-12-22 4:03 UTC (permalink / raw)
To: openembedded-core
Having fetched the source and unpacked it to a temporary directory, we
then move part of it to the destination directory, or if the source is at
the top level we move the whole temporary directory, but in the latter
case we were later attempting to delete the temporary directory which no
longer existed. Clear out the variable so that doesn't happen.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
scripts/lib/recipetool/create.py | 2 ++
1 file changed, 2 insertions(+)
diff --git a/scripts/lib/recipetool/create.py b/scripts/lib/recipetool/create.py
index 4887604..873b7dd 100644
--- a/scripts/lib/recipetool/create.py
+++ b/scripts/lib/recipetool/create.py
@@ -404,6 +404,8 @@ def create_recipe(args):
# to just remove it first
os.rmdir(args.extract_to)
shutil.move(srctree, args.extract_to)
+ if tempsrc == srctree:
+ tempsrc = None
logger.info('Source extracted to %s' % args.extract_to)
if outfile == '-':
--
2.5.0
^ permalink raw reply related [flat|nested] 29+ messages in thread
* [PATCH 25/26] oe-selftest: add tests for simple devtool add / recipetool create URL case
2015-12-22 4:02 [PATCH 00/26] devtool/recipetool improvements Paul Eggleton
` (23 preceding siblings ...)
2015-12-22 4:03 ` [PATCH 24/26] recipetool: create: fix error when extracting source to a specified directory Paul Eggleton
@ 2015-12-22 4:03 ` Paul Eggleton
2015-12-22 4:03 ` [PATCH 26/26] oe-selftest: devtool: add more explicit check for ls output Paul Eggleton
2015-12-24 2:45 ` [PATCH 00/26] devtool/recipetool improvements Khem Raj
26 siblings, 0 replies; 29+ messages in thread
From: Paul Eggleton @ 2015-12-22 4:03 UTC (permalink / raw)
To: openembedded-core
Add an oe-selftest test case for the newly supported syntax with only
the remote URL specified (auto-detecting name and version).
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
meta/lib/oeqa/selftest/devtool.py | 63 ++++++++++++++++++++++++++++++++----
meta/lib/oeqa/selftest/recipetool.py | 19 +++++++++++
2 files changed, 76 insertions(+), 6 deletions(-)
diff --git a/meta/lib/oeqa/selftest/devtool.py b/meta/lib/oeqa/selftest/devtool.py
index 7af82df..41e1b4b 100644
--- a/meta/lib/oeqa/selftest/devtool.py
+++ b/meta/lib/oeqa/selftest/devtool.py
@@ -15,18 +15,43 @@ class DevtoolBase(oeSelfTest):
def _test_recipe_contents(self, recipefile, checkvars, checkinherits):
with open(recipefile, 'r') as f:
+ invar = None
+ invalue = None
for line in f:
- if '=' in line:
+ var = None
+ if invar:
+ value = line.strip().strip('"')
+ if value.endswith('\\'):
+ invalue += ' ' + value[:-1].strip()
+ continue
+ else:
+ invalue += ' ' + value.strip()
+ var = invar
+ value = invalue
+ invar = None
+ elif '=' in line:
splitline = line.split('=', 1)
var = splitline[0].rstrip()
value = splitline[1].strip().strip('"')
- if var in checkvars:
- needvalue = checkvars.pop(var)
- self.assertEqual(value, needvalue, 'values for %s do not match' % var)
- if line.startswith('inherit '):
+ if value.endswith('\\'):
+ invalue = value[:-1].strip()
+ invar = var
+ continue
+ elif line.startswith('inherit '):
inherits = line.split()[1:]
- self.assertEqual(checkvars, {}, 'Some variables not found: %s' % checkvars)
+ if var and var in checkvars:
+ needvalue = checkvars.pop(var)
+ if needvalue is None:
+ self.fail('Variable %s should not appear in recipe')
+ self.assertEqual(value, needvalue, 'values for %s do not match' % var)
+
+
+ missingvars = {}
+ for var, value in checkvars.iteritems():
+ if value is not None:
+ missingvars[var] = value
+ self.assertEqual(missingvars, {}, 'Some expected variables not found in recipe: %s' % checkvars)
for inherit in checkinherits:
self.assertIn(inherit, inherits, 'Missing inherit of %s' % inherit)
@@ -304,6 +329,32 @@ class DevtoolTests(DevtoolBase):
checkvars['SRCREV'] = checkrev
self._test_recipe_contents(recipefile, checkvars, [])
+ def test_devtool_add_fetch_simple(self):
+ # Fetch source from a remote URL, auto-detecting name
+ tempdir = tempfile.mkdtemp(prefix='devtoolqa')
+ self.track_for_cleanup(tempdir)
+ testver = '1.6.0'
+ url = 'http://www.ivarch.com/programs/sources/pv-%s.tar.bz2' % testver
+ testrecipe = 'pv'
+ srcdir = os.path.join(self.workspacedir, 'sources', testrecipe)
+ # Test devtool add
+ self.track_for_cleanup(self.workspacedir)
+ self.add_command_to_tearDown('bitbake-layers remove-layer */workspace')
+ result = runCmd('devtool add %s' % url)
+ self.assertTrue(os.path.exists(os.path.join(self.workspacedir, 'conf', 'layer.conf')), 'Workspace directory not created. %s' % result.output)
+ self.assertTrue(os.path.isfile(os.path.join(srcdir, 'configure')), 'Unable to find configure script in source directory')
+ # Test devtool status
+ result = runCmd('devtool status')
+ self.assertIn(testrecipe, result.output)
+ self.assertIn(srcdir, result.output)
+ # Check recipe
+ recipefile = get_bb_var('FILE', testrecipe)
+ self.assertIn('%s_%s.bb' % (testrecipe, testver), recipefile, 'Recipe file incorrectly named')
+ checkvars = {}
+ checkvars['S'] = None
+ checkvars['SRC_URI'] = url.replace(testver, '${PV}')
+ self._test_recipe_contents(recipefile, checkvars, [])
+
@testcase(1164)
def test_devtool_modify(self):
# Clean up anything in the workdir/sysroot/sstate cache
diff --git a/meta/lib/oeqa/selftest/recipetool.py b/meta/lib/oeqa/selftest/recipetool.py
index bbfce7c..34e383f 100644
--- a/meta/lib/oeqa/selftest/recipetool.py
+++ b/meta/lib/oeqa/selftest/recipetool.py
@@ -401,6 +401,25 @@ class RecipetoolTests(RecipetoolBase):
inherits = ['autotools', 'pkgconfig']
self._test_recipe_contents(recipefile, checkvars, inherits)
+ def test_recipetool_create_simple(self):
+ # Try adding a recipe
+ temprecipe = os.path.join(self.tempdir, 'recipe')
+ os.makedirs(temprecipe)
+ pv = '1.7.3.0'
+ srcuri = 'http://www.dest-unreach.org/socat/download/socat-%s.tar.bz2' % pv
+ result = runCmd('recipetool create %s -o %s' % (srcuri, temprecipe))
+ dirlist = os.listdir(temprecipe)
+ if len(dirlist) < 1 or not os.path.isfile(os.path.join(temprecipe, dirlist[0])):
+ self.fail('recipetool did not create recipe file; output:\n%s' % result.output)
+ self.assertEqual(dirlist[0], 'socat_%s.bb' % pv, 'Recipe file incorrectly named')
+ checkvars = {}
+ checkvars['LICENSE'] = 'Unknown GPLv2'
+ checkvars['LIC_FILES_CHKSUM'] = 'file://COPYING.OpenSSL;md5=5c9bccc77f67a8328ef4ebaf468116f4 file://COPYING;md5=b234ee4d69f5fce4486a80fdaf4a4263'
+ # We don't check DEPENDS since they are variable for this recipe depending on what's in the sysroot
+ checkvars['S'] = None
+ checkvars['SRC_URI'] = srcuri.replace(pv, '${PV}')
+ inherits = ['autotools']
+ self._test_recipe_contents(os.path.join(temprecipe, dirlist[0]), checkvars, inherits)
class RecipetoolAppendsrcBase(RecipetoolBase):
def _try_recipetool_appendsrcfile(self, testrecipe, newfile, destfile, options, expectedlines, expectedfiles):
--
2.5.0
^ permalink raw reply related [flat|nested] 29+ messages in thread
* [PATCH 26/26] oe-selftest: devtool: add more explicit check for ls output
2015-12-22 4:02 [PATCH 00/26] devtool/recipetool improvements Paul Eggleton
` (24 preceding siblings ...)
2015-12-22 4:03 ` [PATCH 25/26] oe-selftest: add tests for simple devtool add / recipetool create URL case Paul Eggleton
@ 2015-12-22 4:03 ` Paul Eggleton
2015-12-24 2:45 ` [PATCH 00/26] devtool/recipetool improvements Khem Raj
26 siblings, 0 replies; 29+ messages in thread
From: Paul Eggleton @ 2015-12-22 4:03 UTC (permalink / raw)
To: openembedded-core
test_devtool_deploy_target is failing on the Yocto Project autobuilder
apparently when it attempts to cut out some fields from the list. It
doesn't fail here and I can't see what the problem lines are, so add a
check for lines with too few fields so we can get a look at them next
time it fails.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
meta/lib/oeqa/selftest/devtool.py | 2 ++
1 file changed, 2 insertions(+)
diff --git a/meta/lib/oeqa/selftest/devtool.py b/meta/lib/oeqa/selftest/devtool.py
index 41e1b4b..955379b 100644
--- a/meta/lib/oeqa/selftest/devtool.py
+++ b/meta/lib/oeqa/selftest/devtool.py
@@ -93,6 +93,8 @@ class DevtoolBase(oeSelfTest):
filelist = []
for line in output.splitlines():
splitline = line.split()
+ if len(splitline) < 8:
+ self.fail('_process_ls_output: invalid output line: %s' % line)
# Remove trailing . on perms
splitline[0] = splitline[0].rstrip('.')
# Remove leading . on paths
--
2.5.0
^ permalink raw reply related [flat|nested] 29+ messages in thread
* Re: [PATCH 00/26] devtool/recipetool improvements
2015-12-22 4:02 [PATCH 00/26] devtool/recipetool improvements Paul Eggleton
` (25 preceding siblings ...)
2015-12-22 4:03 ` [PATCH 26/26] oe-selftest: devtool: add more explicit check for ls output Paul Eggleton
@ 2015-12-24 2:45 ` Khem Raj
2015-12-28 19:03 ` Paul Eggleton
26 siblings, 1 reply; 29+ messages in thread
From: Khem Raj @ 2015-12-24 2:45 UTC (permalink / raw)
To: Paul Eggleton; +Cc: Patches and discussions about the oe-core layer
Paul
Can this series be backported to jethro as well ?
most of them seems to be incremental changes
hopefully not breaking backward compatibility in some
unforseen ways
On Mon, Dec 21, 2015 at 8:02 PM, Paul Eggleton
<paul.eggleton@linux.intel.com> wrote:
> Apologies for the flood of patches today, this is the last one.
>
> There are quite a few changes here, but the common theme is making
> devtool (and recipetool) a bit easier to use, as well as making
> "devtool add" and "recipetool create" more effective at picking up
> things from the specified source tree.
>
> I also extended the argparse fix to show the help text on error to all
> of the command-line utilities that use it.
>
>
> The following changes since commit 2a1edfd9cfa16ec334c0758b47677d4fee5e79a8:
>
> bitbake.conf: Add filename and lineno to BB_SIGNATURE_EXCLUDE_FLAGS (2015-12-22 00:01:31 +0000)
>
> are available in the git repository at:
>
> git://git.openembedded.org/openembedded-core-contrib paule/devtool11
> http://cgit.openembedded.org/cgit.cgi/openembedded-core-contrib/log/?h=paule/devtool11
>
> Paul Eggleton (26):
> scripts: print usage in argparse-using scripts when a command-line error occurs
> recipetool: create: handle https://....git URLs
> recipetool: create: avoid traceback on fetch error
> recipetool: create: fix do_install handling for makefile-only software
> recipetool: create: minor fix for potential issue in python handling
> recipetool: create: prevent attempting to unpack entire DL_DIR
> recipetool: create: detect when specified URL returns a web page
> recipetool: create: set up priority system for recipe handlers
> recipetool: create: support extracting name and version from build scripts
> recipetool: create: basic extraction of name/version from filename
> devtool: edit-recipe: add new subcommand
> devtool: add: tweak help text
> devtool: split out function for naming bbappend
> devtool: update-recipe: use correct method to get bbappend filename
> scripts/lib/argparse_oe: handle intermixing of optional positional arguments
> devtool: add: allow source tree to be omitted
> devtool: add: figure out recipe name from recipetool
> devtool: add: allow specifying URL as positional argument
> devtool: modify: default source tree path
> devtool: status: list recipe file within workspace if one exists
> devtool: reset: print message about leaving source tree behind
> devtool: sync: tweak help / messages
> recipetool: create: improve autotools support
> recipetool: create: fix error when extracting source to a specified directory
> oe-selftest: add tests for simple devtool add / recipetool create URL case
> oe-selftest: devtool: add more explicit check for ls output
>
> meta/lib/oeqa/selftest/devtool.py | 82 +++++-
> meta/lib/oeqa/selftest/recipetool.py | 21 +-
> scripts/contrib/devtool-stress.py | 5 +-
> scripts/devtool | 13 +-
> scripts/lib/argparse_oe.py | 66 +++++
> scripts/lib/devtool/__init__.py | 15 ++
> scripts/lib/devtool/standard.py | 315 +++++++++++++++++------
> scripts/lib/recipetool/create.py | 206 ++++++++++++---
> scripts/lib/recipetool/create_buildsys.py | 296 +++++++++++++++++----
> scripts/lib/recipetool/create_buildsys_python.py | 7 +-
> scripts/lib/scriptutils.py | 5 +
> scripts/oe-pkgdata-util | 5 +-
> scripts/oe-publish-sdk | 5 +-
> scripts/oe-selftest | 3 +-
> scripts/recipetool | 7 +-
> scripts/send-error-report | 6 +-
> scripts/test-remote-image | 3 +-
> 17 files changed, 872 insertions(+), 188 deletions(-)
> create mode 100644 scripts/lib/argparse_oe.py
>
> --
> 2.5.0
>
> --
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-core
^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: [PATCH 00/26] devtool/recipetool improvements
2015-12-24 2:45 ` [PATCH 00/26] devtool/recipetool improvements Khem Raj
@ 2015-12-28 19:03 ` Paul Eggleton
0 siblings, 0 replies; 29+ messages in thread
From: Paul Eggleton @ 2015-12-28 19:03 UTC (permalink / raw)
To: Khem Raj; +Cc: Patches and discussions about the oe-core layer
On Wed, 23 Dec 2015 18:45:50 Khem Raj wrote:
> Can this series be backported to jethro as well ?
> most of them seems to be incremental changes
> hopefully not breaking backward compatibility in some
> unforseen ways
Some of it should be OK, but I'd be nervous about backporting the entire
series. I'll prepare another jethro series soon (though I'm still waiting for
my last set to be merged.)
Cheers,
Paul
--
Paul Eggleton
Intel Open Source Technology Centre
^ permalink raw reply [flat|nested] 29+ messages in thread
end of thread, other threads:[~2015-12-28 19:03 UTC | newest]
Thread overview: 29+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2015-12-22 4:02 [PATCH 00/26] devtool/recipetool improvements Paul Eggleton
2015-12-22 4:02 ` [PATCH 01/26] scripts: print usage in argparse-using scripts when a command-line error occurs Paul Eggleton
2015-12-22 4:02 ` [PATCH 02/26] recipetool: create: handle https://....git URLs Paul Eggleton
2015-12-22 4:02 ` [PATCH 03/26] recipetool: create: avoid traceback on fetch error Paul Eggleton
2015-12-22 4:02 ` [PATCH 04/26] recipetool: create: fix do_install handling for makefile-only software Paul Eggleton
2015-12-22 4:02 ` [PATCH 05/26] recipetool: create: minor fix for potential issue in python handling Paul Eggleton
2015-12-22 4:02 ` [PATCH 06/26] recipetool: create: prevent attempting to unpack entire DL_DIR Paul Eggleton
2015-12-22 4:03 ` [PATCH 07/26] recipetool: create: detect when specified URL returns a web page Paul Eggleton
2015-12-22 4:03 ` [PATCH 08/26] recipetool: create: set up priority system for recipe handlers Paul Eggleton
2015-12-22 4:03 ` [PATCH 09/26] recipetool: create: support extracting name and version from build scripts Paul Eggleton
2015-12-22 4:03 ` [PATCH 10/26] recipetool: create: basic extraction of name/version from filename Paul Eggleton
2015-12-22 4:03 ` [PATCH 11/26] devtool: edit-recipe: add new subcommand Paul Eggleton
2015-12-22 4:03 ` [PATCH 12/26] devtool: add: tweak help text Paul Eggleton
2015-12-22 4:03 ` [PATCH 13/26] devtool: split out function for naming bbappend Paul Eggleton
2015-12-22 4:03 ` [PATCH 14/26] devtool: update-recipe: use correct method to get bbappend filename Paul Eggleton
2015-12-22 4:03 ` [PATCH 15/26] scripts/lib/argparse_oe: handle intermixing of optional positional arguments Paul Eggleton
2015-12-22 4:03 ` [PATCH 16/26] devtool: add: allow source tree to be omitted Paul Eggleton
2015-12-22 4:03 ` [PATCH 17/26] devtool: add: figure out recipe name from recipetool Paul Eggleton
2015-12-22 4:03 ` [PATCH 18/26] devtool: add: allow specifying URL as positional argument Paul Eggleton
2015-12-22 4:03 ` [PATCH 19/26] devtool: modify: default source tree path Paul Eggleton
2015-12-22 4:03 ` [PATCH 20/26] devtool: status: list recipe file within workspace if one exists Paul Eggleton
2015-12-22 4:03 ` [PATCH 21/26] devtool: reset: print message about leaving source tree behind Paul Eggleton
2015-12-22 4:03 ` [PATCH 22/26] devtool: sync: tweak help / messages Paul Eggleton
2015-12-22 4:03 ` [PATCH 23/26] recipetool: create: improve autotools support Paul Eggleton
2015-12-22 4:03 ` [PATCH 24/26] recipetool: create: fix error when extracting source to a specified directory Paul Eggleton
2015-12-22 4:03 ` [PATCH 25/26] oe-selftest: add tests for simple devtool add / recipetool create URL case Paul Eggleton
2015-12-22 4:03 ` [PATCH 26/26] oe-selftest: devtool: add more explicit check for ls output Paul Eggleton
2015-12-24 2:45 ` [PATCH 00/26] devtool/recipetool improvements Khem Raj
2015-12-28 19:03 ` Paul Eggleton
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.