[PATCHv4 0/4] devtool: add: automatically set branch for git URLs if not master

[PATCHv4 0/4] devtool: add: automatically set branch for git URLs if not master

[Chang Rebecca Swee Fun]

Hi Ross,

Change log on v4:
Rebased to latest master HEAD.
Broke up Stanley's commit into 2 separate commits as they fix different issues.
And updated my patch on disabling PREMIRRORS and MIRRORS by setting them in
dummy recipe instead to not affecting fetcher process of other dependencies recipes.

Change log on v3:
Rebased to latest master HEAD

Thanks.

Regards,
Rebecca

Chang Rebecca Swee Fun (2):
  recipetool: create: disable PREMIRRORS and MIRRORS by default
  recipetool: create: being able to set branch when revision is provided

Stanley Phoong (2):
  recipetool: create: handle git URLs specifying only a tag
  recipetool: create: replacing PV in SRCURI

 scripts/lib/recipetool/create.py | 75 ++++++++++++++++++++++++++++++++++++++--
 scripts/lib/scriptutils.py       |  9 ++++-
 2 files changed, 81 insertions(+), 3 deletions(-)

-- 
2.7.4

[PATCHv4 1/4] recipetool: create: disable PREMIRRORS and MIRRORS by default

[Chang Rebecca Swee Fun]

When creating new recipes, we are almost certainly fetching a new
source rather that something that has already been fetched. I have
disable PREMIRRORS and MIRRORS settings in the recipe that created
by devtool while leaving an option for users to enable them manually
if needed.

Signed-off-by: Chang Rebecca Swee Fun <rebecca.swee.fun.chang@intel.com>
---
 scripts/lib/recipetool/create.py | 3 ++-
 scripts/lib/scriptutils.py       | 9 ++++++++-
 2 files changed, 10 insertions(+), 2 deletions(-)

diff --git a/scripts/lib/recipetool/create.py b/scripts/lib/recipetool/create.py
index 359eb9a..f6ea422 100644
--- a/scripts/lib/recipetool/create.py
+++ b/scripts/lib/recipetool/create.py
@@ -446,7 +446,7 @@ def create_recipe(args):
         srctree = os.path.join(tempsrc, 'source')
 
         try:
-            checksums, ftmpdir = scriptutils.fetch_url(tinfoil, srcuri, srcrev, srctree, logger, preserve_tmp=args.keep_temp)
+            checksums, ftmpdir = scriptutils.fetch_url(tinfoil, srcuri, srcrev, srctree, logger, preserve_tmp=args.keep_temp, mirrors=args.mirrors)
         except scriptutils.FetchUrlFailure as e:
             logger.error(str(e))
             sys.exit(1)
@@ -1169,5 +1169,6 @@ def register_commands(subparsers):
     parser_create.add_argument('--keep-temp', action="store_true", help='Keep temporary directory (for debugging)')
     parser_create.add_argument('--fetch-dev', action="store_true", help='For npm, also fetch devDependencies')
     parser_create.add_argument('--devtool', action="store_true", help=argparse.SUPPRESS)
+    parser_create.add_argument('--mirrors', action="store_true", help='Enable PREMIRRORS and MIRRORS for source tree fetching (disable by default).')
     parser_create.set_defaults(func=create_recipe)
 
diff --git a/scripts/lib/scriptutils.py b/scripts/lib/scriptutils.py
index 9785438..11f1a78 100644
--- a/scripts/lib/scriptutils.py
+++ b/scripts/lib/scriptutils.py
@@ -102,7 +102,7 @@ class FetchUrlFailure(Exception):
     def __str__(self):
         return "Failed to fetch URL %s" % self.url
 
-def fetch_url(tinfoil, srcuri, srcrev, destdir, logger, preserve_tmp=False):
+def fetch_url(tinfoil, srcuri, srcrev, destdir, logger, preserve_tmp=False, mirrors=False):
     """
     Fetch the specified URL using normal do_fetch and do_unpack tasks, i.e.
     any dependencies that need to be satisfied in order to support the fetch
@@ -150,6 +150,13 @@ def fetch_url(tinfoil, srcuri, srcrev, destdir, logger, preserve_tmp=False):
                 f.write('WORKDIR = "%s"\n' % tmpworkdir)
                 # Set S out of the way so it doesn't get created under the workdir
                 f.write('S = "%s"\n' % os.path.join(tmpdir, 'emptysrc'))
+                if not mirrors:
+                    # We do not need PREMIRRORS since we are almost certainly
+                    # fetching new source rather than something that has already
+                    # been fetched. Hence, we disable them by default.
+                    # However, we provide an option for users to enable it.
+                    f.write('PREMIRRORS = ""\n')
+                    f.write('MIRRORS = ""\n')
 
             logger.info('Fetching %s...' % srcuri)
 
-- 
2.7.4

[PATCHv4 2/4] recipetool: create: being able to set branch when revision is provided

[Chang Rebecca Swee Fun]

When recipetool create is run on a git URL and a revision specified
that is not on master, and "branch=" isn't already in the URL, then
we should get the correct branch and append the branch to the URL.

If the revision was found on multiple branches, we will display error
to inform user to provide a correct branch and exit.

[YOCTO #11389]

Signed-off-by: Chang Rebecca Swee Fun <rebecca.swee.fun.chang@intel.com>
---
 scripts/lib/recipetool/create.py | 41 ++++++++++++++++++++++++++++++++++++++++
 1 file changed, 41 insertions(+)

diff --git a/scripts/lib/recipetool/create.py b/scripts/lib/recipetool/create.py
index f6ea422..4016a54 100644
--- a/scripts/lib/recipetool/create.py
+++ b/scripts/lib/recipetool/create.py
@@ -422,6 +422,8 @@ def create_recipe(args):
     source = args.source
     srcsubdir = ''
     srcrev = '${AUTOREV}'
+    branch_re = re.compile(';branch=([^;]+)')
+    branch = ''
 
     if os.path.isfile(source):
         source = 'file://%s' % os.path.abspath(source)
@@ -440,6 +442,19 @@ def create_recipe(args):
             srcrev = res.group(1)
             srcuri = rev_re.sub('', srcuri)
 
+        # Check whether branch info is provided
+        branch = branch_re.search(srcuri)
+        nobranch_re = re.compile(';nobranch=1')
+        nobranch = nobranch_re.search(srcuri)
+        # Back up a copy of srcuri
+        srcuri_copy = srcuri
+        if not branch and not nobranch and srcrev != '${AUTOREV}':
+            # Append nobranch=1 in the following conditions:
+            # 1. User did not set 'branch=' in srcuri, and
+            # 2. User did not set 'nobranch=1' in srcuri, and
+            # 3. Source revision is not '${AUTOREV}'
+            srcuri = srcuri + ';nobranch=1'
+
         tmpparent = tinfoil.config_data.getVar('BASE_WORKDIR')
         bb.utils.mkdirhier(tmpparent)
         tempsrc = tempfile.mkdtemp(prefix='recipetool-', dir=tmpparent)
@@ -475,6 +490,27 @@ def create_recipe(args):
             logger.error('URL %s resulted in an empty source tree' % fetchuri)
             sys.exit(1)
 
+        # Check for branch info with SRCREV provided
+        if not branch and not nobranch and srcrev and (srcrev != '${AUTOREV}'):
+            # Command to check branch using commit hash
+            cmd = 'git branch -r --contains'
+            try:
+                check_branch, check_branch_err = bb.process.run('%s %s' % (cmd, srcrev), cwd=srctree)
+            except bb.process.ExecutionError as err:
+                logger.error(str(err))
+                sys.exit(1)
+            get_branch = [x.strip() for x in check_branch.splitlines()]
+            # Remove HEAD reference point and drop remote prefix
+            get_branch = [x.split('/', 1)[1] for x in get_branch if not x.startswith('origin/HEAD')]
+            if len(get_branch) == 1:
+                # If get_branch contains only ONE object, then store result into 'branch'
+                branch = get_branch[0]
+            else:
+                # If get_branch contains more than one objects, then display error and exit.
+                mbrch = '\n  ' + '\n  '.join(get_branch)
+                logger.error('Revision %s was found on multiple branches: %s\nPlease provide the correct branch in the source URL with ;branch=<branch> (and ensure you use quotes around the URL to avoid the shell interpreting the ";")' % (srcrev, mbrch))
+                sys.exit(1)
+
         if os.path.exists(os.path.join(srctree, '.gitmodules')) and srcuri.startswith('git://'):
             srcuri = 'gitsm://' + srcuri[6:]
             logger.info('Fetching submodules...')
@@ -595,6 +631,11 @@ def create_recipe(args):
 
     if not srcuri:
         lines_before.append('# No information for SRC_URI yet (only an external source tree was specified)')
+    # Check if users has provide a branch
+    append_branch = branch_re.search(srcuri)
+    if branch and not append_branch:
+        # Append the correct branch into SRC_URI
+        srcuri = srcuri_copy + (';branch=%s' % str(branch))
     lines_before.append('SRC_URI = "%s"' % srcuri)
     for key, value in sorted(checksums.items()):
         lines_before.append('SRC_URI[%s] = "%s"' % (key, value))
-- 
2.7.4

[PATCHv4 3/4] recipetool: create: handle git URLs specifying only a tag

[Chang Rebecca Swee Fun]

From: Stanley Phoong <stanley.cheong.kwan.phoong@intel.com>

If a git URL is passed to recipetool create with a tag=, recipetool
should handle it assuming that the tag is valid.

[YOCTO #11393]

Signed-off-by: Stanley Phoong <stanley.cheong.kwan.phoong@intel.com>
---
 scripts/lib/recipetool/create.py | 33 ++++++++++++++++++++++++++++++---
 1 file changed, 30 insertions(+), 3 deletions(-)

diff --git a/scripts/lib/recipetool/create.py b/scripts/lib/recipetool/create.py
index 4016a54..f0bea66 100644
--- a/scripts/lib/recipetool/create.py
+++ b/scripts/lib/recipetool/create.py
@@ -455,6 +455,19 @@ def create_recipe(args):
             # 3. Source revision is not '${AUTOREV}'
             srcuri = srcuri + ';nobranch=1'
 
+        # Create an empty storeTagName to ensure the checkers do not point to a null variable
+        storeTagName = ''
+        tag_re = re.compile(';tag=([^;]+)')
+        tag = tag_re.search(srcuri)
+        if tag:
+            scheme, host, path, user, pswd, parms = bb.fetch2.decodeurl(srcuri)
+            # Keep a copy of tag and append nobranch=1 then remove tag from URL,
+            # Will re-introduce tag argument after bitbake fetcher process is complete.
+            storeTagName = parms['tag']
+            parms.update({('nobranch', '1')})
+            del parms['tag']
+            srcuri = bb.fetch2.encodeurl((scheme, host, path, user, pswd, parms))
+
         tmpparent = tinfoil.config_data.getVar('BASE_WORKDIR')
         bb.utils.mkdirhier(tmpparent)
         tempsrc = tempfile.mkdtemp(prefix='recipetool-', dir=tmpparent)
@@ -511,6 +524,18 @@ def create_recipe(args):
                 logger.error('Revision %s was found on multiple branches: %s\nPlease provide the correct branch in the source URL with ;branch=<branch> (and ensure you use quotes around the URL to avoid the shell interpreting the ";")' % (srcrev, mbrch))
                 sys.exit(1)
 
+        if storeTagName:
+            # Re-introduced tag variable from storeTagName
+            # Check srcrev using tag and check validity of the tag
+            try:
+                cmd = ('git rev-list -n 1 %s' % (storeTagName))
+                check_tag, check_tag_err = bb.process.run('%s' % cmd, cwd=srctree)
+                srcrev = check_tag.split()[0]
+            except bb.process.ExecutionError as err:
+                logger.error(str(err))
+                logger.error("Possibly wrong tag name is provided")
+                sys.exit(1)
+
         if os.path.exists(os.path.join(srctree, '.gitmodules')) and srcuri.startswith('git://'):
             srcuri = 'gitsm://' + srcuri[6:]
             logger.info('Fetching submodules...')
@@ -633,9 +658,11 @@ def create_recipe(args):
         lines_before.append('# No information for SRC_URI yet (only an external source tree was specified)')
     # Check if users has provide a branch
     append_branch = branch_re.search(srcuri)
-    if branch and not append_branch:
-        # Append the correct branch into SRC_URI
-        srcuri = srcuri_copy + (';branch=%s' % str(branch))
+    if (branch and not append_branch) or storeTagName:
+        srcuri = srcuri_copy
+        if branch:
+            # Append the correct branch into SRC_URI
+            srcuri = srcuri + (';branch=%s' % str(branch))
     lines_before.append('SRC_URI = "%s"' % srcuri)
     for key, value in sorted(checksums.items()):
         lines_before.append('SRC_URI[%s] = "%s"' % (key, value))
-- 
2.7.4

[PATCHv4 4/4] recipetool: create: replacing PV in SRCURI

[Chang Rebecca Swee Fun]

From: Stanley Phoong <stanley.cheong.kwan.phoong@intel.com>

During recipe creation, it seems that the automation for replacing
${PV} at the SRCURI for tag, (e.g mbed-tls-${PV}) is causing some
issue due to PV assuming it's a git source. A fix is implemented in
this patch to resolve this issue.

Signed-off-by: Stanley Phoong <stanley.cheong.kwan.phoong@intel.com>
---
 scripts/lib/recipetool/create.py | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/scripts/lib/recipetool/create.py b/scripts/lib/recipetool/create.py
index f0bea66..27f9c9c 100644
--- a/scripts/lib/recipetool/create.py
+++ b/scripts/lib/recipetool/create.py
@@ -424,6 +424,7 @@ def create_recipe(args):
     srcrev = '${AUTOREV}'
     branch_re = re.compile(';branch=([^;]+)')
     branch = ''
+    pv_srcpv = False
 
     if os.path.isfile(source):
         source = 'file://%s' % os.path.abspath(source)
@@ -670,6 +671,7 @@ def create_recipe(args):
         lines_before.append('')
         lines_before.append('# Modify these as desired')
         lines_before.append('PV = "%s+git${SRCPV}"' % (realpv or '1.0'))
+        pv_srcpv = True
         if not args.autorev and srcrev == '${AUTOREV}':
             if os.path.exists(os.path.join(srctree, '.git')):
                 (stdout, _) = bb.process.run('git rev-parse HEAD', cwd=srctree)
@@ -796,7 +798,7 @@ def create_recipe(args):
                 skipblank = True
                 continue
         elif line.startswith('SRC_URI = '):
-            if realpv:
+            if realpv and not pv_srcpv:
                 line = line.replace(realpv, '${PV}')
         elif line.startswith('PV = '):
             if realpv:
-- 
2.7.4

Re: [PATCHv4 3/4] recipetool: create: handle git URLs specifying only a tag

[Burton, Ross]

[-- Attachment #1: Type: text/plain, Size: 1469 bytes --]

The eSDK tests are failing with this applied:

oeqa.utils.subprocesstweak.OETestCalledProcessError: Command '.
/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-x86-64/build/build/tmp/work/qemux86_64-poky-linux/core-image-minimal/1.0-r0/testsdkext/environment-setup-core2-64-poky-linux
> /dev/null;

devtool add myapp
/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-x86-64/build/build/tmp/work/qemux86_64-poky-linux/core-image-minimal/1.0-r0/testsdkext/myapp;

returned non-zero exit status 1

Standard Output: NOTE: Starting bitbake server...
NOTE: Starting bitbake server...
Traceback (most recent call last):
  File
"/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-x86-64/build/build/tmp/work/qemux86_64-poky-linux/core-image-minimal/1.0-r0/testsdkext/layers/build/scripts/recipetool",
line 121, in <module>
    ret = main()
  File
"/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-x86-64/build/build/tmp/work/qemux86_64-poky-linux/core-image-minimal/1.0-r0/testsdkext/layers/build/scripts/recipetool",
line 110, in main
    ret = args.func(args)
  File
"/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-x86-64/build/build/tmp/work/qemux86_64-poky-linux/core-image-minimal/1.0-r0/testsdkext/layers/build/scripts/lib/recipetool/create.py",
line 662, in create_recipe
    if (branch and not append_branch) or storeTagName:
UnboundLocalError: local variable 'storeTagName' referenced before
assignment

Ross

[-- Attachment #2: Type: text/html, Size: 1643 bytes --]

Re: [PATCHv4 2/4] recipetool: create: being able to set branch when revision is provided

[Burton, Ross]

[-- Attachment #1: Type: text/plain, Size: 1917 bytes --]

On 27 July 2017 at 09:40, Chang Rebecca Swee Fun <
rebecca.swee.fun.chang@intel.com> wrote:

> When recipetool create is run on a git URL and a revision specified
> that is not on master, and "branch=" isn't already in the URL, then
> we should get the correct branch and append the branch to the URL.
>
> If the revision was found on multiple branches, we will display error
> to inform user to provide a correct branch and exit.
>

I think this is causing the selftests to fail:

2017-08-02 04:16:40,204 - oe-selftest - INFO - FAIL [12.984s]:
test_recipetool_create_git (recipetool.RecipetoolTests)
2017-08-02 04:16:40,204 - oe-selftest - INFO -
----------------------------------------------------------------------
2017-08-02 04:16:40,205 - oe-selftest - INFO - Traceback (most recent call
last):
  File
"/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/meta/lib/oeqa/core/decorator/__init__.py",
line 32, in wrapped_f
    return func(*args, **kwargs)
  File
"/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/meta/lib/oeqa/selftest/cases/recipetool.py",
line 403, in test_recipetool_create_git
    self._test_recipe_contents(recipefile, checkvars, inherits)
  File
"/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/meta/lib/oeqa/selftest/cases/devtool.py",
line 52, in _test_recipe_contents
    self.assertEqual(value, needvalue, 'values for %s do not match' % var)
AssertionError: 'git://git.yoctoproject.org/libmatchbox;branch=master' !=
'git://git.yoctoproject.org/libmatchbox'
- git://git.yoctoproject.org/libmatchbox;branch=master
?                                       --------------
+ git://git.yoctoproject.org/libmatchbox

 : values for SRC_URI do not match

Looks like recipetool is adding branch=master even though that is the
default value so I'd suggest should be filtered out.

Ross

[-- Attachment #2: Type: text/html, Size: 2793 bytes --]