[PATCH 0/3] recipetool: bug fixes and enhancement

[PATCH 0/3] recipetool: bug fixes and enhancement

[Chang Rebecca Swee Fun]

Hi,

This series of patches are bug fix implementation for 11389 and 11393.
These changes are crutial in terms of improving recipe creation in
recipetool. The recipe should at least buildable and able to find
SRCREV on specifc branch/ tag if it is not master.

This patch series is dependent on 2 patches on mailing list:-
recipetool: create: fix incorrect URL variable usage
http://lists.openembedded.org/pipermail/openembedded-core/2017-August/140789.html

and

recipetool: create: disable PREMIRRORS and MIRRORS by default
http://lists.openembedded.org/pipermail/openembedded-core/2017-August/140618.html

Thanks and best regards,
Rebecca

Chang Rebecca Swee Fun (1):
  recipetool: create: being able to set branch when revision is provided

Stanley Phoong (2):
  recipetool: create: handle git URLs specifying only a tag
  recipetool: create: replacing PV in SRCURI

 scripts/lib/recipetool/create.py | 73 +++++++++++++++++++++++++++++++++++++++-
 1 file changed, 72 insertions(+), 1 deletion(-)

-- 
2.7.4

[PATCH 1/3] recipetool: create: being able to set branch when revision is provided

[Chang Rebecca Swee Fun]

This change is to improve the buildability of the recipe created by
recipetool and devtool.

When recipetool create is run on a git URL and a revision specified
that is not on master, and "branch=" isn't already in the URL, then
we should get the correct branch and append the branch to the URL.

If the revision was found on multiple branches and 'master' is not
in the list, we will display error to inform user to provide a
correct branch and exit.

[YOCTO #11389]

Signed-off-by: Chang Rebecca Swee Fun <rebecca.swee.fun.chang@intel.com>
---
 scripts/lib/recipetool/create.py | 48 ++++++++++++++++++++++++++++++++++++++++
 1 file changed, 48 insertions(+)

diff --git a/scripts/lib/recipetool/create.py b/scripts/lib/recipetool/create.py
index c47ad78..d2e6fc0 100644
--- a/scripts/lib/recipetool/create.py
+++ b/scripts/lib/recipetool/create.py
@@ -422,6 +422,7 @@ def create_recipe(args):
     source = args.source
     srcsubdir = ''
     srcrev = '${AUTOREV}'
+    srcbranch = ''
 
     if os.path.isfile(source):
         source = 'file://%s' % os.path.abspath(source)
@@ -440,6 +441,19 @@ def create_recipe(args):
             srcrev = res.group(1)
             srcuri = rev_re.sub('', srcuri)
 
+        # Check whether users provides any branch info in fetchuri.
+        # If true, we will skip all branch checking process to honor all user's input.
+        scheme, network, path, user, passwd, params = bb.fetch2.decodeurl(fetchuri)
+        srcbranch = params.get('branch')
+        nobranch = params.get('nobranch')
+        if not srcbranch and not nobranch and srcrev != '${AUTOREV}':
+            # Append nobranch=1 in the following conditions:
+            # 1. User did not set 'branch=' in srcuri, and
+            # 2. User did not set 'nobranch=1' in srcuri, and
+            # 3. Source revision is not '${AUTOREV}'
+            params['nobranch'] = '1'
+            fetchuri = bb.fetch2.encodeurl((scheme, network, path, user, passwd, params))
+
         tmpparent = tinfoil.config_data.getVar('BASE_WORKDIR')
         bb.utils.mkdirhier(tmpparent)
         tempsrc = tempfile.mkdtemp(prefix='recipetool-', dir=tmpparent)
@@ -475,6 +489,40 @@ def create_recipe(args):
             logger.error('URL %s resulted in an empty source tree' % fetchuri)
             sys.exit(1)
 
+        # We need this checking mechanism to improve the recipe created by recipetool and devtool
+        # is able to parse and build by bitbake.
+        # If there is no input for branch name, then check for branch name with SRCREV provided.
+        if not srcbranch and not nobranch and srcrev and (srcrev != '${AUTOREV}'):
+            try:
+                cmd = 'git branch -r --contains'
+                check_branch, check_branch_err = bb.process.run('%s %s' % (cmd, srcrev), cwd=srctree)
+            except bb.process.ExecutionError as err:
+                logger.error(str(err))
+                sys.exit(1)
+            get_branch = [x.strip() for x in check_branch.splitlines()]
+            # Remove HEAD reference point and drop remote prefix
+            get_branch = [x.split('/', 1)[1] for x in get_branch if not x.startswith('origin/HEAD')]
+            if 'master' in get_branch:
+                # If it is master, we do not need to append 'branch=master' as this is default.
+                # Even with the case where get_branch has multiple objects, if 'master' is one
+                # of them, we should default take from 'master'
+                srcbranch = ''
+            elif len(get_branch) == 1:
+                # If 'master' isn't in get_branch and get_branch contains only ONE object, then store result into 'srcbranch'
+                srcbranch = get_branch[0]
+            else:
+                # If get_branch contains more than one objects, then display error and exit.
+                mbrch = '\n  ' + '\n  '.join(get_branch)
+                logger.error('Revision %s was found on multiple branches: %s\nPlease provide the correct branch in the source URL with ;branch=<branch> (and ensure you use quotes around the URL to avoid the shell interpreting the ";")' % (srcrev, mbrch))
+                sys.exit(1)
+
+        # Since we might have a value in srcbranch, we need to
+        # recontruct the srcuri to include 'branch' in params.
+        if srcbranch:
+            scheme, network, path, user, passwd, params = bb.fetch2.decodeurl(srcuri)
+            params['branch'] = srcbranch
+            srcuri = bb.fetch2.encodeurl((scheme, network, path, user, passwd, params))
+
         if os.path.exists(os.path.join(srctree, '.gitmodules')) and srcuri.startswith('git://'):
             srcuri = 'gitsm://' + srcuri[6:]
             logger.info('Fetching submodules...')
-- 
2.7.4

[PATCH 2/3] recipetool: create: handle git URLs specifying only a tag

[Chang Rebecca Swee Fun]

From: Stanley Phoong <stanley.cheong.kwan.phoong@intel.com>

If a git URL is passed to recipetool create with a tag=, recipetool
should handle it assuming that the tag is valid.

[YOCTO #11393]

Signed-off-by: Stanley Phoong <stanley.cheong.kwan.phoong@intel.com>
---
 scripts/lib/recipetool/create.py | 23 ++++++++++++++++++++++-
 1 file changed, 22 insertions(+), 1 deletion(-)

diff --git a/scripts/lib/recipetool/create.py b/scripts/lib/recipetool/create.py
index d2e6fc0..52c10e5 100644
--- a/scripts/lib/recipetool/create.py
+++ b/scripts/lib/recipetool/create.py
@@ -423,6 +423,7 @@ def create_recipe(args):
     srcsubdir = ''
     srcrev = '${AUTOREV}'
     srcbranch = ''
+    storeTagName = ''
 
     if os.path.isfile(source):
         source = 'file://%s' % os.path.abspath(source)
@@ -446,13 +447,21 @@ def create_recipe(args):
         scheme, network, path, user, passwd, params = bb.fetch2.decodeurl(fetchuri)
         srcbranch = params.get('branch')
         nobranch = params.get('nobranch')
+        tag = params.get('tag')
         if not srcbranch and not nobranch and srcrev != '${AUTOREV}':
             # Append nobranch=1 in the following conditions:
             # 1. User did not set 'branch=' in srcuri, and
             # 2. User did not set 'nobranch=1' in srcuri, and
             # 3. Source revision is not '${AUTOREV}'
             params['nobranch'] = '1'
-            fetchuri = bb.fetch2.encodeurl((scheme, network, path, user, passwd, params))
+        if tag:
+            # Keep a copy of tag and append nobranch=1 then remove tag from URL.
+            # Bitbake fetcher unable to fetch when {AUTOREV} and tag is set at the same time.
+            # We will re-introduce tag argument after bitbake fetcher process is complete.
+            storeTagName = params['tag']
+            params['nobranch'] = '1'
+            del params['tag']
+        fetchuri = bb.fetch2.encodeurl((scheme, network, path, user, passwd, params))
 
         tmpparent = tinfoil.config_data.getVar('BASE_WORKDIR')
         bb.utils.mkdirhier(tmpparent)
@@ -523,6 +532,18 @@ def create_recipe(args):
             params['branch'] = srcbranch
             srcuri = bb.fetch2.encodeurl((scheme, network, path, user, passwd, params))
 
+        if storeTagName:
+            # Re-introduced tag variable from storeTagName
+            # Check srcrev using tag and check validity of the tag
+            try:
+                cmd = ('git rev-list -n 1 %s' % (storeTagName))
+                check_tag, check_tag_err = bb.process.run('%s' % cmd, cwd=srctree)
+                srcrev = check_tag.split()[0]
+            except bb.process.ExecutionError as err:
+                logger.error(str(err))
+                logger.error("Possibly wrong tag name is provided")
+                sys.exit(1)
+
         if os.path.exists(os.path.join(srctree, '.gitmodules')) and srcuri.startswith('git://'):
             srcuri = 'gitsm://' + srcuri[6:]
             logger.info('Fetching submodules...')
-- 
2.7.4

[PATCH 3/3] recipetool: create: replacing PV in SRCURI

[Chang Rebecca Swee Fun]

From: Stanley Phoong <stanley.cheong.kwan.phoong@intel.com>

During recipe creation, it seems that the automation for replacing
${PV} at the SRCURI for tag, (e.g mbed-tls-${PV}) is causing some
issue due to PV assuming it's a git source. A fix is implemented in
this patch to resolve this issue.

Signed-off-by: Stanley Phoong <stanley.cheong.kwan.phoong@intel.com>
---
 scripts/lib/recipetool/create.py | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/scripts/lib/recipetool/create.py b/scripts/lib/recipetool/create.py
index 52c10e5..bab3d5d 100644
--- a/scripts/lib/recipetool/create.py
+++ b/scripts/lib/recipetool/create.py
@@ -424,6 +424,7 @@ def create_recipe(args):
     srcrev = '${AUTOREV}'
     srcbranch = ''
     storeTagName = ''
+    pv_srcpv = False
 
     if os.path.isfile(source):
         source = 'file://%s' % os.path.abspath(source)
@@ -671,6 +672,7 @@ def create_recipe(args):
         lines_before.append('')
         lines_before.append('# Modify these as desired')
         lines_before.append('PV = "%s+git${SRCPV}"' % (realpv or '1.0'))
+        pv_srcpv = True
         if not args.autorev and srcrev == '${AUTOREV}':
             if os.path.exists(os.path.join(srctree, '.git')):
                 (stdout, _) = bb.process.run('git rev-parse HEAD', cwd=srctree)
@@ -797,7 +799,7 @@ def create_recipe(args):
                 skipblank = True
                 continue
         elif line.startswith('SRC_URI = '):
-            if realpv:
+            if realpv and not pv_srcpv:
                 line = line.replace(realpv, '${PV}')
         elif line.startswith('PV = '):
             if realpv:
-- 
2.7.4

Re: [PATCH 2/3] recipetool: create: handle git URLs specifying only a tag

[Christopher Larson]

[-- Attachment #1: Type: text/plain, Size: 3768 bytes --]

rev-parse is a better bet than rev-list for ref validation, IMO.

On Tue, Aug 15, 2017 at 1:24 AM, Chang Rebecca Swee Fun <
rebecca.swee.fun.chang@intel.com> wrote:

> From: Stanley Phoong <stanley.cheong.kwan.phoong@intel.com>
>
> If a git URL is passed to recipetool create with a tag=, recipetool
> should handle it assuming that the tag is valid.
>
> [YOCTO #11393]
>
> Signed-off-by: Stanley Phoong <stanley.cheong.kwan.phoong@intel.com>
> ---
>  scripts/lib/recipetool/create.py | 23 ++++++++++++++++++++++-
>  1 file changed, 22 insertions(+), 1 deletion(-)
>
> diff --git a/scripts/lib/recipetool/create.py b/scripts/lib/recipetool/
> create.py
> index d2e6fc0..52c10e5 100644
> --- a/scripts/lib/recipetool/create.py
> +++ b/scripts/lib/recipetool/create.py
> @@ -423,6 +423,7 @@ def create_recipe(args):
>      srcsubdir = ''
>      srcrev = '${AUTOREV}'
>      srcbranch = ''
> +    storeTagName = ''
>
>      if os.path.isfile(source):
>          source = 'file://%s' % os.path.abspath(source)
> @@ -446,13 +447,21 @@ def create_recipe(args):
>          scheme, network, path, user, passwd, params =
> bb.fetch2.decodeurl(fetchuri)
>          srcbranch = params.get('branch')
>          nobranch = params.get('nobranch')
> +        tag = params.get('tag')
>          if not srcbranch and not nobranch and srcrev != '${AUTOREV}':
>              # Append nobranch=1 in the following conditions:
>              # 1. User did not set 'branch=' in srcuri, and
>              # 2. User did not set 'nobranch=1' in srcuri, and
>              # 3. Source revision is not '${AUTOREV}'
>              params['nobranch'] = '1'
> -            fetchuri = bb.fetch2.encodeurl((scheme, network, path, user,
> passwd, params))
> +        if tag:
> +            # Keep a copy of tag and append nobranch=1 then remove tag
> from URL.
> +            # Bitbake fetcher unable to fetch when {AUTOREV} and tag is
> set at the same time.
> +            # We will re-introduce tag argument after bitbake fetcher
> process is complete.
> +            storeTagName = params['tag']
> +            params['nobranch'] = '1'
> +            del params['tag']
> +        fetchuri = bb.fetch2.encodeurl((scheme, network, path, user,
> passwd, params))
>
>          tmpparent = tinfoil.config_data.getVar('BASE_WORKDIR')
>          bb.utils.mkdirhier(tmpparent)
> @@ -523,6 +532,18 @@ def create_recipe(args):
>              params['branch'] = srcbranch
>              srcuri = bb.fetch2.encodeurl((scheme, network, path, user,
> passwd, params))
>
> +        if storeTagName:
> +            # Re-introduced tag variable from storeTagName
> +            # Check srcrev using tag and check validity of the tag
> +            try:
> +                cmd = ('git rev-list -n 1 %s' % (storeTagName))
> +                check_tag, check_tag_err = bb.process.run('%s' % cmd,
> cwd=srctree)
> +                srcrev = check_tag.split()[0]
> +            except bb.process.ExecutionError as err:
> +                logger.error(str(err))
> +                logger.error("Possibly wrong tag name is provided")
> +                sys.exit(1)
> +
>          if os.path.exists(os.path.join(srctree, '.gitmodules')) and
> srcuri.startswith('git://'):
>              srcuri = 'gitsm://' + srcuri[6:]
>              logger.info('Fetching submodules...')
> --
> 2.7.4
>
> --
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-core
>



-- 
Christopher Larson
kergoth at gmail dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Senior Software Engineer, Mentor Graphics

[-- Attachment #2: Type: text/html, Size: 5302 bytes --]