[PATCH V3 0/2] bitbake-layers: add a ability to query layer dependencies from layer index

[PATCH V3 0/2] bitbake-layers: add a ability to query layer dependencies from layer index

[Chong Lu]

Change since V2:
re-split for IPv6 URL parsing

The following changes since commit 35c9fa0588ed8e88b541a6c80cc1517324616cea:

  maintainers: Update for non-maintained recipes (2015-01-20 21:39:41 +0000)

are available in the git repository at:

  git://git.pokylinux.org/poky-contrib chonglu/layerindex
  http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=chonglu/layerindex

Chong Lu (2):
  bitbake.conf: Add two variables for layer index
  bitbake-layers: add a ability to query layer dependencies from layer index

 bitbake/bin/bitbake-layers | 190 +++++++++++++++++++++++++++++++++++++++++++++
 meta/conf/bitbake.conf     |   6 ++
 2 files changed, 196 insertions(+)

-- 
1.9.1

[PATCH V3 1/2] bitbake.conf: Add two variables for layer index

[Chong Lu]

Add BITBAKE_LAYERINDEX_URL variable that bitbake-layers can use to find layer index.
Add LAYER_FETCH_DIR variable that bitbake-layers can use to specify fetch directory.

[YOCTO #5348]

Signed-off-by: Chong Lu <Chong.Lu@windriver.com>
---
 meta/conf/bitbake.conf | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
index d22e9e8..a8f4dc9 100644
--- a/meta/conf/bitbake.conf
+++ b/meta/conf/bitbake.conf
@@ -549,6 +549,12 @@ SELECTED_OPTIMIZATION[vardeps] += "FULL_OPTIMIZATION DEBUG_OPTIMIZATION"
 BUILD_OPTIMIZATION = "-O2 -pipe"
 
 ##################################################################
+# The OE layer index.
+##################################################################
+BITBAKE_LAYERINDEX_URL ??= "http://layers.openembedded.org/layerindex/api/"
+LAYER_FETCH_DIR ??= "${COREBASE}"
+
+##################################################################
 # Download locations and utilities.
 ##################################################################
 
-- 
1.9.1

[PATCH V3 2/2] bitbake-layers: add a ability to query layer dependencies from layer index

[Chong Lu]

Add a command to query layer dependencies from layer index. Fetch layer and its
dependency layers and add them into conf/bblayers.conf.

[YOCTO #5348]

Signed-off-by: Chong Lu <Chong.Lu@windriver.com>
---
 bitbake/bin/bitbake-layers | 190 +++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 190 insertions(+)

diff --git a/bitbake/bin/bitbake-layers b/bitbake/bin/bitbake-layers
index 9879498..222f685 100755
--- a/bitbake/bin/bitbake-layers
+++ b/bitbake/bin/bitbake-layers
@@ -27,6 +27,8 @@ import sys
 import fnmatch
 from collections import defaultdict
 import re
+import httplib, urlparse, json
+import subprocess
 
 bindir = os.path.dirname(__file__)
 topdir = os.path.dirname(bindir)
@@ -157,6 +159,194 @@ usage: remove-layer <layerdir>
                 sys.stderr.write("No layers matching %s found in BBLAYERS\n" % item)
 
 
+    def get_json_data(self, apiurl):
+        proxy_settings = os.environ.get("http_proxy", None)
+        conn = None
+        _parsedurl = urlparse.urlparse(apiurl)
+        path = _parsedurl.path
+        query = _parsedurl.query
+        def parse_url(url):
+            parsedurl = urlparse.urlparse(url)
+            if parsedurl.netloc[0] == '[':
+                host, port = parsedurl.netloc[1:].split(']', 1)
+                if ':' in port:
+                    port = port.rsplit(':', 1)[1]
+                else:
+                    port = None
+            else:
+                if parsedurl.netloc.count(':') == 1:
+                    (host, port) = parsedurl.netloc.split(":")
+                else:
+                    host = parsedurl.netloc
+                    port = None
+            return (host, 80 if port is None else int(port))
+
+        if proxy_settings is None:
+            host, port = parse_url(apiurl)
+            conn = httplib.HTTPConnection(host, port)
+            conn.request("GET", path + "?" + query)
+        else:
+            host, port = parse_url(proxy_settings)
+            conn = httplib.HTTPConnection(host, port)
+            conn.request("GET", apiurl)
+
+        r = conn.getresponse()
+        if r.status != 200:
+            raise Exception("Failed to read " + path + ": %d %s" % (r.status, r.reason))
+        return json.loads(r.read())
+
+
+    def get_layer_deps(self, layername, layeritems, layerbranches, layerdependencies, selfname=False):
+        def layeritems_info_id(items_name, layeritems):
+            litems_id = ""
+            for li in layeritems:
+                if li['name'] == items_name:
+                    litems_id = li['id']
+                    break
+            if litems_id:
+                return litems_id
+
+        def layerbranches_info(items_id, layerbranches):
+            lbranch = {}
+            for lb in layerbranches:
+                # branch is master.
+                if lb['layer'] == items_id and lb['branch'] == 1:
+                    lbranch['id'] = lb['id']
+                    lbranch['vcs_subdir'] = lb['vcs_subdir']
+                    break
+            if not lbranch['id']:
+                logger.error("The id of layerBranches is not found.")
+                return
+            else:
+                return lbranch
+
+        def layerdependencies_info(lb_id, layerdependencies):
+            ld_deps = []
+            for ld in layerdependencies:
+                if ld['layerbranch'] == lb_id and not ld['dependency'] in ld_deps:
+                    ld_deps.append(ld['dependency'])
+            if not ld_deps:
+                logger.error("The dependency of layerDependencies is not found.")
+                return
+            else:
+                return ld_deps
+
+        def layeritems_info_name_subdir(items_id, layeritems):
+            litems = {}
+            for li in layeritems:
+                if li['id'] == items_id:
+                    litems['vcs_url'] = li['vcs_url']
+                    litems['name'] = li['name']
+                    break
+            return litems
+
+        if selfname:
+            selfid = layeritems_info_id(layername, layeritems)
+            selfsubdir = layerbranches_info(selfid, layerbranches)['vcs_subdir']
+            selfurl = layeritems_info_name_subdir(selfid, layeritems)['vcs_url']
+            if selfsubdir and selfurl:
+                return selfurl, selfsubdir
+            else:
+                logger.error("Can NOT get %s git repo and subdir" % layername)
+                return
+        ldict = {}
+        itemsid = layeritems_info_id(layername, layeritems)
+        if not itemsid:
+            return layername, None
+        lbid = layerbranches_info(itemsid, layerbranches)['id']
+        for dependency in layerdependencies_info(lbid, layerdependencies):
+            lname = layeritems_info_name_subdir(dependency, layeritems)['name']
+            lurl = layeritems_info_name_subdir(dependency, layeritems)['vcs_url']
+            lsubdir = layerbranches_info(dependency, layerbranches)['vcs_subdir']
+            ldict[lname] = lurl, lsubdir
+        return None, ldict
+
+
+    def get_fetch_layer(self, fetchdir, url, subdir):
+        layername = self.get_layer_name(url)
+        if os.path.splitext(layername)[1] == '.git':
+            layername = os.path.splitext(layername)[0]
+        repodir = os.path.join(fetchdir, layername)
+        layerdir = os.path.join(repodir, subdir)
+        if not os.path.exists(repodir):
+            logger.plain("Cloning into '%s'..." % repodir)
+            result = subprocess.call('git clone -q %s %s' % (url, repodir), shell = True)
+            if result:
+                logger.error("Failed to download %s" % url)
+            else:
+                return layername, layerdir
+        elif os.path.exists(layerdir):
+            return layername, layerdir
+        else:
+            logger.error("%s is not in %s" % (url, subdir))
+
+
+    def do_show_layer_deps(self, args):
+        """Find layer dependencies from layer index. Fetch it and its dependency layers. Add them to conf/bblayers.conf.
+
+usage: show-layer-deps [-a] <layername,...>
+
+Options:
+  -a   fetch layer and add to conf/bblayers.conf
+"""
+        fetch_add = False
+        layernames = ""
+        for arg in args.split():
+            if arg == '-a':
+                fetch_add = True
+            elif not arg.startswith('-'):
+                layernames = arg
+            else:
+                sys.stderr.write("show-layer-deps: invalid option %s\n" % arg)
+                self.do_help('')
+                return
+        if not layernames:
+            sys.stderr.write("Please specify layer name.\n")
+            return
+        self.init_bbhandler()
+        apiurl = self.bbhandler.config_data.getVar('BITBAKE_LAYERINDEX_URL', True)
+        if not apiurl:
+            logger.error("Can NOT get BITBAKE_LAYERINDEX_URL.")
+        apilinks = self.get_json_data(apiurl)
+        layeritems = self.get_json_data(apilinks['layerItems'])
+        layerbranches = self.get_json_data(apilinks['layerBranches'])
+        layerdependencies = self.get_json_data(apilinks['layerDependencies'])
+        invaluenames = []
+        repourls = []
+        display = True
+        for layername in layernames.split(','):
+            if not layername == "meta":
+                invaluename, layerdict = self.get_layer_deps(layername, layeritems, layerbranches, layerdependencies)
+                if layerdict:
+                    repourls.append(self.get_layer_deps(layername, layeritems, layerbranches, layerdependencies, selfname=True))
+                    for layer in layerdict:
+                        if display:
+                            logger.plain("%s  %s  %s  %s" % ("Layer".ljust(20), "Dependencies".ljust(20), "Git repository".ljust(55), "Subdirectory"))
+                            logger.plain('=' * 115)
+                            display = False
+                        logger.plain("%s  %s  %s  %s" % (layername.ljust(20), layer.ljust(20), layerdict[layer][0].ljust(55), layerdict[layer][1]))
+                        if not layer == "openembedded-core" and not (layerdict[layer][0], layerdict[layer][1]) in repourls:
+                            repourls.append((layerdict[layer][0], layerdict[layer][1]))
+                if invaluename and not invaluename in invaluenames:
+                    invaluenames.append(invaluename)
+        for invaluename in invaluenames:
+            logger.warn("%s is not found in layer index." % invaluename)
+        if fetch_add and repourls:
+            fetchdir = self.bbhandler.config_data.getVar('LAYER_FETCH_DIR', True)
+            if not fetchdir:
+                logger.error("Can NOT get LAYER_FETCH_DIR.")
+                return
+            if not os.path.exists(fetchdir):
+                os.makedirs(fetchdir)
+            for repourl, subdir in repourls:
+                name, layerdir = self.get_fetch_layer(fetchdir, repourl, subdir)
+                if subdir:
+                    logger.plain("Add \"%s\" to conf/bblayers.conf" % subdir)
+                else:
+                    logger.plain("Add \"%s\" to conf/bblayers.conf" % name)
+                self.do_add_layer(layerdir)
+
+
     def version_str(self, pe, pv, pr = None):
         verstr = "%s" % pv
         if pr:
-- 
1.9.1

Re: [PATCH V3 2/2] bitbake-layers: add a ability to query layer dependencies from layer index

[Bernhard Reutner-Fischer]

On 23 January 2015 at 06:28, Chong Lu <Chong.Lu@windriver.com> wrote:
> Add a command to query layer dependencies from layer index. Fetch layer and its
> dependency layers and add them into conf/bblayers.conf.
>
> [YOCTO #5348]
>
> Signed-off-by: Chong Lu <Chong.Lu@windriver.com>
> ---
>  bitbake/bin/bitbake-layers | 190 +++++++++++++++++++++++++++++++++++++++++++++
>  1 file changed, 190 insertions(+)
>
> diff --git a/bitbake/bin/bitbake-layers b/bitbake/bin/bitbake-layers
> index 9879498..222f685 100755
> --- a/bitbake/bin/bitbake-layers
> +++ b/bitbake/bin/bitbake-layers
> @@ -27,6 +27,8 @@ import sys
>  import fnmatch
>  from collections import defaultdict
>  import re
> +import httplib, urlparse, json
> +import subprocess
>
>  bindir = os.path.dirname(__file__)
>  topdir = os.path.dirname(bindir)
> @@ -157,6 +159,194 @@ usage: remove-layer <layerdir>
>                  sys.stderr.write("No layers matching %s found in BBLAYERS\n" % item)
>
>
> +    def get_json_data(self, apiurl):
> +        proxy_settings = os.environ.get("http_proxy", None)
> +        conn = None
> +        _parsedurl = urlparse.urlparse(apiurl)
> +        path = _parsedurl.path
> +        query = _parsedurl.query
> +        def parse_url(url):
> +            parsedurl = urlparse.urlparse(url)
> +            if parsedurl.netloc[0] == '[':
> +                host, port = parsedurl.netloc[1:].split(']', 1)
> +                if ':' in port:
> +                    port = port.rsplit(':', 1)[1]
> +                else:
> +                    port = None
> +            else:
> +                if parsedurl.netloc.count(':') == 1:
> +                    (host, port) = parsedurl.netloc.split(":")
> +                else:
> +                    host = parsedurl.netloc
> +                    port = None
> +            return (host, 80 if port is None else int(port))

well yea, except that does not help code-reuse to fix the
wrong splitport patterns used elsewhere in bitbake, unfortunately..

Re: [PATCH V3 0/2] bitbake-layers: add a ability to query layer dependencies from layer index

[Chong Lu]

ping

//Chong

On 01/23/2015 01:28 PM, Chong Lu wrote:
> Change since V2:
> re-split for IPv6 URL parsing
>
> The following changes since commit 35c9fa0588ed8e88b541a6c80cc1517324616cea:
>
>    maintainers: Update for non-maintained recipes (2015-01-20 21:39:41 +0000)
>
> are available in the git repository at:
>
>    git://git.pokylinux.org/poky-contrib chonglu/layerindex
>    http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=chonglu/layerindex
>
> Chong Lu (2):
>    bitbake.conf: Add two variables for layer index
>    bitbake-layers: add a ability to query layer dependencies from layer index
>
>   bitbake/bin/bitbake-layers | 190 +++++++++++++++++++++++++++++++++++++++++++++
>   meta/conf/bitbake.conf     |   6 ++
>   2 files changed, 196 insertions(+)
>

Re: [PATCH V3 0/2] bitbake-layers: add a ability to query layer dependencies from layer index

[Paul Eggleton]

Hi Chong,

On Friday 23 January 2015 13:28:32 Chong Lu wrote:
> Change since V2:
> re-split for IPv6 URL parsing
> 
> The following changes since commit 35c9fa0588ed8e88b541a6c80cc1517324616cea:
> 
>   maintainers: Update for non-maintained recipes (2015-01-20 21:39:41 +0000)
> 
> are available in the git repository at:
> 
>   git://git.pokylinux.org/poky-contrib chonglu/layerindex
>   http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=chonglu/layerindex
> 
> Chong Lu (2):
>   bitbake.conf: Add two variables for layer index
>   bitbake-layers: add a ability to query layer dependencies from layer index

Apologies for the delay in getting back to you. This is definitely improving, 
some feedback though:

* I think the new subcommand would make more sense if it were switched around 
so that it was fetching by default (and renamed as appropriate). Perhaps 
"layerindex-fetch" (with -n to avoid fetching and just report what it would 
fetch).

* I think the layer index URL specified in the OE variable should be to the 
root rather than to /api - instead have that suffix added by the code that does 
the API calls. This makes it usable as a URL for the index for presenting to 
the user if needed.

* Can you rename the two variables to BBLAYERS_LAYERINDEX_URL and 
BBLAYERS_FETCH_DIR (sorry I didn't think of this earlier).

* Please mention bitbake-layers in the comment above the settings in 
bitbake.conf so it's clear where these variables are used.

* By using -q with git you lose the status output which can be very useful 
given how long it can take to fetch some layers e.g. meta-oe, so can you drop 
this?

* Can you add a check to see if the layer has already been fetched before 
fetching it and skip the fetch if it does (but not skip the add)?

* If I run "bitbake-layers show-layer-deps meta-security" I get an error:
ERROR: Can NOT get meta-security git repo and subdir
(The code currently expects there to be a subdir, but this is an optional 
field, not all layers have a subdirectory, meta-security is one example.)

* I think you may be able to get away with skipping the initial parse by 
passing config_only=True to init_bbhandler(). If you can, this will save a bit 
of time each time you run the command.

Thanks,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre

Re: [PATCH V3 0/2] bitbake-layers: add a ability to query layer dependencies from layer index

[Paul Eggleton]

On Monday 02 February 2015 16:05:52 Paul Eggleton wrote:
> Hi Chong,
> 
> On Friday 23 January 2015 13:28:32 Chong Lu wrote:
> > Change since V2:
> > re-split for IPv6 URL parsing
> > 
> > The following changes since commit 
35c9fa0588ed8e88b541a6c80cc1517324616cea:
> >   maintainers: Update for non-maintained recipes (2015-01-20 21:39:41
> >   +0000)
> > 
> > are available in the git repository at:
> >   git://git.pokylinux.org/poky-contrib chonglu/layerindex
> >   http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=chonglu/layerindex
> > 
> > Chong Lu (2):
> >   bitbake.conf: Add two variables for layer index
> >   bitbake-layers: add a ability to query layer dependencies from layer
> >   index
> 
> Apologies for the delay in getting back to you. This is definitely
> improving, some feedback though:
> 
> * I think the new subcommand would make more sense if it were switched
> around so that it was fetching by default (and renamed as appropriate).
> Perhaps "layerindex-fetch" (with -n to avoid fetching and just report what
> it would fetch).
> 
> * I think the layer index URL specified in the OE variable should be to the
> root rather than to /api - instead have that suffix added by the code that
> does the API calls. This makes it usable as a URL for the index for
> presenting to the user if needed.
> 
> * Can you rename the two variables to BBLAYERS_LAYERINDEX_URL and
> BBLAYERS_FETCH_DIR (sorry I didn't think of this earlier).
> 
> * Please mention bitbake-layers in the comment above the settings in
> bitbake.conf so it's clear where these variables are used.
> 
> * By using -q with git you lose the status output which can be very useful
> given how long it can take to fetch some layers e.g. meta-oe, so can you
> drop this?
> 
> * Can you add a check to see if the layer has already been fetched before
> fetching it and skip the fetch if it does (but not skip the add)?
> 
> * If I run "bitbake-layers show-layer-deps meta-security" I get an error:
> ERROR: Can NOT get meta-security git repo and subdir
> (The code currently expects there to be a subdir, but this is an optional
> field, not all layers have a subdirectory, meta-security is one example.)
> 
> * I think you may be able to get away with skipping the initial parse by
> passing config_only=True to init_bbhandler(). If you can, this will save a
> bit of time each time you run the command.

I just noticed another issue - dependencies-of-dependencies aren't handled 
here either; e.g. meta-security's dependencies have just been updated so that 
it now depends on meta-networking, and meta-networking depends on meta-python 
but that's not listed when I run bitbake-layers show-layer-deps meta-security. 
I think maybe this is a deficiency in the layer index API - it would be faster 
if there was one call we could make to get the entire list of dependencies, 
URLs and subdirectories for the entire dependency chain.

I know this is probably further than you expected to go with this bug, but is 
that API enhancement something you could look into adding?

Cheers,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre

Re: [PATCH V3 0/2] bitbake-layers: add a ability to query layer dependencies from layer index

[Chong Lu]

On 02/03/2015 12:05 AM, Paul Eggleton wrote:
> Hi Chong,
>
> On Friday 23 January 2015 13:28:32 Chong Lu wrote:
>> Change since V2:
>> re-split for IPv6 URL parsing
>>
>> The following changes since commit 35c9fa0588ed8e88b541a6c80cc1517324616cea:
>>
>>    maintainers: Update for non-maintained recipes (2015-01-20 21:39:41 +0000)
>>
>> are available in the git repository at:
>>
>>    git://git.pokylinux.org/poky-contrib chonglu/layerindex
>>    http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=chonglu/layerindex
>>
>> Chong Lu (2):
>>    bitbake.conf: Add two variables for layer index
>>    bitbake-layers: add a ability to query layer dependencies from layer index
> Apologies for the delay in getting back to you. This is definitely improving,
> some feedback though:
>
> * I think the new subcommand would make more sense if it were switched around
> so that it was fetching by default (and renamed as appropriate). Perhaps
> "layerindex-fetch" (with -n to avoid fetching and just report what it would
> fetch).
>
> * I think the layer index URL specified in the OE variable should be to the
> root rather than to /api - instead have that suffix added by the code that does
> the API calls. This makes it usable as a URL for the index for presenting to
> the user if needed.
>
> * Can you rename the two variables to BBLAYERS_LAYERINDEX_URL and
> BBLAYERS_FETCH_DIR (sorry I didn't think of this earlier).

I don't understand why does uname? Or, would you give me a example?

>
> * Please mention bitbake-layers in the comment above the settings in
> bitbake.conf so it's clear where these variables are used.

And I don't understand this one """

Please mention bitbake-layers in the comment above the settings in
bitbake.conf so it's clear where these variables are used.

"""
Would you explain in details?

Best Regards
Chong

>
> * By using -q with git you lose the status output which can be very useful
> given how long it can take to fetch some layers e.g. meta-oe, so can you drop
> this?
>
> * Can you add a check to see if the layer has already been fetched before
> fetching it and skip the fetch if it does (but not skip the add)?
>
> * If I run "bitbake-layers show-layer-deps meta-security" I get an error:
> ERROR: Can NOT get meta-security git repo and subdir
> (The code currently expects there to be a subdir, but this is an optional
> field, not all layers have a subdirectory, meta-security is one example.)
>
> * I think you may be able to get away with skipping the initial parse by
> passing config_only=True to init_bbhandler(). If you can, this will save a bit
> of time each time you run the command.
>
> Thanks,
> Paul
>

Re: [PATCH V3 0/2] bitbake-layers: add a ability to query layer dependencies from layer index

[Paul Eggleton]

Hi Chong,

On Friday 06 February 2015 14:52:14 Chong Lu wrote:
> On 02/03/2015 12:05 AM, Paul Eggleton wrote:
> > On Friday 23 January 2015 13:28:32 Chong Lu wrote:
> >> Change since V2:
> >> re-split for IPv6 URL parsing
> >> 
> >> The following changes since commit 
35c9fa0588ed8e88b541a6c80cc1517324616cea:
> >>    maintainers: Update for non-maintained recipes (2015-01-20 21:39:41
> >>    +0000)
> >> 
> >> are available in the git repository at:
> >>    git://git.pokylinux.org/poky-contrib chonglu/layerindex
> >>    http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=chonglu/layerind
> >>    ex
> >> 
> >> Chong Lu (2):
> >>    bitbake.conf: Add two variables for layer index
> >>    bitbake-layers: add a ability to query layer dependencies from layer
> >>    index
> > 
> > Apologies for the delay in getting back to you. This is definitely
> > improving, some feedback though:
> > 
> > * I think the new subcommand would make more sense if it were switched
> > around so that it was fetching by default (and renamed as appropriate).
> > Perhaps "layerindex-fetch" (with -n to avoid fetching and just report
> > what it would fetch).
> > 
> > * I think the layer index URL specified in the OE variable should be to
> > the
> > root rather than to /api - instead have that suffix added by the code that
> > does the API calls. This makes it usable as a URL for the index for
> > presenting to the user if needed.
> > 
> > * Can you rename the two variables to BBLAYERS_LAYERINDEX_URL and
> > BBLAYERS_FETCH_DIR (sorry I didn't think of this earlier).
> 
> I don't understand why does uname? Or, would you give me a example?

I'm not sure I understand the question - but FWIW the variables I'm talking 
about renaming are currently named BITBAKE_LAYERINDEX_URL and LAYER_FETCH_DIR. 
The reason I suggest naming them BBLAYERS_LAYERINDEX_URL and
BBLAYERS_FETCH_DIR respectively is it is more closely related to the area 
where the values are actually used. Is that what you were asking about?

> > * Please mention bitbake-layers in the comment above the settings in
> > bitbake.conf so it's clear where these variables are used.
> 
> And I don't understand this one """
> 
> Please mention bitbake-layers in the comment above the settings in
> bitbake.conf so it's clear where these variables are used.
> 
> """
> Would you explain in details?

You're adding defaults for these variables in bitbake.conf. We should mention 
above those defaults what these variables are actually being used for. i.e.:

 ##################################################################
+# Settings used by bitbake-layers.
+##################################################################
+BBLAYERS_LAYERINDEX_URL ??= "http://layers.openembedded.org/layerindex/api/"
+BBLAYERS_FETCH_DIR ??= "${COREBASE}"
+
+##################################################################

 
> > * By using -q with git you lose the status output which can be very useful
> > given how long it can take to fetch some layers e.g. meta-oe, so can you
> > drop this?
> > 
> > * Can you add a check to see if the layer has already been fetched before
> > fetching it and skip the fetch if it does (but not skip the add)?
> > 
> > * If I run "bitbake-layers show-layer-deps meta-security" I get an error:
> > ERROR: Can NOT get meta-security git repo and subdir
> > (The code currently expects there to be a subdir, but this is an optional
> > field, not all layers have a subdirectory, meta-security is one example.)
> > 
> > * I think you may be able to get away with skipping the initial parse by
> > passing config_only=True to init_bbhandler(). If you can, this will save a
> > bit of time each time you run the command.

Cheers,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre

Re: [PATCH V3 0/2] bitbake-layers: add a ability to query layer dependencies from layer index

[Paul Eggleton]

On Tuesday 03 February 2015 10:12:19 Paul Eggleton wrote:
> On Monday 02 February 2015 16:05:52 Paul Eggleton wrote:
> > Hi Chong,
> > 
> > On Friday 23 January 2015 13:28:32 Chong Lu wrote:
> > > Change since V2:
> > > re-split for IPv6 URL parsing
> > > 
> > > The following changes since commit
> 
> 35c9fa0588ed8e88b541a6c80cc1517324616cea:
> > >   maintainers: Update for non-maintained recipes (2015-01-20 21:39:41
> > >   +0000)
> > > 
> > > are available in the git repository at:
> > >   git://git.pokylinux.org/poky-contrib chonglu/layerindex
> > >   http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=chonglu/layerind
> > >   ex
> > > 
> > > Chong Lu (2):
> > >   bitbake.conf: Add two variables for layer index
> > >   bitbake-layers: add a ability to query layer dependencies from layer
> > >   index
> > 
> > Apologies for the delay in getting back to you. This is definitely
> > improving, some feedback though:
> > 
> > * I think the new subcommand would make more sense if it were switched
> > around so that it was fetching by default (and renamed as appropriate).
> > Perhaps "layerindex-fetch" (with -n to avoid fetching and just report what
> > it would fetch).
> > 
> > * I think the layer index URL specified in the OE variable should be to
> > the
> > root rather than to /api - instead have that suffix added by the code that
> > does the API calls. This makes it usable as a URL for the index for
> > presenting to the user if needed.
> > 
> > * Can you rename the two variables to BBLAYERS_LAYERINDEX_URL and
> > BBLAYERS_FETCH_DIR (sorry I didn't think of this earlier).
> > 
> > * Please mention bitbake-layers in the comment above the settings in
> > bitbake.conf so it's clear where these variables are used.
> > 
> > * By using -q with git you lose the status output which can be very useful
> > given how long it can take to fetch some layers e.g. meta-oe, so can you
> > drop this?
> > 
> > * Can you add a check to see if the layer has already been fetched before
> > fetching it and skip the fetch if it does (but not skip the add)?
> > 
> > * If I run "bitbake-layers show-layer-deps meta-security" I get an error:
> > ERROR: Can NOT get meta-security git repo and subdir
> > (The code currently expects there to be a subdir, but this is an optional
> > field, not all layers have a subdirectory, meta-security is one example.)
> > 
> > * I think you may be able to get away with skipping the initial parse by
> > passing config_only=True to init_bbhandler(). If you can, this will save a
> > bit of time each time you run the command.
> 
> I just noticed another issue - dependencies-of-dependencies aren't handled
> here either; e.g. meta-security's dependencies have just been updated so
> that it now depends on meta-networking, and meta-networking depends on
> meta-python but that's not listed when I run bitbake-layers show-layer-deps
> meta-security. I think maybe this is a deficiency in the layer index API -
> it would be faster if there was one call we could make to get the entire
> list of dependencies, URLs and subdirectories for the entire dependency
> chain.
> 
> I know this is probably further than you expected to go with this bug, but
> is that API enhancement something you could look into adding?

Yet another concern - we need to deal with layer branches properly, i.e. we 
can't just assume master or otherwise people using stable branches will have 
compatibility issues. In many cases we won't be able to rely upon the OE-
Core/poky branch being the matching name either, so it might have to be 
something we require the user to specify on the command line.

Cheers,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre