[PATCH] Add support for remote layering.

[PATCH] Add support for remote layering.

[Jeremy Puhlman]

> * Maybe need to split into "bootstrap" steps (e.g where pseduo is
   established, layers downloaded etc)

 So for the most part, the last product we shipped was largely arranged
in this manner. We were using collections, but in many cases layers is
just a reimplementation at a lower level(which is definitely goodness).
During ELC I had a number of discussions primarily around remote layer
fetch and most of the responses were meh, or that they wanted to make
the tool external to bitbake. So after the conference was over I went
ahead and worked out a version of the implementation  that could be done
either in a include or in the bitbake code itself. The version in the
git repo below is obviously just the bitbake implementation.

I have a rough implementation largely based off of collection.inc + some
enhancements we made internally for remote collection fetching. Some of
it is messy(setting up the early fetching vars with out using another
config), but for the ones I list in the patch header it appears to work
well.

I do not believe that this is complete, but it is probably a reasonable
starting point.

Pull URL: git://gitorious.org/jpwork/bb.git
Branch: remote-layering
Browse: https://gitorious.org/jpwork/bb/trees/remote-layering

[PATCH] Add support for remote layering.

[Jeremy Puhlman]

The bulk of this patch is based on Chris Larson's collection.inc,
but refactored for use inside bitbake, and for layers.

Adds two new configuration options:

LAYER_UNPACKDIR - directory to unpack downloaded layers for use in configuration.
LAYER_REMOTE_STASH - location to store downloaded layers for use in configuration.

Layers can be specified in the current manner:
BBLAYERS = " \
   /path/to/layer/one \
   /path/to/layer/two \
"

Or you can specify uris:

BBLAYERS = " \
	http://url.to.my.tarball/tarball.tar.gz \
	git://gitserver.com/git/repo;proto=http \
        /path/to/layer/three \
        file:///path/to/layer/four \
"

Currently there is a single layer option, that can be added to a uri, layerBase=<subpath>.
This option would be used to specify if a layer starts somewhere other then the base of the
tarball/scm repository. For example if you wanted to add oe-core you would do:

BBLAYERS  = "git://git.openembedded.org/openembedded-core;protocol=git;tag=master;layerBase=meta"

Currently for fetch the following uris should work  : 'git','http','https','ftp','file'
Currently for fetch2 the following uris should work : 'git','http','https','ftp','file','svn'

There are certainly some ugly bits in here, and there are certainly lots of places to improve,
but this gets a good chunk of the base support in.

Signed-off-by: Jeremy Puhlman <jpuhlman@mvista.com>
---
 lib/bb/cooker.py       |    3 +-
 lib/bb/fetch/layer.py  |   65 ++++++++++++++++
 lib/bb/fetch2/layer.py |   76 ++++++++++++++++++
 lib/bb/remotelayer.py  |  197 ++++++++++++++++++++++++++++++++++++++++++++++++
 4 files changed, 340 insertions(+), 1 deletions(-)
 create mode 100644 lib/bb/fetch/layer.py
 create mode 100644 lib/bb/fetch2/layer.py
 create mode 100644 lib/bb/remotelayer.py

diff --git a/lib/bb/cooker.py b/lib/bb/cooker.py
index d4415d3..0099a65 100644
--- a/lib/bb/cooker.py
+++ b/lib/bb/cooker.py
@@ -575,6 +575,7 @@ class BBCooker:
             path, _ = os.path.split(path)
 
     def parseConfigurationFiles(self, files):
+        from bb.remotelayer import RemoteLayers
         data = self.configuration.data
         bb.parse.init_parser(data)
         for f in files:
@@ -586,7 +587,7 @@ class BBCooker:
             data = _parse(layerconf, data)
 
             layers = (bb.data.getVar('BBLAYERS', data, True) or "").split()
-
+            layers = RemoteLayers(layers,data,parselog).getLayers()
             data = bb.data.createCopy(data)
             for layer in layers:
                 parselog.debug(2, "Adding layer %s", layer)
diff --git a/lib/bb/fetch/layer.py b/lib/bb/fetch/layer.py
new file mode 100644
index 0000000..3b0d319
--- /dev/null
+++ b/lib/bb/fetch/layer.py
@@ -0,0 +1,65 @@
+"""
+BitBake 'layer' fetch implementation.
+
+"""
+
+# Copyright (C) 2011 Jeremy Puhlman 
+#
+#   Classes for obtaining upstream sources for the
+#   BitBake build tools.
+#   Copyright (C) 2003, 2004  Chris Larson
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License version 2 as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License along
+# with this program; if not, write to the Free Software Foundation, Inc.,
+# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
+
+class LayerFetch:
+
+    def __init__(self, url, pathurl, logger, ld):
+	self.parselog = logger
+	self.tarballpath = self._fetch(url,pathurl,ld)
+
+    def supported(self, ud):
+	return ud.type in ['git','http','https','ftp','file']
+
+    def getTarballPath(self):
+        return self.tarballpath
+
+    def _fetch(self, url, pathurl, ld):
+        urldata = {}
+        try:
+            urldata[url] = bb.fetch.FetchData(url,ld)
+        except bb.fetch.NoMethodError: 
+            self.parselog.debug(1, "No method for %s" % url)
+            return
+        ud = urldata[url]
+        filename=os.path.basename(ud.path)
+
+	if not self.supported(ud):
+           bb.fatal("Layer fetching does not support %s uris" % ud.type)
+        if not ud.setup:
+           ud.setup_localpath(ld)
+        m = ud.method
+        tarballpath = bb.fetch.localpath(url, ld)
+        if os.path.exists(tarballpath) :
+           return tarballpath
+        try: 
+           m.go(url,ud,ld)
+        except (bb.fetch.MissingParameterError,
+           bb.fetch.FetchError,
+           bb.fetch.MD5SumError):
+
+           import sys
+           (type, value, traceback) = sys.exc_info()
+           self.parselog.debug(1, "layer fetch failure: %s" % value)
+           return
+        return tarballpath
diff --git a/lib/bb/fetch2/layer.py b/lib/bb/fetch2/layer.py
new file mode 100644
index 0000000..09de8a0
--- /dev/null
+++ b/lib/bb/fetch2/layer.py
@@ -0,0 +1,76 @@
+"""
+BitBake 'layer' fetch2 implementation.
+
+"""
+
+# Copyright (C) 2011 Jeremy Puhlman 
+#
+#   Classes for obtaining upstream sources for the
+#   BitBake build tools.
+#   Copyright (C) 2003, 2004  Chris Larson
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License version 2 as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License along
+# with this program; if not, write to the Free Software Foundation, Inc.,
+# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
+
+class LayerFetch:
+
+    def __init__(self, url, pathurl, logger, ld):
+	self.parselog = logger
+	self.tarballpath = self._fetch(url,pathurl,ld)
+
+    def supported(self, ud):
+	return ud.type in ['git','http','https','ftp', 'svn', 'file']
+
+    def getTarballPath(self):
+        return self.tarballpath
+
+    def _fetch(self, url, pathurl, ld):
+        urldata = {}
+        layer_unpackdir =  ld.getVar("LAYER_UNPACKDIR", True)		
+        try:
+            urldata[url] = bb.fetch.FetchData(url,ld)
+        except bb.fetch.NoMethodError: 
+            self.parselog.debug(1, "No method for %s" % url)
+            return
+        ud = urldata[url]
+        filename=os.path.basename(ud.path)
+
+	if not self.supported(ud):
+           bb.fatal("Layer fetching does not support %s uris" % ud.type)
+        if not ud.setup:
+           ud.setup_localpath(ld)
+        m = ud.method
+	if ud.type not in ['git']:
+              tarballpath = bb.fetch.localpath(url, ld)
+              if os.path.exists(bb.fetch.localpath(url, ld)) :
+                 return tarballpath		
+        try: 
+           m.download(url,ud,ld)
+        except (bb.fetch.MissingParameterError,
+           bb.fetch.FetchError,
+           bb.fetch.MD5SumError):
+
+           import sys
+           (type, value, traceback) = sys.exc_info()
+           self.parselog.debug(1, "layer fetch failure: %s" % value)
+           return
+	
+	if ud.type in ['git']:
+           unpackedscm = "%s/%s" % (layer_unpackdir, ud.mirrortarball)
+           if m.need_update(url,ud,ld):
+              m.build_mirror_data(url, ud,ld)
+              m.unpack(ud, unpackedscm ,ld)
+           tarballpath = unpackedscm + "/git" 
+
+        return tarballpath
+
diff --git a/lib/bb/remotelayer.py b/lib/bb/remotelayer.py
new file mode 100644
index 0000000..2a61e71
--- /dev/null
+++ b/lib/bb/remotelayer.py
@@ -0,0 +1,197 @@
+"""
+BitBake 'remote layer' handling.
+
+"""
+
+# Copyright (C) 2011 Jeremy Puhlman 
+#
+#   Classes for obtaining upstream sources for the
+#   BitBake build tools.
+#   Copyright (C) 2003, 2004  Chris Larson
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License version 2 as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License along
+# with this program; if not, write to the Free Software Foundation, Inc.,
+# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
+
+class RemoteLayers:
+    """
+    Get remote layers.
+    """
+
+    def __init__(self, layers, data, logger):
+        self.parselog = logger
+        self.layers = self._getRemoteLayers(layers, data)
+
+    def getLayers(self):
+        return self.layers    
+
+    """
+    Ugly. We have not loaded bitbake.conf yet, and need the values set for fetching to work
+    Could load preliminary bitbake.conf, but this works.
+    """
+    def _setFetchValues(self,data):
+	"""Need these for any fetching to work"""
+        def setifunset(d,k,v): 
+           if d.getVar(k,0) is None:
+                d.setVar(k,v)
+
+        setifunset(data, "TMPDIR", "${TOPDIR}/tmp")
+        setifunset(data, "LAYER_REMOTE_STASH", "${TMPDIR}/layers-remote")
+        setifunset(data, "LAYER_UNPACKDIR", "${TMPDIR}/layers")
+
+        ld = data.createCopy()
+        """Provide settings for fetchers.""" 
+        setifunset(ld, "CACHE", "${TMPDIR}/cache")
+        setifunset(ld, "OVERRIDES", "")
+        setifunset(ld, "CVSDIR", "${DL_DIR}/cvs")
+        setifunset(ld, "GITDIR", "${DL_DIR}/git")
+        setifunset(ld, "SVNDIR", "${DL_DIR}/svn")
+        setifunset(ld, "FETCHCOMMAND", "")
+        setifunset(ld, "FETCHCOMMAND_cvs", "/usr/bin/env cvs -d${CVSROOT} co ${CVSCOOPTS} ${CVSMODULE}")
+        setifunset(ld, "FETCHCOMMAND_svn", "/usr/bin/env svn co ${SVNCOOPTS} ${SVNROOT} ${SVNMODULE}")
+        setifunset(ld, "FETCHCOMMAND_wget", "/usr/bin/env wget -t 5 -q --passive-ftp -P ${DL_DIR} ${URI}")
+        setifunset(ld, "FETCHCMD_cvs", "/usr/bin/env cvs ")
+        setifunset(ld, "FETCHCMD_svn", "/usr/bin/env svn ")
+        setifunset(ld, "FETCHCMD_bzr", "/usr/bin/env bzr ")
+        setifunset(ld, "FETCHCMD_hg", "/usr/bin/env hg ")
+        setifunset(ld, "FETCHCMD_wget", "/usr/bin/env wget -t 5 -q")
+        setifunset(ld, "CHECKCOMMAND_wget", "/usr/bin/env wget --spider -t 5 --passive-ftp -P ${DL_DIR} '${URI}'")
+        setifunset(ld, "UPDATECOMMAND", "")
+        setifunset(ld, "UPDATECOMMAND_cvs", "/usr/bin/env cvs -d${CVSROOT} update ${CVSCOOPTS}")
+        setifunset(ld, "UPDATECOMMAND_svn", "/usr/bin/env svn update ${SVNCOOPTS}")
+        setifunset(ld, "BB_GENERATE_MIRROR_TARBALLS", "1")
+
+        localdir=data.getVar("LAYER_REMOTE_STASH", 1)
+
+        if not os.path.isdir(localdir):
+            bb.utils.mkdirhier(localdir)
+        ld.setVar("DL_DIR", localdir)
+        ld.delVar("MIRRORS")
+        ld.delVar("PREMIRRORS")
+        ld.delVar("SRC_TARBALL_STASH")
+	return ld 
+    
+    """
+    Unpack remote layer. Code largely pulled from collections.inc
+    """
+    def _layerUnpack(self, layer, data):
+        """ Unpack a layer archive and return the path to it. """
+        from hashlib import md5
+
+        handlers = {
+            ("tar"): "tar x --no-same-owner -f %s",
+            ("tar.gz", "tgz", "tar.Z"): "tar xz --no-same-owner -f %s",
+            ("tar.bz2", "tbz", "tbz2"): "tar xj --no-same-owner -f %s",
+            ("zip", "jar"): "unzip -q -o %s",
+        }
+        basename = os.path.basename(layer)
+        try:
+            cmd, name = ((cmd, basename[:-len(e)-1]) for (exts, cmd) in handlers.iteritems()
+                         for e in exts
+                         if basename.endswith(e)).next()
+        except StopIteration:
+            bb.fatal("No method available to unpack %s (unsupported file type?)" % layer)
+        else:
+            outpath = os.path.join(data.getVar("LAYER_UNPACKDIR", 1), name)
+            cmd = "cd %s && PATH=\"%s\" %s" % (outpath, data.getVar("PATH", 1), cmd)
+
+        try:
+            layerdata = open(layer, "r").read()
+        except IOError:
+            bb.fatal("Unable to open %s to calculate md5 sum" % layer)
+
+        md5obj = md5()
+        md5obj.update(layerdata)
+        md5sum = md5obj.hexdigest()
+        md5file = os.path.join(outpath, "md5")
+        if os.path.exists(md5file):
+            try:
+                oldmd5sum = open(md5file).read()
+            except IOError:
+                pass
+            else:
+                if oldmd5sum == md5sum:
+                    self.parselog.debug(1, "Using existing %s for layer '%s'" % (outpath, name))
+                    return outpath, name
+
+            self.parselog.plain("Removing old unpacked layer at %s" % outpath)
+        
+        if not os.path.isdir(outpath):
+            os.makedirs(outpath)
+
+        self.parselog.debug(1,"Unpacking %s to %s/" % (layer, outpath))
+        ret = os.system(cmd % layer)
+        if ret != 0:
+            bb.fatal("Unable to unpack %s" % layer)
+        md5out = open(md5file, "w")
+        md5out.write(md5sum)
+        md5out.close()
+        return outpath, name
+
+    """
+    Add method for grab layer specific data from uri.
+    """
+    def _getLayerSettings(self, urlinfo):
+        layersettings={}
+        layersettingslist = ["layerBase"]
+        for setting in urlinfo.path.split(";") + urlinfo.params.split(";"):
+            for layersetting in layersettingslist:
+                if setting.startswith("%s=" % layersetting):
+                    layersettings[layersetting] = setting.split("=")[1]
+        return [layersettings]
+
+    """
+    Main hooks for grabing remote layers
+    """
+    def _getRemoteLayers(self, layers, data):
+        from itertools import izip, chain
+        from glob import glob
+        from urlparse import urlparse, urlunparse
+	from bb.fetch.layer import LayerFetch
+        if not layers:
+            return []
+    
+        globbed = []
+        layersettings = []
+        localpath = ""
+	ld = self._setFetchValues(data)
+        for path in layers:
+            pathurl = urlparse(path)
+            layersettings += self._getLayerSettings(pathurl)
+            if not pathurl[0]:
+                localpath = glob(os.path.normpath(path))
+            else:
+                tarballpath = LayerFetch(path,pathurl,self.parselog, ld).getTarballPath()
+                if tarballpath:
+                    localpath=[tarballpath]
+            if not localpath:
+                self.parselog.plain("No matches in filesystem for %s in BBLAYERS" % path)
+            globbed += localpath
+        layers = globbed
+        layers_unpacked = ""
+        for (layer, settings) in izip(layers, layersettings):
+            origpath = layer
+            basepath=""
+            if "layerBase" in settings:
+               basepath = settings["layerBase"]
+            if not os.path.isdir(layer):
+                unpacked, name = self._layerUnpack(layer, ld)
+                if unpacked:
+                    layer = unpacked
+                    for dir in glob("%s/*/" % layer):
+                        layerdir = dir + "/" +  basepath
+                        layers_unpacked += layerdir + " "
+            else:
+                layers_unpacked += layer + "/" + basepath + " "
+        return layers_unpacked.encode('ascii').split()
+
+
-- 
1.7.3.2

[PATCH 0/1] Add support for remote layering.

[Jeremy Puhlman]

Missed prefix mail. Resent due to limited response.

> * Maybe need to split into "bootstrap" steps (e.g where pseduo is
>   established, layers downloaded etc)

So for the most part, the last product we shipped was largely arranged
in this manner. We were using collections, but in many cases layers is
just a reimplementation at a lower level(which is definitely goodness).
During ELC I had a number of discussions primarily around remote layer
fetch and most of the responses were meh, or that they wanted to make
the tool external to bitbake. So after the conference was over I went
ahead and worked out a version of the implementation  that could be done
either in a include or in the bitbake code itself. The version in the
git repo below is obviously just the bitbake implementation.

I have a rough implementation largely based off of collection.inc + some
enhancements we made internally for remote collection fetching. Some of
it is messy(setting up the early fetching vars with out using another
config), but for the ones I list in the patch header it appears to work
well.

I do not believe that this is complete, but it is probably a reasonable
starting point.

Pull URL: git://gitorious.org/jpwork/bb.git
Branch: remote-layering
Browse: https://gitorious.org/jpwork/bb/trees/remote-layering


---
Add support for remote layering.

The bulk of this patch is based on Chris Larson's collection.inc,
but refactored for use inside bitbake, and for layers.

Adds two new configuration options:

LAYER_UNPACKDIR - directory to unpack downloaded layers for use in
configuration.
LAYER_REMOTE_STASH - location to store downloaded layers for use in
configuration.

Layers can be specified in the current manner:
BBLAYERS = " \
   /path/to/layer/one \
   /path/to/layer/two \
"

Or you can specify uris:

BBLAYERS = " \
	http://url.to.my.tarball/tarball.tar.gz \
	git://gitserver.com/git/repo;proto=http \
        /path/to/layer/three \
        file:///path/to/layer/four \
"

Currently there is a single layer option, that can be added to a uri,
layerBase=<subpath>.
This option would be used to specify if a layer starts somewhere other
then the base of the
tarball/scm repository. For example if you wanted to add oe-core you
would do:

BBLAYERS  =
"git://git.openembedded.org/openembedded-core;protocol=git;tag=master;layerBase=meta"

Currently for fetch the following uris should work  :
'git','http','https','ftp','file'
Currently for fetch2 the following uris should work :
'git','http','https','ftp','file','svn'

There are certainly some ugly bits in here, and there are certainly lots
of places to improve,
but this gets a good chunk of the base support in.

Signed-off-by: Jeremy Puhlman <jpuhlman at mvista.com>

Re: [PATCH] Add support for remote layering.

[Richard Purdie]

Hi Jeremy,

Sorry for the delayed response, I took the opportunity to take some
vacation for a change and its taken me a few days to catch up on email.

Thanks for posting this. Whilst some of what I mention below may sound
negative, its not meant this way and what you have gives a nice basis to
start some discussion from on this topic. I do want to take a step back
and ensure we integrate this functionality at the right level within the
stack. I've made comments below.

On Thu, 2011-04-28 at 11:09 -0700, Jeremy Puhlman wrote:
> The bulk of this patch is based on Chris Larson's collection.inc,
> but refactored for use inside bitbake, and for layers.
> 
> Adds two new configuration options:
> 
> LAYER_UNPACKDIR - directory to unpack downloaded layers for use in configuration.
> LAYER_REMOTE_STASH - location to store downloaded layers for use in configuration.
> 
> Layers can be specified in the current manner:
> BBLAYERS = " \
>    /path/to/layer/one \
>    /path/to/layer/two \
> "
> 
> Or you can specify uris:
> 
> BBLAYERS = " \
> 	http://url.to.my.tarball/tarball.tar.gz \
> 	git://gitserver.com/git/repo;proto=http \
>         /path/to/layer/three \
>         file:///path/to/layer/four \
> "

This is good but what I don't really like about this is that whilst you
can specify where to get the layer from, it isn't clear where the layer
ends up locally. Compare the above to something like:

BBLAYERS = " \
 	/path/to/layer/one;srcuri=http://url.to.my.tarball/tarball.tar.gz \
 	/path/to/layer/two;srcuri=git://gitserver.com/git/repo;proto=http \
        /path/to/layer/three \
        /path/to/layer/four \
"

This is more ugly but it does clearly set an expectation for both where
its coming from and where it ends up on disk. It also means we can then
set specific revisions to checkout or other information e.g. whether the
revisions should auto-increment. The syntax would probably come
naturally to anyone who has used SRC_URI (and hence it also flows nicely
into the fetcher code).

> Currently there is a single layer option, that can be added to a uri, layerBase=<subpath>.
> This option would be used to specify if a layer starts somewhere other then the base of the
> tarball/scm repository. For example if you wanted to add oe-core you would do:

Ideally this could become an option that any of our fetchers would
support...

>  lib/bb/cooker.py       |    3 +-
>  lib/bb/fetch/layer.py  |   65 ++++++++++++++++
>  lib/bb/fetch2/layer.py |   76 ++++++++++++++++++
>  lib/bb/remotelayer.py  |  197 ++++++++++++++++++++++++++++++++++++++++++++++++

I'm not going to dive into the patches at this point but "layer" isn't
really a fetcher as such as it doesn't correspond to a specific source
type, its just a wrapper around other fetch methods. I'm therefore a
little worried this confuses the abstraction we currently have there.

Going back to the high level approach discussion, I'm also wondering if
there should be some "bblayers" tool which gets run prior to bitbake
(maybe automatically in some cases) which handles the parsing of the
BBLAYERS options and does whatever is needed to handle the setup there,
then hands off to bitbake itself.

I'm very keen to get the abstraction level of this code right so I'd be
interested in your (and others) thoughts on this...

Cheers,

Richard

Re: [PATCH] Add support for remote layering.

[Jeremy Puhlman]

On 5/6/2011 6:15 AM, Richard Purdie wrote:
> Hi Jeremy,
> 
> Sorry for the delayed response, I took the opportunity to take some
> vacation for a change and its taken me a few days to catch up on email.

No worries. Everyone needs some time off. :)

> Thanks for posting this. Whilst some of what I mention below may sound
> negative, its not meant this way and what you have gives a nice basis to
> start some discussion from on this topic. I do want to take a step back
> and ensure we integrate this functionality at the right level within the
> stack. I've made comments below.

Understood.



>> Or you can specify uris:
>>
>> BBLAYERS = " \
>> 	http://url.to.my.tarball/tarball.tar.gz \
>> 	git://gitserver.com/git/repo;proto=http \
>>         /path/to/layer/three \
>>         file:///path/to/layer/four \
>> "
> 
> This is good but what I don't really like about this is that whilst you
> can specify where to get the layer from, it isn't clear where the layer
> ends up locally. Compare the above to something like:
> 
> BBLAYERS = " \
>  	/path/to/layer/one;srcuri=http://url.to.my.tarball/tarball.tar.gz \
>  	/path/to/layer/two;srcuri=git://gitserver.com/git/repo;proto=http \
>         /path/to/layer/three \
>         /path/to/layer/four \
> "

So currently it gets stashed in the locations specified above. Currently
the code has the ability to extract values out of the uri as new bits.
The above would be reasonable, but you could do as well:

 BBLAYERS = " \
>  	http://url.to.my.tarball/tarball.tar.gz;local=/path/to/layer \
>  	git://gitserver.com/git/repo;proto=httplocalLayer=/path/to/layer/two \
>         /path/to/layer/three \
>         /path/to/layer/four \

With the current implementation the above is fairly straight forward.
(not that I am implying implementation should necessarily dictate the
what is done, just noting the above is significantly easier to implement
from the point the code is now.)


> 
> This is more ugly but it does clearly set an expectation for both where
> its coming from and where it ends up on disk. It also means we can then
> set specific revisions to checkout or other information e.g. whether the
> revisions should auto-increment. The syntax would probably come
> naturally to anyone who has used SRC_URI (and hence it also flows nicely
> into the fetcher code).

Actually the code already passes what ever the uri off to the base
fetchers already. So in reality what ever the global and uri setting is
for the fetcher, it should work here as well.

> I'm not going to dive into the patches at this point but "layer" isn't
> really a fetcher as such as it doesn't correspond to a specific source
> type, its just a wrapper around other fetch methods. I'm therefore a
> little worried this confuses the abstraction we currently have there.

This I can definately agree with. The primary issue was I actually was
trying to make the code work with both fetcher implementations largely
because I still use both, and it was either use two separated by ifs on
fetcher implementations, add to __init__ or just drop fetch support at
all. Using the separate file was useful for demonstration purposes. The
actual code placed there is fairly sparce since it just dealt with the
difference in fetchers, and most if not all common code was handled in
the remote layers file above.

> 
> Going back to the high level approach discussion, I'm also wondering if
> there should be some "bblayers" tool which gets run prior to bitbake
> (maybe automatically in some cases) which handles the parsing of the
> BBLAYERS options and does whatever is needed to handle the setup there,
> then hands off to bitbake itself.

So we have been using a tool to generate bblayers via a whole higher
level mirroring setup, and then let bitbake handle all the grunt work
doing the fetching. This works in our case. FWIW I have been trying to
push the company to open those, and we are still in discussions.

> 
> I'm very keen to get the abstraction level of this code right so I'd be
> interested in your (and others) thoughts on this...

Yeah which is kind of why I reposted. The response was limited.

Re: [PATCH] Add support for remote layering.

[Jeremy Puhlman]

> So currently it gets stashed in the locations specified above. Currently
> the code has the ability to extract values out of the uri as new bits.
> The above would be reasonable, but you could do as well:
> 
>  BBLAYERS = " \
>>  	http://url.to.my.tarball/tarball.tar.gz;local=/path/to/layer \
>>  	git://gitserver.com/git/repo;proto=httplocalLayer=/path/to/layer/two \
>>         /path/to/layer/three \
>>         /path/to/layer/four \
> 
> With the current implementation the above is fairly straight forward.
> (not that I am implying implementation should necessarily dictate the
> what is done, just noting the above is significantly easier to implement
> from the point the code is now.)

Arg. Too early this morning. That should have been something like:

>  BBLAYERS = " \
>>  	http://url.to.my.tarball/tarball.tar.gz;localLayer=/path/to/layer \
>>  	git://gitserver.com/git/repo;proto=http;localLayer=/path/to/layer/two \
>>         /path/to/layer/three \
>>         /path/to/layer/four \
>

or something to that effect.

Re: [PATCH] Add support for remote layering.

[Paul Eggleton]

Returning a discussion that has begun on the wiki to the mailing list:

Jeremy Puhlman wrote:
> Paul Eggleton wrote:
>> A good start but naturally we want to avoid the bits that hard-code
>> variable values. I wonder about situations where people want their own
>> versions of layers or to share remote layers (e.g. between an oe-core setup
>> and a poky setup); however people can just use local layers for this.
>
> In reality they are setting "reasonable" defaults. Hardcoding implies
> something that is unchangeable. All the values are "set if unset", so they
> can be changed in the bblayers.conf file. Prior to the introduction of
> layers, all the values were hardcoded, just in a bitbake.conf file, so you
> always had working fetchers. With out that the fetchers are basically broken
> until after you have loaded up the layers. Given the fact that nearly every
> bitbake.conf file contains the same settings for defining fetchcmd and a like,
> simply having a reasonable default setting for the things the fetchers need
> is not really that terrible an idea irrespective of the remote layering.

I think it's just about the placing of these defaults, I don't necessarily 
object to having them in the first place. My main concerns are around the 
defaults for variables like TMPDIR which may end up actually being used in 
preference to what the user has set in local.conf. Obviously we have a 
chicken-and-egg situation where we depend on having the main layer present to 
read any configuration other than bblayers.conf - perhaps this indicates we 
need to restructure some of the config files if we want to make this work 
nicely.

> As for the second question. I have some content tools that I am cleaning up
> that basically provide methods for defining layers and groups of layers
> together for mirroring and sharing. That might address what your talking
> about there. They work along with the patch posted here, but could be
> modified to output things a bit differently if needed.

I'd definitely be interested in seeing these tools and trying to incorporate 
them into what we're doing for layer tooling in 1.1. Is there a lot of work 
left to do before you can release these?

Cheers,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre

Re: [PATCH] Add support for remote layering.

[Jeremy Puhlman]

> I think it's just about the placing of these defaults, I don't necessarily 
> object to having them in the first place. My main concerns are around the 
> defaults for variables like TMPDIR which may end up actually being used in 
> preference to what the user has set in local.conf. Obviously we have a 
> chicken-and-egg situation where we depend on having the main layer present to 
> read any configuration other than bblayers.conf - perhaps this indicates we 
> need to restructure some of the config files if we want to make this work 
> nicely.

So with regard to the code specifically:

TMPDIR really is the only major offender there as far as I can tell, all
the other variables are set in the local copy of the data and disappar
after the fetch. That variable setting could get moved over to the local
copy as well, though we would need to change the way LAYER_REMOTE_STASH
and LAYER_UNPACKDIR are set to make sure they are correct for the rest
of the parse but that is not a big deal. Or we could set those via
something relative other then TMPDIR, TOPDIR perhaps.

General case:

In reality looking at the bulk of the settings there, there are a couple
different groups there.

There is the stuff to make the fetchers work. I think those should
probably make their way in to the the code in general. Setifunset means
they can be changed early(bblayers), or overridden with the first
bitbake.conf and in general they are rarely used in any user
configurations(local.conf) or at least shouldn't. Having working
fetchers from the first parsed line of configuration data seems like a
generally good idea, but that is just me.

There is the stuff only related to the fiddling with remote layers,
which I talked above, can be changed to use any location in general.

> 
> I'd definitely be interested in seeing these tools and trying to incorporate 
> them into what we're doing for layer tooling in 1.1. Is there a lot of work 
> left to do before you can release these?

Work this week has been a bit slammed. Hopefully I will be able to post
something this weekend, maybe Monday.

Re: [PATCH] Add support for remote layering.

[Paul Eggleton]

Hi Jeremy

Have looked at this further and I really think this needs to be a separate 
utility, but one still written in python using bitbake's infrastructure (like 
bitbake-layers). The advantages of this approach:

1) Avoids reworking bitbake's initialisation and avoids possible re-execution. 
A separate utility can take advantage of a full parse and utilise all of the 
user's configuration.

2) Fetching/updating layers should be (able to be) a conscious action rather 
than something that happens implicitly as part of the build process

I'm trying to put something together that does this at the moment, with a view 
to there being components that allow integration with the content tools you 
posted earlier. Will keep you posted.

Cheers,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre

Re: [PATCH] Add support for remote layering.

[Jeremy Puhlman]

On 7/1/2011 6:24 AM, Paul Eggleton wrote:
> Hi Jeremy
> 
> Have looked at this further and I really think this needs to be a separate 
> utility, but one still written in python using bitbake's infrastructure (like 
> bitbake-layers). The advantages of this approach:
> 
> 1) Avoids reworking bitbake's initialisation and avoids possible re-execution.

Neither of those are issues in the current patch. Setting up the
fetchers so they work, inside a local copy of the data is hardly
reworking the initialization.

> A separate utility can take advantage of a full parse and utilise all of the 
> user's configuration.

This is certainly the case, however in the time that we have used this
type of structure we have never needed all of the config data because
you are always more less very early in the process. More or less you
could create a bitbake-layers style "utilty" with the current patch. In
the current patch, there is only a single line of code in the cooker,
everything else is competely isolated.

> 2) Fetching/updating layers should be (able to be) a conscious action rather 
> than something that happens implicitly as part of the build process

This seems rather arbitrary. As noted this is not some theoretical
process that I am proposing. We have tons of customers already use, and
have used this type of model for a very long time. While I can certainly
see the advantages of an external tool, why is it an either or? the
reality is with the current patch, we already have bitbake-layers style
utility. Its called bitbake. :) Run it with no argument and the first
thing it does is fetch everything.


> I'm trying to put something together that does this at the moment, with a view 
> to there being components that allow integration with the content tools you 
> posted earlier. Will keep you posted.

For what it is worth, I would still like to see something like the patch
get integrated. The single line in the cooker could be surrounnded by a
BB_ var, and you more or less wouldn't know the difference. Also if
BBLAYERS is just directories like it is now, every line is more or less
ignored and passed back untouched.



-- 
Jeremy Puhlman
Montavista Sofware, LLC.

Re: [PATCH] Add support for remote layering.

[Jeremy Puhlman]

Paul,

> 
> 2) Fetching/updating layers should be (able to be) a conscious action rather 
> than something that happens implicitly as part of the build process
> 

Just a quick follow up, the part of this statement that to me is wrong,
is it appears you are dictating work flow. While, imho, the layer
management code which is above this level, is the place to dicate
workflow. The patch is to core bitbake and isolated at that. This patch
adds an additional work flow, and can be used in either state(the
results of a current BBLAYERS setting is exactly the same today as it as
after the patch).  More or less the patch is useful in a scope larger
then just layer-tooling.

-- 
Jeremy Puhlman
Montavista Sofware, LLC.

Re: [PATCH] Add support for remote layering.

[Richard Purdie]

On Fri, 2011-07-01 at 11:43 -0700, Jeremy Puhlman wrote:
> Just a quick follow up, the part of this statement that to me is wrong,
> is it appears you are dictating work flow. While, imho, the layer
> management code which is above this level, is the place to dicate
> workflow. The patch is to core bitbake and isolated at that. This patch
> adds an additional work flow, and can be used in either state(the
> results of a current BBLAYERS setting is exactly the same today as it as
> after the patch).  More or less the patch is useful in a scope larger
> then just layer-tooling.

Let me try and convey at least one of the things I'm worrying about with
this patch. We're looking at establishing layer tooling which operate
outside of or "above" bitbake. There are a variety of reasons for that,
rightly or wrongly but this patch implies we're going to reverse that.

If we go forward with the patch and don't reverse that decision it means
that layers can sometimes be a thing that are "above" bitbake, sometimes
not depending on the circumstances. I'm trying hard to avoid that kind
of interwoven complexity as it makes code changes very hard to make in
the future and leads to frequent breakage where multiple usage methods
exist.

As an example consider the case someone calls a script asking for new
commits in layer Y to be added to flattened layer Z. We then have to add
code checking if layer Y or layer X is a "bitbake remote layer" and then
handle the updating of the layer accordingly. Can something external to
the bitbake code do that? When we extend the later tooling are we
expected to extend this code to match functionality?

The abstraction in the remote layers code isn't strong enough to cope
with that kind of interaction in its own right and I'm not sure its
possible to do that simply with a line in a bblayers.conf file and still
be readable.

So I guess my question is how do you see this moving forwards? Are you
planning to use external layer tooling at all or are you wanting
anything that external tooling can do exposed also by the bblayers.conf
URI? If this is some kind of stopgap solution for compatibility and is a
single blocking issue for Montavista that might be reasonable. If its
the start of a move to reverse the order of the stack and put bitbake at
the top and layer tooling secondary, I'm worried.

Also looking at the patch at a technical level, is there any reason to
use _layerUnpack() instead of the fetch2 unpack method? The patch is
much cleaner than it was, that much is good but needing to set so many
fetcher variables like that is a sign that worries me a little. I guess
the solution there is to code them as fallback defaults into the
fetchers themselves.

Cheers,

Richard

Re: [PATCH] Add support for remote layering.

[Jeremy Puhlman]

On 7/1/2011 2:37 PM, Richard Purdie wrote:
> On Fri, 2011-07-01 at 11:43 -0700, Jeremy Puhlman wrote:
>> Just a quick follow up, the part of this statement that to me is wrong,
>> is it appears you are dictating work flow. While, imho, the layer
>> management code which is above this level, is the place to dicate
>> workflow. The patch is to core bitbake and isolated at that. This patch
>> adds an additional work flow, and can be used in either state(the
>> results of a current BBLAYERS setting is exactly the same today as it as
>> after the patch).  More or less the patch is useful in a scope larger
>> then just layer-tooling.
> 
> Let me try and convey at least one of the things I'm worrying about with
> this patch. We're looking at establishing layer tooling which operate
> outside of or "above" bitbake. There are a variety of reasons for that,
> rightly or wrongly but this patch implies we're going to reverse that.

I think that only is true if you view the patch in the scope of the only
purpose for the patch is an element of the layer tooling. Being able to
specify a uri as a layer is in and of it self a useful ability,
irrespective of whether there exists higher order tools to create that list.

> If we go forward with the patch and don't reverse that decision it means
> that layers can sometimes be a thing that are "above" bitbake, sometimes
> not depending on the circumstances. 

This kind of goes back to work flow issue. Unless you plan to make it so
you cannot use the layers with out the specific layering tools that are
in development, bitbake will always need to support different work flows.

> I'm trying hard to avoid that kind
> of interwoven complexity as it makes code changes very hard to make in
> the future and leads to frequent breakage where multiple usage methods
> exist.

Well in this case, there is little that is interwoven. The remote layers
bit in the main thread of the execution is a single line in the cooker.
Paul was saying that he wanted to more or less still implement the tool
with in the bitbake code. Ironically, that will likely be more
interwoven then this patch.

The patch on its own with out regard to layer tooling for this project,
can pretty much stand on its own(which is why I provided it a month or
two before we pushed up the content tools).

> 
> As an example consider the case someone calls a script asking for new
> commits in layer Y to be added to flattened layer Z. We then have to add
> code checking if layer Y or layer X is a "bitbake remote layer" and then
> handle the updating of the layer accordingly. 

When ever you evaluate any of the layers, they are downloaded and
unpacked prior to examination of any of the meta data(stay
bblayers.conf). More or less any place where you would evalute
bblayers.conf you would evaluate the collection uri's. How you
update(i.e. if they are git for example) should be setable via the
already in place fetch mechanism.

More or less you would just need to run the BBLAYERS var through the
remote layers class then the end result is the BBLAYERS is just a list
of directories like it is now, which is why there is only one line of
change in the current thread of execution.

> Can something external to
> the bitbake code do that?

Unless your planning on creating tooling that doesn't use any of the
bitbake internals, not unless you want to duplicate a whole bunch of code.

> When we extend the later tooling are we
> expected to extend this code to match functionality?

Well if the end results of the layer tooling is a BBLAYERS with a list
of directories as it is now, there is nothing to extend. The code
handles that now. More or less if unless you are going in a drastically
different direction with the BBLAYERS list, I am not sure what would
need to be updated.

> 
> The abstraction in the remote layers code isn't strong enough to cope
> with that kind of interaction in its own right and I'm not sure its
> possible to do that simply with a line in a bblayers.conf file and still
> be readable.

My problem with this is currently using the same uri mechanism used in
the rest of bitbake fetches. Why pick on it here?

> 
> So I guess my question is how do you see this moving forwards? 

Well paul more or less said he was going to be writing the remote
layering tools with hooks in to bitbake like bitbake-layers, so I am not
entirely certain what external means in this context. With the current
patch, you could basically write the entire tool calling in to bitbake
and pass uri's to the layer fetch code, and either pass in our pass out
the path of the final destination.


> Are you
> planning to use external layer tooling at all or are you wanting
> anything that external tooling can do exposed also by the bblayers.conf
> URI?

Well it is two fold. Right now, the actual form of the end layer-tooling
doesn't exist. So no I really couldn't commit to using what ever is
produced. However, the entire reason why we provided this patch and the
content-tools, is so that we could nudge them in a direction were we
would be much more likely to use them.

More or less the entire point of the yocto project was to stop inventing
the same tools over and over again for each distro, so we can
concentrate on the actual value adds that each company/distro/group
brings to the table.

Basically what your implementing here has already been done, ergo in the
spirit of the purpose of the project, we were trying to avoid
reinventing the same tools again either here or in
yocto/oe-core/bitbake. The path you guys are tracking, is more or less
the same track we started on and we ended up with what we provided to you.

> If this is some kind of stopgap solution for compatibility and is a
> single blocking issue for Montavista that might be reasonable. If its
> the start of a move to reverse the order of the stack and put bitbake at
> the top and layer tooling secondary, I'm worried.

Actually the remote fetching is just a mechanism to empower the
layer-tooling and not duplicate code. I am not sure of you have looked
at the content-tools I provided last month, but those are basically our
"layer-tools". They create two things, collections(which at this point
would be layers) and solutions, which are sets of collections/layers
that work together. All of that is well above the bitbake level. Given
that the layers need to exist/be defined prior to the first execution of
bitbake, I wouldn't even begin to fathom how one would make
layer-tooling secondary to bitbake.

So I don't think I am trying to do that.

> 
> Also looking at the patch at a technical level, is there any reason to
> use _layerUnpack() instead of the fetch2 unpack method?

The code is more or less, a minor reworking of OE's collection.inc from
Chris. So it has just carried that a long for the ride. In addition, it
would mean the code could only work with fetch2. fetch does not provide
unpack. I originally put this together to work on both version of
fetch(which it does). It shouldn't be an issue to call unpack if the
fetcher is fetch2.

> The patch is
> much cleaner than it was, that much is good but needing to set so many
> fetcher variables like that is a sign that worries me a little. 

I have not submitted an updated patch. What you are looking at would
have to be the original. :) I got feed back and responded and never got
any comment on those responses.

> I guess
> the solution there is to code them as fallback defaults into the
> fetchers themselves.

Well as I noted at the start, the setting of the fetcher stuff is
primarily because, the fetchers don't start off with reasonable defaults
so they are more or less broken until we parse conf/bitbake.conf.

In the setup I/we use, we do not start our projects with any local
layers/collections at all, so we would not have access to bitbake.conf,
unless we went back to sourceing the mini one that used to be provided.
We can move the setting of that stuff to the init of the fetchers, and
that wouldn't be a big deal.

But once you eliminate the stuff to make fetching work, the list would
almost entirely go away.

Re: [PATCH] Add support for remote layering.

[Richard Purdie]

On Fri, 2011-07-01 at 17:33 -0700, Jeremy Puhlman wrote:
> On 7/1/2011 2:37 PM, Richard Purdie wrote:
> > On Fri, 2011-07-01 at 11:43 -0700, Jeremy Puhlman wrote:
> >> Just a quick follow up, the part of this statement that to me is wrong,
> >> is it appears you are dictating work flow. While, imho, the layer
> >> management code which is above this level, is the place to dicate
> >> workflow. The patch is to core bitbake and isolated at that. This patch
> >> adds an additional work flow, and can be used in either state(the
> >> results of a current BBLAYERS setting is exactly the same today as it as
> >> after the patch).  More or less the patch is useful in a scope larger
> >> then just layer-tooling.
> > 
> > Let me try and convey at least one of the things I'm worrying about with
> > this patch. We're looking at establishing layer tooling which operate
> > outside of or "above" bitbake. There are a variety of reasons for that,
> > rightly or wrongly but this patch implies we're going to reverse that.
> 
> I think that only is true if you view the patch in the scope of the only
> purpose for the patch is an element of the layer tooling. Being able to
> specify a uri as a layer is in and of it self a useful ability,
> irrespective of whether there exists higher order tools to create that list.

I agree the patch has a use in its own right, standalone. I am however
asking us all to take a step back and consider the big picture which is
what I need do with a lot of what we're doing.

My point is that whilst in isolation its ok, I don't think that approach
can scale to fulfil all the varying needs of our users. If it can't, we
need to look at what can, and then how we could include equivalent
functionality.

> Well in this case, there is little that is interwoven. The remote layers
> bit in the main thread of the execution is a single line in the cooker.
> Paul was saying that he wanted to more or less still implement the tool
> with in the bitbake code. Ironically, that will likely be more
> interwoven then this patch.
> 
> The patch on its own with out regard to layer tooling for this project,
> can pretty much stand on its own(which is why I provided it a month or
> two before we pushed up the content tools).

The interwoven comment has nothing to do with how many lines of code it
adds/removes. My concern is that we have a ton of different operations
we really need to have around layers and they can't all work through the
bitbake command itself as it simply wasn't designed for it.

Ending up in a case where the bitbake command may do some things to
layers some of the time isn't the consistent user experience I'd like to
have. Would you like to explain to a user that yes, bitbake can fetch a
remote layer but can't update it?

Even with the code we're discussing, you need an external tool to create
the list and to update it at appropriate points from upstream so you
really might as well have that code actually do the fetch as well
(calling bitbake's fetchers)?

I'm much more worried about the workflow and its implications than I am
about the actual code.

> > As an example consider the case someone calls a script asking for new
> > commits in layer Y to be added to flattened layer Z. We then have to add
> > code checking if layer Y or layer X is a "bitbake remote layer" and then
> > handle the updating of the layer accordingly. 
> 
> When ever you evaluate any of the layers, they are downloaded and
> unpacked prior to examination of any of the meta data(stay
> bblayers.conf). More or less any place where you would evalute
> bblayers.conf you would evaluate the collection uri's. How you
> update(i.e. if they are git for example) should be setable via the
> already in place fetch mechanism.
> 
> More or less you would just need to run the BBLAYERS var through the
> remote layers class then the end result is the BBLAYERS is just a list
> of directories like it is now, which is why there is only one line of
> change in the current thread of execution.

As I've said, I don't care about the single line of code change. The
important thing is whether the work flow can consistently scale into
what we as project all need. Unfortunately at the moment its looking
like it doesn't.

> > The abstraction in the remote layers code isn't strong enough to cope
> > with that kind of interaction in its own right and I'm not sure its
> > possible to do that simply with a line in a bblayers.conf file and still
> > be readable.
> 
> My problem with this is currently using the same uri mechanism used in
> the rest of bitbake fetches. Why pick on it here?

I think layers are a bit different to a lot of the source code we
currently fetch. We might want to push things back, transfer commits
between layers and perform a number of other operations. These
operations all require "state" type data that currently we can't store
in a SRC_URI. I'm not even sure we want to store it there.

> Well paul more or less said he was going to be writing the remote
> layering tools with hooks in to bitbake like bitbake-layers, so I am not
> entirely certain what external means in this context. With the current
> patch, you could basically write the entire tool calling in to bitbake
> and pass uri's to the layer fetch code, and either pass in our pass out
> the path of the final destination.

I think Paul is planning something very similar in spirit to the code
you have with the difference you'd use a different tool to actually
manipulate the build environment and fetch the layers first rather than
having bitbake do it itself.

> > Are you
> > planning to use external layer tooling at all or are you wanting
> > anything that external tooling can do exposed also by the bblayers.conf
> > URI?
> 
> Well it is two fold. Right now, the actual form of the end layer-tooling
> doesn't exist. So no I really couldn't commit to using what ever is
> produced.

There has been discussion about this in person at Collab/ELC and on
the mailing lists/wikis but we probably need to do better at
communicating this I guess :/

>  However, the entire reason why we provided this patch and the
> content-tools, is so that we could nudge them in a direction were we
> would be much more likely to use them.
> 
> More or less the entire point of the yocto project was to stop inventing
> the same tools over and over again for each distro, so we can
> concentrate on the actual value adds that each company/distro/group
> brings to the table.
> 
> Basically what your implementing here has already been done, ergo in the
> spirit of the purpose of the project, we were trying to avoid
> reinventing the same tools again either here or in
> yocto/oe-core/bitbake. The path you guys are tracking, is more or less
> the same track we started on and we ended up with what we provided to you.

The whole point of Yocto is to collaborate. I'm pleased to have your
input in the form of the patches and I really do appreciate that.

This doesn't however guarantee we take them "as is" since we've also
consulted with a number of other people about what their needs are and
your solution whilst evidently perfect for you doesn't meet all the
criteria we established during the planning process. We're trying to
write some tools that should work for everyone. It may require some
changes in process for some people, hopefully these will be logical and
improve the user experience and understanding.

There are a variety of ways I think we could even directly make your use
case work (such as bitbake automatically calling the external update
tool if some environment variable is set). Collaboration means there are
more than just your requirements that need to be worked into this at
this point.

> > If this is some kind of stopgap solution for compatibility and is a
> > single blocking issue for Montavista that might be reasonable. If its
> > the start of a move to reverse the order of the stack and put bitbake at
> > the top and layer tooling secondary, I'm worried.
> 
> Actually the remote fetching is just a mechanism to empower the
> layer-tooling and not duplicate code. I am not sure of you have looked
> at the content-tools I provided last month, but those are basically our
> "layer-tools". They create two things, collections(which at this point
> would be layers) and solutions, which are sets of collections/layers
> that work together. All of that is well above the bitbake level. Given
> that the layers need to exist/be defined prior to the first execution of
> bitbake, I wouldn't even begin to fathom how one would make
> layer-tooling secondary to bitbake.
> 
> So I don't think I am trying to do that.

So why put all the work into that content-tool code and then have the
workflow so bitbake does the actual fetching? It shouldn't be hard to
write a "bitbake-layer-fetch" script that does this and it would make a
clearer separation of responsibility for the different scripts. What is
the reason for not doing that?

> > Also looking at the patch at a technical level, is there any reason to
> > use _layerUnpack() instead of the fetch2 unpack method?
> 
> The code is more or less, a minor reworking of OE's collection.inc from
> Chris. So it has just carried that a long for the ride. In addition, it
> would mean the code could only work with fetch2. fetch does not provide
> unpack. I originally put this together to work on both version of
> fetch(which it does). It shouldn't be an issue to call unpack if the
> fetcher is fetch2.

What's the opinion of fetch2 at this point? I'm hoping we can rely on
that for all future development work at this point.

> > The patch is
> > much cleaner than it was, that much is good but needing to set so many
> > fetcher variables like that is a sign that worries me a little. 
> 
> I have not submitted an updated patch. What you are looking at would
> have to be the original. :) I got feed back and responded and never got
> any comment on those responses.

I'm probably confusing this with the first version you sent out I guess,
sorry.

> > I guess
> > the solution there is to code them as fallback defaults into the
> > fetchers themselves.
> 
> Well as I noted at the start, the setting of the fetcher stuff is
> primarily because, the fetchers don't start off with reasonable defaults
> so they are more or less broken until we parse conf/bitbake.conf.
> 
> In the setup I/we use, we do not start our projects with any local
> layers/collections at all, so we would not have access to bitbake.conf,
> unless we went back to sourceing the mini one that used to be provided.
> We can move the setting of that stuff to the init of the fetchers, and
> that wouldn't be a big deal.

I understand that. You're not thinking about what I'm suggesting ;-)

> But once you eliminate the stuff to make fetching work, the list would
> almost entirely go away.

For example, what I mean is that instead of:

        fetchcmd = data.getVar("FETCHCOMMAND", d, True)

in wget.py, we could update this to something like:

        defaultcmd = "/usr/bin/env wget -t 5 -q --passive-ftp -P ${DL_DIR} ${URI}"
        fetchcmd = data.getVar("FETCHCOMMAND", d, True) or defaultcmd

Cheers,

Richard

Re: [PATCH] Add support for remote layering.

[Paul Eggleton]

On Saturday 02 July 2011 01:33:58 Jeremy Puhlman wrote:
> On 7/1/2011 2:37 PM, Richard Purdie wrote:
> > If we go forward with the patch and don't reverse that decision it means
> > that layers can sometimes be a thing that are "above" bitbake, sometimes
> > not depending on the circumstances.
>
> This kind of goes back to work flow issue. Unless you plan to make it so
> you cannot use the layers with out the specific layering tools that are
> in development, bitbake will always need to support different work flows.

We aren't going to tie bitbake layer support of local layers to the layer 
tooling, no. We're expressly trying to avoid that.

You suggest that I'm trying to "dictate workflow" however I'd argue that I'm 
doing the opposite (see below regarding updating).

> > I'm trying hard to avoid that kind
> > of interwoven complexity as it makes code changes very hard to make in
> > the future and leads to frequent breakage where multiple usage methods
> > exist.
> 
> Well in this case, there is little that is interwoven. The remote layers
> bit in the main thread of the execution is a single line in the cooker.
> Paul was saying that he wanted to more or less still implement the tool
> with in the bitbake code. Ironically, that will likely be more
> interwoven then this patch.

I don't think that they will. Firstly, no hooks are needed to fetch a layer 
remotely - it's just a fetch; you call cooker to run through a parse and then 
run the fetch operations you need. No changes to the bitbake core should be 
necessary, as far as I can see. (I don't object to reworking fetcher 
initialisation so that they are set up correctly "out of the box", though, 
that may be helpful to both approaches.)

Secondly, the changes I have previously implemented within bitbake for use by 
bitbake-layers are generic and have negligible impact on bitbake operation. 
If you have any technical concerns with these, in all seriousness please reply 
to the patches on bitbake-devel; there's always room for improvement.

> When ever you evaluate any of the layers, they are downloaded and
> unpacked prior to examination of any of the meta data(stay
> bblayers.conf). More or less any place where you would evalute
> bblayers.conf you would evaluate the collection uri's. How you
> update(i.e. if they are git for example) should be setable via the
> already in place fetch mechanism.

Updating is something I would like to allow to be a conscious action. That 
way, if you want to fetch down the metadata, then disconnect and carry on 
building, you can do so easily. Also, if you want to stay with the current 
version you have on disk, you know bitbake is not going to update the metadata 
in the course of doing the build, because you didn't explicitly ask it to. 
You're also free to make any additional changes on top of the fetched metadata 
before running the build. If bitbake is doing all the fetching/updating of 
metadata then immediately jumping into the build, there's no room for that - 
unless you use Ctrl+C to break out, which isn't really ideal.

I think the piece I am missing at the moment is why having it as an external 
tool that largely replicates the same outward functionality as having it 
within bitbake presents a problem for the use case that you have. Is there 
some technical capability that we couldn't have using this approach - other 
than the fact that you're just calling bitbake and it does everything? If 
that's the only objection, would it not achieve the same thing if you had a 
small shell script that ran the fetch/update then bitbake?

> More or less you would just need to run the BBLAYERS var through the
> remote layers class then the end result is the BBLAYERS is just a list
> of directories like it is now, which is why there is only one line of
> change in the current thread of execution.
> 
> > Can something external to
> > the bitbake code do that?
> 
> Unless your planning on creating tooling that doesn't use any of the
> bitbake internals, not unless you want to duplicate a whole bunch of code.

I don't plan to duplicate any significant amount of code. I'm not seeing that 
you couldn't do what was described using an external tool.

> > When we extend the later tooling are we
> > expected to extend this code to match functionality?
> 
> Well if the end results of the layer tooling is a BBLAYERS with a list
> of directories as it is now, there is nothing to extend. The code
> handles that now. More or less if unless you are going in a drastically
> different direction with the BBLAYERS list, I am not sure what would
> need to be updated.

Well, with my solution I would prefer to see BBLAYERS unchanged, so it just 
points to the local checkout of the remote layer.

> > The abstraction in the remote layers code isn't strong enough to cope
> > with that kind of interaction in its own right and I'm not sure its
> > possible to do that simply with a line in a bblayers.conf file and still
> > be readable.
> 
> My problem with this is currently using the same uri mechanism used in
> the rest of bitbake fetches. Why pick on it here?

Fetching metadata and fetching source code are two different things. You don't 
really expect to have to do anything to the source code before bitbake builds 
it - bitbake and whatever metadata you have takes care of that. However with 
the metadata you may wish to make modifications prior to building.

> > So I guess my question is how do you see this moving forwards?
> 
> Well paul more or less said he was going to be writing the remote
> layering tools with hooks in to bitbake like bitbake-layers, so I am not
> entirely certain what external means in this context. With the current
> patch, you could basically write the entire tool calling in to bitbake
> and pass uri's to the layer fetch code, and either pass in our pass out
> the path of the final destination.

External means it's not part of the normal bitbake execution flow and there's 
nothing that needs to be changed within the bitbake core, unless it's of a 
generic nature (and right now I don't see any needed changes with my 
solution). No hooks should be necessary.

> > Are you planning to use external layer tooling at all or are you wanting
> > anything that external tooling can do exposed also by the bblayers.conf
> > URI?
> 
> Well it is two fold. Right now, the actual form of the end layer-tooling
> doesn't exist. So no I really couldn't commit to using what ever is
> produced. However, the entire reason why we provided this patch and the
> content-tools, is so that we could nudge them in a direction were we
> would be much more likely to use them.

And that's a good thing, agreed.

> More or less the entire point of the yocto project was to stop inventing
> the same tools over and over again for each distro, so we can
> concentrate on the actual value adds that each company/distro/group
> brings to the table.

Right. To me, that means going from a list of requirements (and I'll admit, it 
has been an informal list for this particular feature) and trying to come 
up with an implementation that fulfils those requirements.

> Basically what your implementing here has already been done, ergo in the
> spirit of the purpose of the project, we were trying to avoid
> reinventing the same tools again either here or in
> yocto/oe-core/bitbake. The path you guys are tracking, is more or less
> the same track we started on and we ended up with what we provided to you.

You have come up with an implementation that works well for you, and you've 
published that implementation. Thankyou for that. However, there are some 
aspects of that implementation that concern Richard and I so I'm trying to 
come up with an alternative implementation (using some of the same code) that 
meets the requirements. If we're not going to meet some important requirements 
that you have then I'm more than happy to stop and re-evaluate the situation.

> > Also looking at the patch at a technical level, is there any reason to
> > use _layerUnpack() instead of the fetch2 unpack method?
> 
> The code is more or less, a minor reworking of OE's collection.inc from
> Chris. So it has just carried that a long for the ride. In addition, it
> would mean the code could only work with fetch2. fetch does not provide
> unpack. I originally put this together to work on both version of
> fetch(which it does). It shouldn't be an issue to call unpack if the
> fetcher is fetch2.

Understood. However, I'd hope we can soon move to a state where we only 
have one set of fetchers. Is there anything preventing Montavista from using 
fetch2 currently?

> > The patch is
> > much cleaner than it was, that much is good but needing to set so many
> > fetcher variables like that is a sign that worries me a little.
> 
> I have not submitted an updated patch. What you are looking at would
> have to be the original. :) I got feed back and responded and never got
> any comment on those responses.

Well, we definitely would need the "layer" fetcher code to be refactored out; 
Richard made that comment earlier. I would also hope that as you suggested we 
can move to having the initialisation within the fetchers themselves so that 
the layer fetching isn't doing anything special in that regard. We need to 
establish the direction we want to go in however.

Cheers,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre

Re: [PATCH] Add support for remote layering.

[Jeremy Puhlman]

> I agree the patch has a use in its own right, standalone. I am however
> asking us all to take a step back and consider the big picture which is
> what I need do with a lot of what we're doing.
> 
> My point is that whilst in isolation its ok, I don't think that approach
> can scale to fulfil all the varying needs of our users. If it can't, we
> need to look at what can, and then how we could include equivalent
> functionality.

Okay.

> Even with the code we're discussing, you need an external tool to create
> the list and to update it at appropriate points from upstream so you
> really might as well have that code actually do the fetch as well
> (calling bitbake's fetchers)?

Well in its current state, you don't need one, but it makes it a lot
easier. OTOH all this discussion more or less centers around BBLAYERS
which is already easy to use.

> I'm much more worried about the workflow and its implications than I am
> about the actual code.

Okay. More ore less this really only expands the options for workflow.

> I think layers are a bit different to a lot of the source code we
> currently fetch. We might want to push things back, transfer commits
> between layers and perform a number of other operations.

Unless we are writing a new set of fetchers(internal or otherwise), the
limitation is the fetchers, since they don't do this.

> These
> operations all require "state" type data that currently we can't store
> in a SRC_URI. I'm not even sure we want to store it there.

Wouldn't you have the same issue with a git repo tracking recipe?

> 
> There has been discussion about this in person at Collab/ELC and on
> the mailing lists/wikis but we probably need to do better at
> communicating this I guess :/

Yes. Been following a long, trying to contribute where it made sense.
OTOH, in the end, if it is insufficient for our needs we can't use it.
> The whole point of Yocto is to collaborate. I'm pleased to have your
> input in the form of the patches and I really do appreciate that.
> 
> This doesn't however guarantee we take them "as is" since we've also
> consulted with a number of other people about what their needs are and
> your solution whilst evidently perfect for you doesn't meet all the
> criteria we established during the planning process. 

I never suggested that anything be taken as is. In the irc discussions
with Paul, we talked about the bits that we wanted to change.

> We're trying to
> write some tools that should work for everyone. It may require some
> changes in process for some people, hopefully these will be logical and
> improve the user experience and understanding.

Understood, otherwise I wouldn't be having this e-mail exchange with you
at all.

> There are a variety of ways I think we could even directly make your use
> case work (such as bitbake automatically calling the external update
> tool if some environment variable is set). 

Okay.


> Collaboration means there are
> more than just your requirements that need to be worked into this at
> this point.

Nor have I suggested that you guys were.

> What's the opinion of fetch2 at this point? I'm hoping we can rely on
> that for all future development work at this point.

From the I am just using fetchers in a more or less rudimentary manner,
it feels more or less the same(which kind a was the point). I have not
really used any of the additional stuff that has been added in, but more
powerful and functionally rich code is always a positive thing.

Also I am currently supporting fetch with our older product so when I
wrote the patch, making it work with fetch seemed a good idea.

> 
> I'm probably confusing this with the first version you sent out I guess,
> sorry.
> 

That was my point, there was only one. Really doesn't matter.

>> We can move the setting of that stuff to the init of the fetchers, and
>> that wouldn't be a big deal.
> 
> I understand that. You're not thinking about what I'm suggesting ;-)
> 

I caught what you were striving for there.

-- 
Jeremy Puhlman
Montavista Sofware, LLC.

Re: [PATCH] Add support for remote layering.

[Jeremy Puhlman]

> 
> We aren't going to tie bitbake layer support of local layers to the layer 
> tooling, no. We're expressly trying to avoid that.
> 
> You suggest that I'm trying to "dictate workflow" however I'd argue that I'm 
> doing the opposite (see below regarding updating).

The line that led me to dicate:
Fetching/updating layers should be (able to be) a conscious action
rather than something that happens implicitly as part of the build process.

Was not as firm a statement as was thinking it was. Apologies.

> I don't think that they will. Firstly, no hooks are needed to fetch a layer 
> remotely - it's just a fetch; you call cooker to run through a parse and then 
> run the fetch operations you need. No changes to the bitbake core should be 
> necessary, as far as I can see. (I don't object to reworking fetcher 
> initialisation so that they are set up correctly "out of the box", though, 
> that may be helpful to both approaches.)

I think this is kinda where I get hung up. The run through parse part.
Our projects start off with zero meta data other then a local.conf and
bblayers.conf. In earlier days, we provided collections.inc and
bitbake.conf in a local conf directory provided with bitbake. That is
not done anymore(ergo why the fetchers are broken from go, previously
nothing happened until at lease one bitbake.conf was loaded).


> Secondly, the changes I have previously implemented within bitbake for use by 
> bitbake-layers are generic and have negligible impact on bitbake operation. 
> If you have any technical concerns with these, in all seriousness please reply 
> to the patches on bitbake-devel; there's always room for improvement.

Ill dig in to these, I have not done much digging, but I did see the
flurry of recent patches.

> Updating is something I would like to allow to be a conscious action. That 
> way, if you want to fetch down the metadata, then disconnect and carry on 
> building, you can do so easily. Also, if you want to stay with the current 
> version you have on disk, you know bitbake is not going to update the metadata 
> in the course of doing the build, because you didn't explicitly ask it to. 
> You're also free to make any additional changes on top of the fetched metadata 
> before running the build. If bitbake is doing all the fetching/updating of 
> metadata then immediately jumping into the build, there's no room for that - 
> unless you use Ctrl+C to break out, which isn't really ideal.

So not to defend the patch again here, but this is not a problem with
the patch either. Currently if you run "bitbake" and provide no other
options, the patch will just fetch everything. You wouldn't need to
Ctrl+C to make modifications. Now you can just jump off in to a build if
it is what you want to do. Actually anything that calls cooker basically
causes them to be fetched. Running bitbake-layers for example does the
fetch.

> I think the piece I am missing at the moment is why having it as an external 
> tool that largely replicates the same outward functionality as having it 
> within bitbake presents a problem for the use case that you have. Is there 
> some technical capability that we couldn't have using this approach - other 
> than the fact that you're just calling bitbake and it does everything? If 
> that's the only objection, would it not achieve the same thing if you had a 
> small shell script that ran the fetch/update then bitbake?

Pretty much everything is a small matter of code. It is a matter of
complexity. More or less when you boil it down, bitbakewith out content
is fantastic at a few basic tasks, one of them being fetching. Its a
kinda straight line to let bitbake be bitbake. I get it you don't want
it to be automatic. Do you have something I can look at that addresses
the remote layering? If your not saving the layers data in BBLAYERS
where are you saving it?

> 
> Fetching metadata and fetching source code are two different things. You don't 
> really expect to have to do anything to the source code before bitbake builds 
> it - bitbake and whatever metadata you have takes care of that. However with 
> the metadata you may wish to make modifications prior to building.

It really comes down to use case. Since we started out with a bitbake/oe
derived product, our collections have been always been stowed in tmp.
Basically keep the upstream pristine make changes with mechanisms like
amend.inc/.bbappend in more private layers. Now we are more or less
working with non-oe developers, and folks that are more or less complete
novices when it comes to this type of stuff.

I accept this is only one use case, however I actually think it is more
important then the standard oe-developer case. Most oe-developers could
probably take the layers wiki page listing all the disparate layers and
be good. Its the folks that are not as oe/bitbake sophicticated that
this helps.

Richard was commenting about moving checkin's from layer to layer. While
for me in my personal work, that is fantastic. However, I can already do
that with out the aid of a separate tool. Most people who use git in
there day to day work could as well, whether they have ever heard of
bitbake or oe or not. While that functionality is "neat", it shouldn't,
imho, be added to complicate basic functionality.

>> Basically what your implementing here has already been done, ergo in the
>> spirit of the purpose of the project, we were trying to avoid
>> reinventing the same tools again either here or in
>> yocto/oe-core/bitbake. The path you guys are tracking, is more or less
>> the same track we started on and we ended up with what we provided to you.
> 
> You have come up with an implementation that works well for you, and you've 
> published that implementation. Thankyou for that. However, there are some 
> aspects of that implementation that concern Richard and I so I'm trying to 
> come up with an alternative implementation (using some of the same code) that 
> meets the requirements. If we're not going to meet some important requirements 
> that you have then I'm more than happy to stop and re-evaluate the situation.

Well what we have more or less been able to accomplish with what we have
provided is the ability for a user to generate a project with nothing
locally stored but the uri's. The end user sources a script and runs
bitbake. The layers/collections are retrieved and it is off and running.
The content-tools more or less shuffle around uri's, source mirrors and
prebuilt mirrors. The project lacking more or less any metadata other
the configuration options can get dropped right in to scm, especially
when using remote uris.

Some of the above may not fit in the scope of what we are trying to do
here, which is why I have been pushing some of the lower functionality
that can be reused.

If it comes down to running an extra command prior to running bitbake
then will work. The one thing I would prefer is that the whole set of
layers(including oe-core) be able to be checked out, rather then
checking out oe-core first then checkout any additional layers. In that
same vain supporting archives as valid layer types, would be helpful.

> Understood. However, I'd hope we can soon move to a state where we only 
> have one set of fetchers. Is there anything preventing Montavista from using 
> fetch2 currently?

Backwards compatibility. Currently we are supporting legacy code, that
would require some rework to go with fetch2. For those products I could
halt the forward progression on bitbake, but using a single version of
bitbake would be better for us, though I completely understand the
removal of the older fetch. Its more or less my problem then anything
you need to deal with.

> Well, we definitely would need the "layer" fetcher code to be refactored out; 
> Richard made that comment earlier.

Well the code that lives there now would need to move in to __init__.py
or in to the remotealyers bit with if else bits for fetch/fetch2. OTOH,
it sounds like the decision is not to use the patch at all.

> I would also hope that as you suggested we 
> can move to having the initialisation within the fetchers themselves so that 
> the layer fetching isn't doing anything special in that regard. We need to 
> establish the direction we want to go in however.

Well yeah this is general goodness.


-- 
Jeremy Puhlman
Montavista Sofware, LLC.

Re: [PATCH] Add support for remote layering.

[Paul Eggleton]

On Wednesday 06 July 2011 00:38:41 Jeremy Puhlman wrote:
> Our projects start off with zero meta data other then a local.conf and
> bblayers.conf. In earlier days, we provided collections.inc and
> bitbake.conf in a local conf directory provided with bitbake. That is
> not done anymore(ergo why the fetchers are broken from go, previously
> nothing happened until at lease one bitbake.conf was loaded).

I am aiming for this use case to be supported with my utility. What I would 
also like to handle though is that if you have chosen to set up some 
configuration, then it will be able to be read in before the fetching starts.
 
> I get it you don't want it to be automatic. Do you have something I can look
> at that addresses the remote layering? 

I've thrown together a proof-of-concept "bitbake-fetchlayers"  in the 
"paule/remotelayers" contrib branch:

http://git.yoctoproject.org/cgit/cgit.cgi/poky-
contrib/log/?h=paule/remotelayers

This is by no means a finished utility, may have hideous bugs, etc. This 
requires nothing more than vanilla bitbake to operate. Some notes:

 * BBPATH needs to be set, LAYER_UNPACKDIR also.

 * Currently it requires conf/bitbake.conf, classes/ etc., which are shipped 
with bitbake master but are not present with the copy of bitbake that's in 
Poky; I hope to be able to address this in such a way that the utility does 
not require these.

 * "init" is the command used to fetch multiple layers. I've also provided 
"fetch" as a way to test a single fetch operation; I would expect the latter 
to be removed at some point. (Also, none of these command names are final.)

 * Update is not yet implemented. For your use case, update is no more than a 
re-fetch; however where you are intending to interact with the layers as SCM 
working directories it would be better to do an update in-place. I'm still 
thinking about how best to implement this.

 * Output is rather noisy, this needs to be fixed also.

 * I did have to patch the fetchers as Richard suggested so they have default 
values for the configurable fetch commands. (We'd have to have done this 
anyway.)

> If your not saving the layers data in BBLAYERS where are you saving it?

Well, this is not entirely clear to me yet. However if they are not in 
bblayers.conf and you wish to refer to them again (as in your use case you 
would for updating) they would naturally have to be in another file somewhere.

FWIW bitbake-fetchlayers' "init" command currently takes a list consisting of 
uri,localdir,subdir items, with the last two parameters being optional, and 
outputs a new BBLAYERS definition. This list could alternatively come from a 
file.

> It really comes down to use case. Since we started out with a bitbake/oe
> derived product, our collections have been always been stowed in tmp.
> Basically keep the upstream pristine make changes with mechanisms like
> amend.inc/.bbappend in more private layers. Now we are more or less
> working with non-oe developers, and folks that are more or less complete
> novices when it comes to this type of stuff.
>
> I accept this is only one use case, however I actually think it is more
> important then the standard oe-developer case. Most oe-developers could
> probably take the layers wiki page listing all the disparate layers and
> be good. Its the folks that are not as oe/bitbake sophicticated that
> this helps.

Ultimately anyone coming to bitbake/OE fresh is going to need to know 
something about setting it up and using it. However, any mechanical steps can 
be scripted away. For these kinds of users, running "bitbake <target>" or 
"startbuild.sh" is not going to be much different.
 
> Richard was commenting about moving checkin's from layer to layer. While
> for me in my personal work, that is fantastic. However, I can already do
> that with out the aid of a separate tool. Most people who use git in
> there day to day work could as well, whether they have ever heard of
> bitbake or oe or not. While that functionality is "neat", it shouldn't,
> imho, be added to complicate basic functionality.

Richard's aim is simply to keep the core of bitbake free of complexities that 
may be painful to extend if they are in the core.

> If it comes down to running an extra command prior to running bitbake
> then will work. The one thing I would prefer is that the whole set of
> layers(including oe-core) be able to be checked out, rather then
> checking out oe-core first then checkout any additional layers. In that
> same vain supporting archives as valid layer types, would be helpful.

I think we can support this without any issues - any of bitbake's fetchers 
should be able to be used, including wget and local. You won't need to grab 
oe-core first.

> Backwards compatibility. Currently we are supporting legacy code, that
> would require some rework to go with fetch2. For those products I could
> halt the forward progression on bitbake, but using a single version of
> bitbake would be better for us, though I completely understand the
> removal of the older fetch. Its more or less my problem then anything
> you need to deal with.

Hmm. For simplicity I've only supported fetch2 in bitbake-fetchlayers - in 
fact it forces it at startup. It would not be hard to support fetch also, 
however I hope we could avoid having to do so.

Cheers,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre

Re: [PATCH] Add support for remote layering.

[Jeremy Puhlman]

> I am aiming for this use case to be supported with my utility. What I would 
> also like to handle though is that if you have chosen to set up some 
> configuration, then it will be able to be read in before the fetching starts.

Sounds reasonable.

>> I get it you don't want it to be automatic. Do you have something I can look
>> at that addresses the remote layering? 
> 
> I've thrown together a proof-of-concept "bitbake-fetchlayers"  in the 
> "paule/remotelayers" contrib branch:
> 
> http://git.yoctoproject.org/cgit/cgit.cgi/poky-
> contrib/log/?h=paule/remotelayers
> 
> This is by no means a finished utility, may have hideous bugs, etc. This 
> requires nothing more than vanilla bitbake to operate. Some notes:
> 
>  * BBPATH needs to be set, LAYER_UNPACKDIR also.
> 
>  * Currently it requires conf/bitbake.conf, classes/ etc., which are shipped 
> with bitbake master but are not present with the copy of bitbake that's in 
> Poky; I hope to be able to address this in such a way that the utility does 
> not require these.
> 
>  * "init" is the command used to fetch multiple layers. I've also provided 
> "fetch" as a way to test a single fetch operation; I would expect the latter 
> to be removed at some point. (Also, none of these command names are final.)
> 
>  * Update is not yet implemented. For your use case, update is no more than a 
> re-fetch; however where you are intending to interact with the layers as SCM 
> working directories it would be better to do an update in-place. I'm still 
> thinking about how best to implement this.

I will try and check this out later this week.

>  * Output is rather noisy, this needs to be fixed also.

k.

>  * I did have to patch the fetchers as Richard suggested so they have default 
> values for the configurable fetch commands. (We'd have to have done this 
> anyway.)

Great.

> 
> I think we can support this without any issues - any of bitbake's fetchers 
> should be able to be used, including wget and local. You won't need to grab 
> oe-core first.

Okay.

> Hmm. For simplicity I've only supported fetch2 in bitbake-fetchlayers - in 
> fact it forces it at startup. It would not be hard to support fetch also, 
> however I hope we could avoid having to do so.

That is fine. The legacy code is likely not going to be moving over to
this anyways. It was more of a response to why did you do it, rather
then it needs to support it. Don't bother unless it is really needed by
someone else.


-- 
Jeremy Puhlman
Montavista Sofware, LLC.

Re: [PATCH] Add support for remote layering.

[Paul Eggleton]

On Monday 11 July 2011 18:46:26 Paul Eggleton wrote:
> I've thrown together a proof-of-concept "bitbake-fetchlayers"  in the
> "paule/remotelayers" contrib branch:
> 
> http://git.yoctoproject.org/cgit/cgit.cgi/poky-
> contrib/log/?h=paule/remotelayers

I spent some time looking at this again today, and the contrib branch has now 
been updated. Changes are as follows:

 * bitbake.conf and base.bbclass are no longer needed, so you really only need 
bitbake now. Some fixes to some of the fetchers other than git might still be 
necessary however.

 * Only BBPATH now needs to be set; LAYER_UNPACKDIR now defaults to 
${TOPDIR}/layers.

Cheers,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre