* [PATCH] Add support for remote layering. [not found] <RFC: Layer tooling brainstorming> @ 2011-04-28 18:09 ` Jeremy Puhlman 2011-04-28 18:09 ` Jeremy Puhlman 1 sibling, 0 replies; 20+ messages in thread From: Jeremy Puhlman @ 2011-04-28 18:09 UTC (permalink / raw) To: openembedded-core; +Cc: chris_larson > * Maybe need to split into "bootstrap" steps (e.g where pseduo is established, layers downloaded etc) So for the most part, the last product we shipped was largely arranged in this manner. We were using collections, but in many cases layers is just a reimplementation at a lower level(which is definitely goodness). During ELC I had a number of discussions primarily around remote layer fetch and most of the responses were meh, or that they wanted to make the tool external to bitbake. So after the conference was over I went ahead and worked out a version of the implementation that could be done either in a include or in the bitbake code itself. The version in the git repo below is obviously just the bitbake implementation. I have a rough implementation largely based off of collection.inc + some enhancements we made internally for remote collection fetching. Some of it is messy(setting up the early fetching vars with out using another config), but for the ones I list in the patch header it appears to work well. I do not believe that this is complete, but it is probably a reasonable starting point. Pull URL: git://gitorious.org/jpwork/bb.git Branch: remote-layering Browse: https://gitorious.org/jpwork/bb/trees/remote-layering ^ permalink raw reply [flat|nested] 20+ messages in thread
* [PATCH] Add support for remote layering. [not found] <RFC: Layer tooling brainstorming> 2011-04-28 18:09 ` [PATCH] Add support for remote layering Jeremy Puhlman @ 2011-04-28 18:09 ` Jeremy Puhlman 2011-04-28 18:20 ` [PATCH 0/1] " Jeremy Puhlman 2011-05-06 13:15 ` [PATCH] " Richard Purdie 1 sibling, 2 replies; 20+ messages in thread From: Jeremy Puhlman @ 2011-04-28 18:09 UTC (permalink / raw) To: openembedded-core; +Cc: chris_larson The bulk of this patch is based on Chris Larson's collection.inc, but refactored for use inside bitbake, and for layers. Adds two new configuration options: LAYER_UNPACKDIR - directory to unpack downloaded layers for use in configuration. LAYER_REMOTE_STASH - location to store downloaded layers for use in configuration. Layers can be specified in the current manner: BBLAYERS = " \ /path/to/layer/one \ /path/to/layer/two \ " Or you can specify uris: BBLAYERS = " \ http://url.to.my.tarball/tarball.tar.gz \ git://gitserver.com/git/repo;proto=http \ /path/to/layer/three \ file:///path/to/layer/four \ " Currently there is a single layer option, that can be added to a uri, layerBase=<subpath>. This option would be used to specify if a layer starts somewhere other then the base of the tarball/scm repository. For example if you wanted to add oe-core you would do: BBLAYERS = "git://git.openembedded.org/openembedded-core;protocol=git;tag=master;layerBase=meta" Currently for fetch the following uris should work : 'git','http','https','ftp','file' Currently for fetch2 the following uris should work : 'git','http','https','ftp','file','svn' There are certainly some ugly bits in here, and there are certainly lots of places to improve, but this gets a good chunk of the base support in. Signed-off-by: Jeremy Puhlman <jpuhlman@mvista.com> --- lib/bb/cooker.py | 3 +- lib/bb/fetch/layer.py | 65 ++++++++++++++++ lib/bb/fetch2/layer.py | 76 ++++++++++++++++++ lib/bb/remotelayer.py | 197 ++++++++++++++++++++++++++++++++++++++++++++++++ 4 files changed, 340 insertions(+), 1 deletions(-) create mode 100644 lib/bb/fetch/layer.py create mode 100644 lib/bb/fetch2/layer.py create mode 100644 lib/bb/remotelayer.py diff --git a/lib/bb/cooker.py b/lib/bb/cooker.py index d4415d3..0099a65 100644 --- a/lib/bb/cooker.py +++ b/lib/bb/cooker.py @@ -575,6 +575,7 @@ class BBCooker: path, _ = os.path.split(path) def parseConfigurationFiles(self, files): + from bb.remotelayer import RemoteLayers data = self.configuration.data bb.parse.init_parser(data) for f in files: @@ -586,7 +587,7 @@ class BBCooker: data = _parse(layerconf, data) layers = (bb.data.getVar('BBLAYERS', data, True) or "").split() - + layers = RemoteLayers(layers,data,parselog).getLayers() data = bb.data.createCopy(data) for layer in layers: parselog.debug(2, "Adding layer %s", layer) diff --git a/lib/bb/fetch/layer.py b/lib/bb/fetch/layer.py new file mode 100644 index 0000000..3b0d319 --- /dev/null +++ b/lib/bb/fetch/layer.py @@ -0,0 +1,65 @@ +""" +BitBake 'layer' fetch implementation. + +""" + +# Copyright (C) 2011 Jeremy Puhlman +# +# Classes for obtaining upstream sources for the +# BitBake build tools. +# Copyright (C) 2003, 2004 Chris Larson +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License version 2 as +# published by the Free Software Foundation. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + +class LayerFetch: + + def __init__(self, url, pathurl, logger, ld): + self.parselog = logger + self.tarballpath = self._fetch(url,pathurl,ld) + + def supported(self, ud): + return ud.type in ['git','http','https','ftp','file'] + + def getTarballPath(self): + return self.tarballpath + + def _fetch(self, url, pathurl, ld): + urldata = {} + try: + urldata[url] = bb.fetch.FetchData(url,ld) + except bb.fetch.NoMethodError: + self.parselog.debug(1, "No method for %s" % url) + return + ud = urldata[url] + filename=os.path.basename(ud.path) + + if not self.supported(ud): + bb.fatal("Layer fetching does not support %s uris" % ud.type) + if not ud.setup: + ud.setup_localpath(ld) + m = ud.method + tarballpath = bb.fetch.localpath(url, ld) + if os.path.exists(tarballpath) : + return tarballpath + try: + m.go(url,ud,ld) + except (bb.fetch.MissingParameterError, + bb.fetch.FetchError, + bb.fetch.MD5SumError): + + import sys + (type, value, traceback) = sys.exc_info() + self.parselog.debug(1, "layer fetch failure: %s" % value) + return + return tarballpath diff --git a/lib/bb/fetch2/layer.py b/lib/bb/fetch2/layer.py new file mode 100644 index 0000000..09de8a0 --- /dev/null +++ b/lib/bb/fetch2/layer.py @@ -0,0 +1,76 @@ +""" +BitBake 'layer' fetch2 implementation. + +""" + +# Copyright (C) 2011 Jeremy Puhlman +# +# Classes for obtaining upstream sources for the +# BitBake build tools. +# Copyright (C) 2003, 2004 Chris Larson +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License version 2 as +# published by the Free Software Foundation. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + +class LayerFetch: + + def __init__(self, url, pathurl, logger, ld): + self.parselog = logger + self.tarballpath = self._fetch(url,pathurl,ld) + + def supported(self, ud): + return ud.type in ['git','http','https','ftp', 'svn', 'file'] + + def getTarballPath(self): + return self.tarballpath + + def _fetch(self, url, pathurl, ld): + urldata = {} + layer_unpackdir = ld.getVar("LAYER_UNPACKDIR", True) + try: + urldata[url] = bb.fetch.FetchData(url,ld) + except bb.fetch.NoMethodError: + self.parselog.debug(1, "No method for %s" % url) + return + ud = urldata[url] + filename=os.path.basename(ud.path) + + if not self.supported(ud): + bb.fatal("Layer fetching does not support %s uris" % ud.type) + if not ud.setup: + ud.setup_localpath(ld) + m = ud.method + if ud.type not in ['git']: + tarballpath = bb.fetch.localpath(url, ld) + if os.path.exists(bb.fetch.localpath(url, ld)) : + return tarballpath + try: + m.download(url,ud,ld) + except (bb.fetch.MissingParameterError, + bb.fetch.FetchError, + bb.fetch.MD5SumError): + + import sys + (type, value, traceback) = sys.exc_info() + self.parselog.debug(1, "layer fetch failure: %s" % value) + return + + if ud.type in ['git']: + unpackedscm = "%s/%s" % (layer_unpackdir, ud.mirrortarball) + if m.need_update(url,ud,ld): + m.build_mirror_data(url, ud,ld) + m.unpack(ud, unpackedscm ,ld) + tarballpath = unpackedscm + "/git" + + return tarballpath + diff --git a/lib/bb/remotelayer.py b/lib/bb/remotelayer.py new file mode 100644 index 0000000..2a61e71 --- /dev/null +++ b/lib/bb/remotelayer.py @@ -0,0 +1,197 @@ +""" +BitBake 'remote layer' handling. + +""" + +# Copyright (C) 2011 Jeremy Puhlman +# +# Classes for obtaining upstream sources for the +# BitBake build tools. +# Copyright (C) 2003, 2004 Chris Larson +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License version 2 as +# published by the Free Software Foundation. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + +class RemoteLayers: + """ + Get remote layers. + """ + + def __init__(self, layers, data, logger): + self.parselog = logger + self.layers = self._getRemoteLayers(layers, data) + + def getLayers(self): + return self.layers + + """ + Ugly. We have not loaded bitbake.conf yet, and need the values set for fetching to work + Could load preliminary bitbake.conf, but this works. + """ + def _setFetchValues(self,data): + """Need these for any fetching to work""" + def setifunset(d,k,v): + if d.getVar(k,0) is None: + d.setVar(k,v) + + setifunset(data, "TMPDIR", "${TOPDIR}/tmp") + setifunset(data, "LAYER_REMOTE_STASH", "${TMPDIR}/layers-remote") + setifunset(data, "LAYER_UNPACKDIR", "${TMPDIR}/layers") + + ld = data.createCopy() + """Provide settings for fetchers.""" + setifunset(ld, "CACHE", "${TMPDIR}/cache") + setifunset(ld, "OVERRIDES", "") + setifunset(ld, "CVSDIR", "${DL_DIR}/cvs") + setifunset(ld, "GITDIR", "${DL_DIR}/git") + setifunset(ld, "SVNDIR", "${DL_DIR}/svn") + setifunset(ld, "FETCHCOMMAND", "") + setifunset(ld, "FETCHCOMMAND_cvs", "/usr/bin/env cvs -d${CVSROOT} co ${CVSCOOPTS} ${CVSMODULE}") + setifunset(ld, "FETCHCOMMAND_svn", "/usr/bin/env svn co ${SVNCOOPTS} ${SVNROOT} ${SVNMODULE}") + setifunset(ld, "FETCHCOMMAND_wget", "/usr/bin/env wget -t 5 -q --passive-ftp -P ${DL_DIR} ${URI}") + setifunset(ld, "FETCHCMD_cvs", "/usr/bin/env cvs ") + setifunset(ld, "FETCHCMD_svn", "/usr/bin/env svn ") + setifunset(ld, "FETCHCMD_bzr", "/usr/bin/env bzr ") + setifunset(ld, "FETCHCMD_hg", "/usr/bin/env hg ") + setifunset(ld, "FETCHCMD_wget", "/usr/bin/env wget -t 5 -q") + setifunset(ld, "CHECKCOMMAND_wget", "/usr/bin/env wget --spider -t 5 --passive-ftp -P ${DL_DIR} '${URI}'") + setifunset(ld, "UPDATECOMMAND", "") + setifunset(ld, "UPDATECOMMAND_cvs", "/usr/bin/env cvs -d${CVSROOT} update ${CVSCOOPTS}") + setifunset(ld, "UPDATECOMMAND_svn", "/usr/bin/env svn update ${SVNCOOPTS}") + setifunset(ld, "BB_GENERATE_MIRROR_TARBALLS", "1") + + localdir=data.getVar("LAYER_REMOTE_STASH", 1) + + if not os.path.isdir(localdir): + bb.utils.mkdirhier(localdir) + ld.setVar("DL_DIR", localdir) + ld.delVar("MIRRORS") + ld.delVar("PREMIRRORS") + ld.delVar("SRC_TARBALL_STASH") + return ld + + """ + Unpack remote layer. Code largely pulled from collections.inc + """ + def _layerUnpack(self, layer, data): + """ Unpack a layer archive and return the path to it. """ + from hashlib import md5 + + handlers = { + ("tar"): "tar x --no-same-owner -f %s", + ("tar.gz", "tgz", "tar.Z"): "tar xz --no-same-owner -f %s", + ("tar.bz2", "tbz", "tbz2"): "tar xj --no-same-owner -f %s", + ("zip", "jar"): "unzip -q -o %s", + } + basename = os.path.basename(layer) + try: + cmd, name = ((cmd, basename[:-len(e)-1]) for (exts, cmd) in handlers.iteritems() + for e in exts + if basename.endswith(e)).next() + except StopIteration: + bb.fatal("No method available to unpack %s (unsupported file type?)" % layer) + else: + outpath = os.path.join(data.getVar("LAYER_UNPACKDIR", 1), name) + cmd = "cd %s && PATH=\"%s\" %s" % (outpath, data.getVar("PATH", 1), cmd) + + try: + layerdata = open(layer, "r").read() + except IOError: + bb.fatal("Unable to open %s to calculate md5 sum" % layer) + + md5obj = md5() + md5obj.update(layerdata) + md5sum = md5obj.hexdigest() + md5file = os.path.join(outpath, "md5") + if os.path.exists(md5file): + try: + oldmd5sum = open(md5file).read() + except IOError: + pass + else: + if oldmd5sum == md5sum: + self.parselog.debug(1, "Using existing %s for layer '%s'" % (outpath, name)) + return outpath, name + + self.parselog.plain("Removing old unpacked layer at %s" % outpath) + + if not os.path.isdir(outpath): + os.makedirs(outpath) + + self.parselog.debug(1,"Unpacking %s to %s/" % (layer, outpath)) + ret = os.system(cmd % layer) + if ret != 0: + bb.fatal("Unable to unpack %s" % layer) + md5out = open(md5file, "w") + md5out.write(md5sum) + md5out.close() + return outpath, name + + """ + Add method for grab layer specific data from uri. + """ + def _getLayerSettings(self, urlinfo): + layersettings={} + layersettingslist = ["layerBase"] + for setting in urlinfo.path.split(";") + urlinfo.params.split(";"): + for layersetting in layersettingslist: + if setting.startswith("%s=" % layersetting): + layersettings[layersetting] = setting.split("=")[1] + return [layersettings] + + """ + Main hooks for grabing remote layers + """ + def _getRemoteLayers(self, layers, data): + from itertools import izip, chain + from glob import glob + from urlparse import urlparse, urlunparse + from bb.fetch.layer import LayerFetch + if not layers: + return [] + + globbed = [] + layersettings = [] + localpath = "" + ld = self._setFetchValues(data) + for path in layers: + pathurl = urlparse(path) + layersettings += self._getLayerSettings(pathurl) + if not pathurl[0]: + localpath = glob(os.path.normpath(path)) + else: + tarballpath = LayerFetch(path,pathurl,self.parselog, ld).getTarballPath() + if tarballpath: + localpath=[tarballpath] + if not localpath: + self.parselog.plain("No matches in filesystem for %s in BBLAYERS" % path) + globbed += localpath + layers = globbed + layers_unpacked = "" + for (layer, settings) in izip(layers, layersettings): + origpath = layer + basepath="" + if "layerBase" in settings: + basepath = settings["layerBase"] + if not os.path.isdir(layer): + unpacked, name = self._layerUnpack(layer, ld) + if unpacked: + layer = unpacked + for dir in glob("%s/*/" % layer): + layerdir = dir + "/" + basepath + layers_unpacked += layerdir + " " + else: + layers_unpacked += layer + "/" + basepath + " " + return layers_unpacked.encode('ascii').split() + + -- 1.7.3.2 ^ permalink raw reply related [flat|nested] 20+ messages in thread
* [PATCH 0/1] Add support for remote layering. 2011-04-28 18:09 ` Jeremy Puhlman @ 2011-04-28 18:20 ` Jeremy Puhlman 2011-05-06 13:15 ` [PATCH] " Richard Purdie 1 sibling, 0 replies; 20+ messages in thread From: Jeremy Puhlman @ 2011-04-28 18:20 UTC (permalink / raw) To: Jeremy Puhlman; +Cc: chris_larson, openembedded-core Missed prefix mail. Resent due to limited response. > * Maybe need to split into "bootstrap" steps (e.g where pseduo is > established, layers downloaded etc) So for the most part, the last product we shipped was largely arranged in this manner. We were using collections, but in many cases layers is just a reimplementation at a lower level(which is definitely goodness). During ELC I had a number of discussions primarily around remote layer fetch and most of the responses were meh, or that they wanted to make the tool external to bitbake. So after the conference was over I went ahead and worked out a version of the implementation that could be done either in a include or in the bitbake code itself. The version in the git repo below is obviously just the bitbake implementation. I have a rough implementation largely based off of collection.inc + some enhancements we made internally for remote collection fetching. Some of it is messy(setting up the early fetching vars with out using another config), but for the ones I list in the patch header it appears to work well. I do not believe that this is complete, but it is probably a reasonable starting point. Pull URL: git://gitorious.org/jpwork/bb.git Branch: remote-layering Browse: https://gitorious.org/jpwork/bb/trees/remote-layering --- Add support for remote layering. The bulk of this patch is based on Chris Larson's collection.inc, but refactored for use inside bitbake, and for layers. Adds two new configuration options: LAYER_UNPACKDIR - directory to unpack downloaded layers for use in configuration. LAYER_REMOTE_STASH - location to store downloaded layers for use in configuration. Layers can be specified in the current manner: BBLAYERS = " \ /path/to/layer/one \ /path/to/layer/two \ " Or you can specify uris: BBLAYERS = " \ http://url.to.my.tarball/tarball.tar.gz \ git://gitserver.com/git/repo;proto=http \ /path/to/layer/three \ file:///path/to/layer/four \ " Currently there is a single layer option, that can be added to a uri, layerBase=<subpath>. This option would be used to specify if a layer starts somewhere other then the base of the tarball/scm repository. For example if you wanted to add oe-core you would do: BBLAYERS = "git://git.openembedded.org/openembedded-core;protocol=git;tag=master;layerBase=meta" Currently for fetch the following uris should work : 'git','http','https','ftp','file' Currently for fetch2 the following uris should work : 'git','http','https','ftp','file','svn' There are certainly some ugly bits in here, and there are certainly lots of places to improve, but this gets a good chunk of the base support in. Signed-off-by: Jeremy Puhlman <jpuhlman at mvista.com> ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [PATCH] Add support for remote layering. 2011-04-28 18:09 ` Jeremy Puhlman 2011-04-28 18:20 ` [PATCH 0/1] " Jeremy Puhlman @ 2011-05-06 13:15 ` Richard Purdie 2011-05-12 13:11 ` Jeremy Puhlman 1 sibling, 1 reply; 20+ messages in thread From: Richard Purdie @ 2011-05-06 13:15 UTC (permalink / raw) To: Patches and discussions about the oe-core layer; +Cc: chris_larson Hi Jeremy, Sorry for the delayed response, I took the opportunity to take some vacation for a change and its taken me a few days to catch up on email. Thanks for posting this. Whilst some of what I mention below may sound negative, its not meant this way and what you have gives a nice basis to start some discussion from on this topic. I do want to take a step back and ensure we integrate this functionality at the right level within the stack. I've made comments below. On Thu, 2011-04-28 at 11:09 -0700, Jeremy Puhlman wrote: > The bulk of this patch is based on Chris Larson's collection.inc, > but refactored for use inside bitbake, and for layers. > > Adds two new configuration options: > > LAYER_UNPACKDIR - directory to unpack downloaded layers for use in configuration. > LAYER_REMOTE_STASH - location to store downloaded layers for use in configuration. > > Layers can be specified in the current manner: > BBLAYERS = " \ > /path/to/layer/one \ > /path/to/layer/two \ > " > > Or you can specify uris: > > BBLAYERS = " \ > http://url.to.my.tarball/tarball.tar.gz \ > git://gitserver.com/git/repo;proto=http \ > /path/to/layer/three \ > file:///path/to/layer/four \ > " This is good but what I don't really like about this is that whilst you can specify where to get the layer from, it isn't clear where the layer ends up locally. Compare the above to something like: BBLAYERS = " \ /path/to/layer/one;srcuri=http://url.to.my.tarball/tarball.tar.gz \ /path/to/layer/two;srcuri=git://gitserver.com/git/repo;proto=http \ /path/to/layer/three \ /path/to/layer/four \ " This is more ugly but it does clearly set an expectation for both where its coming from and where it ends up on disk. It also means we can then set specific revisions to checkout or other information e.g. whether the revisions should auto-increment. The syntax would probably come naturally to anyone who has used SRC_URI (and hence it also flows nicely into the fetcher code). > Currently there is a single layer option, that can be added to a uri, layerBase=<subpath>. > This option would be used to specify if a layer starts somewhere other then the base of the > tarball/scm repository. For example if you wanted to add oe-core you would do: Ideally this could become an option that any of our fetchers would support... > lib/bb/cooker.py | 3 +- > lib/bb/fetch/layer.py | 65 ++++++++++++++++ > lib/bb/fetch2/layer.py | 76 ++++++++++++++++++ > lib/bb/remotelayer.py | 197 ++++++++++++++++++++++++++++++++++++++++++++++++ I'm not going to dive into the patches at this point but "layer" isn't really a fetcher as such as it doesn't correspond to a specific source type, its just a wrapper around other fetch methods. I'm therefore a little worried this confuses the abstraction we currently have there. Going back to the high level approach discussion, I'm also wondering if there should be some "bblayers" tool which gets run prior to bitbake (maybe automatically in some cases) which handles the parsing of the BBLAYERS options and does whatever is needed to handle the setup there, then hands off to bitbake itself. I'm very keen to get the abstraction level of this code right so I'd be interested in your (and others) thoughts on this... Cheers, Richard ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [PATCH] Add support for remote layering. 2011-05-06 13:15 ` [PATCH] " Richard Purdie @ 2011-05-12 13:11 ` Jeremy Puhlman 2011-05-12 17:34 ` Jeremy Puhlman 0 siblings, 1 reply; 20+ messages in thread From: Jeremy Puhlman @ 2011-05-12 13:11 UTC (permalink / raw) To: Patches and discussions about the oe-core layer Cc: paul.eggleton@linux.intel.com >> Paul Eggleton, chris_larson On 5/6/2011 6:15 AM, Richard Purdie wrote: > Hi Jeremy, > > Sorry for the delayed response, I took the opportunity to take some > vacation for a change and its taken me a few days to catch up on email. No worries. Everyone needs some time off. :) > Thanks for posting this. Whilst some of what I mention below may sound > negative, its not meant this way and what you have gives a nice basis to > start some discussion from on this topic. I do want to take a step back > and ensure we integrate this functionality at the right level within the > stack. I've made comments below. Understood. >> Or you can specify uris: >> >> BBLAYERS = " \ >> http://url.to.my.tarball/tarball.tar.gz \ >> git://gitserver.com/git/repo;proto=http \ >> /path/to/layer/three \ >> file:///path/to/layer/four \ >> " > > This is good but what I don't really like about this is that whilst you > can specify where to get the layer from, it isn't clear where the layer > ends up locally. Compare the above to something like: > > BBLAYERS = " \ > /path/to/layer/one;srcuri=http://url.to.my.tarball/tarball.tar.gz \ > /path/to/layer/two;srcuri=git://gitserver.com/git/repo;proto=http \ > /path/to/layer/three \ > /path/to/layer/four \ > " So currently it gets stashed in the locations specified above. Currently the code has the ability to extract values out of the uri as new bits. The above would be reasonable, but you could do as well: BBLAYERS = " \ > http://url.to.my.tarball/tarball.tar.gz;local=/path/to/layer \ > git://gitserver.com/git/repo;proto=httplocalLayer=/path/to/layer/two \ > /path/to/layer/three \ > /path/to/layer/four \ With the current implementation the above is fairly straight forward. (not that I am implying implementation should necessarily dictate the what is done, just noting the above is significantly easier to implement from the point the code is now.) > > This is more ugly but it does clearly set an expectation for both where > its coming from and where it ends up on disk. It also means we can then > set specific revisions to checkout or other information e.g. whether the > revisions should auto-increment. The syntax would probably come > naturally to anyone who has used SRC_URI (and hence it also flows nicely > into the fetcher code). Actually the code already passes what ever the uri off to the base fetchers already. So in reality what ever the global and uri setting is for the fetcher, it should work here as well. > I'm not going to dive into the patches at this point but "layer" isn't > really a fetcher as such as it doesn't correspond to a specific source > type, its just a wrapper around other fetch methods. I'm therefore a > little worried this confuses the abstraction we currently have there. This I can definately agree with. The primary issue was I actually was trying to make the code work with both fetcher implementations largely because I still use both, and it was either use two separated by ifs on fetcher implementations, add to __init__ or just drop fetch support at all. Using the separate file was useful for demonstration purposes. The actual code placed there is fairly sparce since it just dealt with the difference in fetchers, and most if not all common code was handled in the remote layers file above. > > Going back to the high level approach discussion, I'm also wondering if > there should be some "bblayers" tool which gets run prior to bitbake > (maybe automatically in some cases) which handles the parsing of the > BBLAYERS options and does whatever is needed to handle the setup there, > then hands off to bitbake itself. So we have been using a tool to generate bblayers via a whole higher level mirroring setup, and then let bitbake handle all the grunt work doing the fetching. This works in our case. FWIW I have been trying to push the company to open those, and we are still in discussions. > > I'm very keen to get the abstraction level of this code right so I'd be > interested in your (and others) thoughts on this... Yeah which is kind of why I reposted. The response was limited. ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [PATCH] Add support for remote layering. 2011-05-12 13:11 ` Jeremy Puhlman @ 2011-05-12 17:34 ` Jeremy Puhlman 2011-05-20 16:45 ` Paul Eggleton 0 siblings, 1 reply; 20+ messages in thread From: Jeremy Puhlman @ 2011-05-12 17:34 UTC (permalink / raw) To: Patches and discussions about the oe-core layer Cc: paul.eggleton@linux.intel.com >> Paul Eggleton, chris_larson > So currently it gets stashed in the locations specified above. Currently > the code has the ability to extract values out of the uri as new bits. > The above would be reasonable, but you could do as well: > > BBLAYERS = " \ >> http://url.to.my.tarball/tarball.tar.gz;local=/path/to/layer \ >> git://gitserver.com/git/repo;proto=httplocalLayer=/path/to/layer/two \ >> /path/to/layer/three \ >> /path/to/layer/four \ > > With the current implementation the above is fairly straight forward. > (not that I am implying implementation should necessarily dictate the > what is done, just noting the above is significantly easier to implement > from the point the code is now.) Arg. Too early this morning. That should have been something like: > BBLAYERS = " \ >> http://url.to.my.tarball/tarball.tar.gz;localLayer=/path/to/layer \ >> git://gitserver.com/git/repo;proto=http;localLayer=/path/to/layer/two \ >> /path/to/layer/three \ >> /path/to/layer/four \ > or something to that effect. ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [PATCH] Add support for remote layering. 2011-05-12 17:34 ` Jeremy Puhlman @ 2011-05-20 16:45 ` Paul Eggleton 2011-05-20 17:42 ` Jeremy Puhlman 0 siblings, 1 reply; 20+ messages in thread From: Paul Eggleton @ 2011-05-20 16:45 UTC (permalink / raw) To: openembedded-core; +Cc: chris_larson Returning a discussion that has begun on the wiki to the mailing list: Jeremy Puhlman wrote: > Paul Eggleton wrote: >> A good start but naturally we want to avoid the bits that hard-code >> variable values. I wonder about situations where people want their own >> versions of layers or to share remote layers (e.g. between an oe-core setup >> and a poky setup); however people can just use local layers for this. > > In reality they are setting "reasonable" defaults. Hardcoding implies > something that is unchangeable. All the values are "set if unset", so they > can be changed in the bblayers.conf file. Prior to the introduction of > layers, all the values were hardcoded, just in a bitbake.conf file, so you > always had working fetchers. With out that the fetchers are basically broken > until after you have loaded up the layers. Given the fact that nearly every > bitbake.conf file contains the same settings for defining fetchcmd and a like, > simply having a reasonable default setting for the things the fetchers need > is not really that terrible an idea irrespective of the remote layering. I think it's just about the placing of these defaults, I don't necessarily object to having them in the first place. My main concerns are around the defaults for variables like TMPDIR which may end up actually being used in preference to what the user has set in local.conf. Obviously we have a chicken-and-egg situation where we depend on having the main layer present to read any configuration other than bblayers.conf - perhaps this indicates we need to restructure some of the config files if we want to make this work nicely. > As for the second question. I have some content tools that I am cleaning up > that basically provide methods for defining layers and groups of layers > together for mirroring and sharing. That might address what your talking > about there. They work along with the patch posted here, but could be > modified to output things a bit differently if needed. I'd definitely be interested in seeing these tools and trying to incorporate them into what we're doing for layer tooling in 1.1. Is there a lot of work left to do before you can release these? Cheers, Paul -- Paul Eggleton Intel Open Source Technology Centre ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [PATCH] Add support for remote layering. 2011-05-20 16:45 ` Paul Eggleton @ 2011-05-20 17:42 ` Jeremy Puhlman 2011-07-01 13:24 ` Paul Eggleton 0 siblings, 1 reply; 20+ messages in thread From: Jeremy Puhlman @ 2011-05-20 17:42 UTC (permalink / raw) To: Paul Eggleton; +Cc: chris_larson, openembedded-core > I think it's just about the placing of these defaults, I don't necessarily > object to having them in the first place. My main concerns are around the > defaults for variables like TMPDIR which may end up actually being used in > preference to what the user has set in local.conf. Obviously we have a > chicken-and-egg situation where we depend on having the main layer present to > read any configuration other than bblayers.conf - perhaps this indicates we > need to restructure some of the config files if we want to make this work > nicely. So with regard to the code specifically: TMPDIR really is the only major offender there as far as I can tell, all the other variables are set in the local copy of the data and disappar after the fetch. That variable setting could get moved over to the local copy as well, though we would need to change the way LAYER_REMOTE_STASH and LAYER_UNPACKDIR are set to make sure they are correct for the rest of the parse but that is not a big deal. Or we could set those via something relative other then TMPDIR, TOPDIR perhaps. General case: In reality looking at the bulk of the settings there, there are a couple different groups there. There is the stuff to make the fetchers work. I think those should probably make their way in to the the code in general. Setifunset means they can be changed early(bblayers), or overridden with the first bitbake.conf and in general they are rarely used in any user configurations(local.conf) or at least shouldn't. Having working fetchers from the first parsed line of configuration data seems like a generally good idea, but that is just me. There is the stuff only related to the fiddling with remote layers, which I talked above, can be changed to use any location in general. > > I'd definitely be interested in seeing these tools and trying to incorporate > them into what we're doing for layer tooling in 1.1. Is there a lot of work > left to do before you can release these? Work this week has been a bit slammed. Hopefully I will be able to post something this weekend, maybe Monday. ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [PATCH] Add support for remote layering. 2011-05-20 17:42 ` Jeremy Puhlman @ 2011-07-01 13:24 ` Paul Eggleton 2011-07-01 17:17 ` Jeremy Puhlman 2011-07-01 18:43 ` Jeremy Puhlman 0 siblings, 2 replies; 20+ messages in thread From: Paul Eggleton @ 2011-07-01 13:24 UTC (permalink / raw) To: Jeremy Puhlman; +Cc: openembedded-core Hi Jeremy Have looked at this further and I really think this needs to be a separate utility, but one still written in python using bitbake's infrastructure (like bitbake-layers). The advantages of this approach: 1) Avoids reworking bitbake's initialisation and avoids possible re-execution. A separate utility can take advantage of a full parse and utilise all of the user's configuration. 2) Fetching/updating layers should be (able to be) a conscious action rather than something that happens implicitly as part of the build process I'm trying to put something together that does this at the moment, with a view to there being components that allow integration with the content tools you posted earlier. Will keep you posted. Cheers, Paul -- Paul Eggleton Intel Open Source Technology Centre ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [PATCH] Add support for remote layering. 2011-07-01 13:24 ` Paul Eggleton @ 2011-07-01 17:17 ` Jeremy Puhlman 2011-07-01 18:43 ` Jeremy Puhlman 1 sibling, 0 replies; 20+ messages in thread From: Jeremy Puhlman @ 2011-07-01 17:17 UTC (permalink / raw) To: Paul Eggleton; +Cc: openembedded-core On 7/1/2011 6:24 AM, Paul Eggleton wrote: > Hi Jeremy > > Have looked at this further and I really think this needs to be a separate > utility, but one still written in python using bitbake's infrastructure (like > bitbake-layers). The advantages of this approach: > > 1) Avoids reworking bitbake's initialisation and avoids possible re-execution. Neither of those are issues in the current patch. Setting up the fetchers so they work, inside a local copy of the data is hardly reworking the initialization. > A separate utility can take advantage of a full parse and utilise all of the > user's configuration. This is certainly the case, however in the time that we have used this type of structure we have never needed all of the config data because you are always more less very early in the process. More or less you could create a bitbake-layers style "utilty" with the current patch. In the current patch, there is only a single line of code in the cooker, everything else is competely isolated. > 2) Fetching/updating layers should be (able to be) a conscious action rather > than something that happens implicitly as part of the build process This seems rather arbitrary. As noted this is not some theoretical process that I am proposing. We have tons of customers already use, and have used this type of model for a very long time. While I can certainly see the advantages of an external tool, why is it an either or? the reality is with the current patch, we already have bitbake-layers style utility. Its called bitbake. :) Run it with no argument and the first thing it does is fetch everything. > I'm trying to put something together that does this at the moment, with a view > to there being components that allow integration with the content tools you > posted earlier. Will keep you posted. For what it is worth, I would still like to see something like the patch get integrated. The single line in the cooker could be surrounnded by a BB_ var, and you more or less wouldn't know the difference. Also if BBLAYERS is just directories like it is now, every line is more or less ignored and passed back untouched. -- Jeremy Puhlman Montavista Sofware, LLC. ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [PATCH] Add support for remote layering. 2011-07-01 13:24 ` Paul Eggleton 2011-07-01 17:17 ` Jeremy Puhlman @ 2011-07-01 18:43 ` Jeremy Puhlman 2011-07-01 21:37 ` Richard Purdie 1 sibling, 1 reply; 20+ messages in thread From: Jeremy Puhlman @ 2011-07-01 18:43 UTC (permalink / raw) To: Paul Eggleton; +Cc: openembedded-core Paul, > > 2) Fetching/updating layers should be (able to be) a conscious action rather > than something that happens implicitly as part of the build process > Just a quick follow up, the part of this statement that to me is wrong, is it appears you are dictating work flow. While, imho, the layer management code which is above this level, is the place to dicate workflow. The patch is to core bitbake and isolated at that. This patch adds an additional work flow, and can be used in either state(the results of a current BBLAYERS setting is exactly the same today as it as after the patch). More or less the patch is useful in a scope larger then just layer-tooling. -- Jeremy Puhlman Montavista Sofware, LLC. ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [PATCH] Add support for remote layering. 2011-07-01 18:43 ` Jeremy Puhlman @ 2011-07-01 21:37 ` Richard Purdie 2011-07-02 0:33 ` Jeremy Puhlman 0 siblings, 1 reply; 20+ messages in thread From: Richard Purdie @ 2011-07-01 21:37 UTC (permalink / raw) To: Patches and discussions about the oe-core layer; +Cc: Paul Eggleton On Fri, 2011-07-01 at 11:43 -0700, Jeremy Puhlman wrote: > Just a quick follow up, the part of this statement that to me is wrong, > is it appears you are dictating work flow. While, imho, the layer > management code which is above this level, is the place to dicate > workflow. The patch is to core bitbake and isolated at that. This patch > adds an additional work flow, and can be used in either state(the > results of a current BBLAYERS setting is exactly the same today as it as > after the patch). More or less the patch is useful in a scope larger > then just layer-tooling. Let me try and convey at least one of the things I'm worrying about with this patch. We're looking at establishing layer tooling which operate outside of or "above" bitbake. There are a variety of reasons for that, rightly or wrongly but this patch implies we're going to reverse that. If we go forward with the patch and don't reverse that decision it means that layers can sometimes be a thing that are "above" bitbake, sometimes not depending on the circumstances. I'm trying hard to avoid that kind of interwoven complexity as it makes code changes very hard to make in the future and leads to frequent breakage where multiple usage methods exist. As an example consider the case someone calls a script asking for new commits in layer Y to be added to flattened layer Z. We then have to add code checking if layer Y or layer X is a "bitbake remote layer" and then handle the updating of the layer accordingly. Can something external to the bitbake code do that? When we extend the later tooling are we expected to extend this code to match functionality? The abstraction in the remote layers code isn't strong enough to cope with that kind of interaction in its own right and I'm not sure its possible to do that simply with a line in a bblayers.conf file and still be readable. So I guess my question is how do you see this moving forwards? Are you planning to use external layer tooling at all or are you wanting anything that external tooling can do exposed also by the bblayers.conf URI? If this is some kind of stopgap solution for compatibility and is a single blocking issue for Montavista that might be reasonable. If its the start of a move to reverse the order of the stack and put bitbake at the top and layer tooling secondary, I'm worried. Also looking at the patch at a technical level, is there any reason to use _layerUnpack() instead of the fetch2 unpack method? The patch is much cleaner than it was, that much is good but needing to set so many fetcher variables like that is a sign that worries me a little. I guess the solution there is to code them as fallback defaults into the fetchers themselves. Cheers, Richard ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [PATCH] Add support for remote layering. 2011-07-01 21:37 ` Richard Purdie @ 2011-07-02 0:33 ` Jeremy Puhlman 2011-07-04 11:34 ` Richard Purdie 2011-07-04 12:39 ` Paul Eggleton 0 siblings, 2 replies; 20+ messages in thread From: Jeremy Puhlman @ 2011-07-02 0:33 UTC (permalink / raw) To: Patches and discussions about the oe-core layer; +Cc: Paul Eggleton On 7/1/2011 2:37 PM, Richard Purdie wrote: > On Fri, 2011-07-01 at 11:43 -0700, Jeremy Puhlman wrote: >> Just a quick follow up, the part of this statement that to me is wrong, >> is it appears you are dictating work flow. While, imho, the layer >> management code which is above this level, is the place to dicate >> workflow. The patch is to core bitbake and isolated at that. This patch >> adds an additional work flow, and can be used in either state(the >> results of a current BBLAYERS setting is exactly the same today as it as >> after the patch). More or less the patch is useful in a scope larger >> then just layer-tooling. > > Let me try and convey at least one of the things I'm worrying about with > this patch. We're looking at establishing layer tooling which operate > outside of or "above" bitbake. There are a variety of reasons for that, > rightly or wrongly but this patch implies we're going to reverse that. I think that only is true if you view the patch in the scope of the only purpose for the patch is an element of the layer tooling. Being able to specify a uri as a layer is in and of it self a useful ability, irrespective of whether there exists higher order tools to create that list. > If we go forward with the patch and don't reverse that decision it means > that layers can sometimes be a thing that are "above" bitbake, sometimes > not depending on the circumstances. This kind of goes back to work flow issue. Unless you plan to make it so you cannot use the layers with out the specific layering tools that are in development, bitbake will always need to support different work flows. > I'm trying hard to avoid that kind > of interwoven complexity as it makes code changes very hard to make in > the future and leads to frequent breakage where multiple usage methods > exist. Well in this case, there is little that is interwoven. The remote layers bit in the main thread of the execution is a single line in the cooker. Paul was saying that he wanted to more or less still implement the tool with in the bitbake code. Ironically, that will likely be more interwoven then this patch. The patch on its own with out regard to layer tooling for this project, can pretty much stand on its own(which is why I provided it a month or two before we pushed up the content tools). > > As an example consider the case someone calls a script asking for new > commits in layer Y to be added to flattened layer Z. We then have to add > code checking if layer Y or layer X is a "bitbake remote layer" and then > handle the updating of the layer accordingly. When ever you evaluate any of the layers, they are downloaded and unpacked prior to examination of any of the meta data(stay bblayers.conf). More or less any place where you would evalute bblayers.conf you would evaluate the collection uri's. How you update(i.e. if they are git for example) should be setable via the already in place fetch mechanism. More or less you would just need to run the BBLAYERS var through the remote layers class then the end result is the BBLAYERS is just a list of directories like it is now, which is why there is only one line of change in the current thread of execution. > Can something external to > the bitbake code do that? Unless your planning on creating tooling that doesn't use any of the bitbake internals, not unless you want to duplicate a whole bunch of code. > When we extend the later tooling are we > expected to extend this code to match functionality? Well if the end results of the layer tooling is a BBLAYERS with a list of directories as it is now, there is nothing to extend. The code handles that now. More or less if unless you are going in a drastically different direction with the BBLAYERS list, I am not sure what would need to be updated. > > The abstraction in the remote layers code isn't strong enough to cope > with that kind of interaction in its own right and I'm not sure its > possible to do that simply with a line in a bblayers.conf file and still > be readable. My problem with this is currently using the same uri mechanism used in the rest of bitbake fetches. Why pick on it here? > > So I guess my question is how do you see this moving forwards? Well paul more or less said he was going to be writing the remote layering tools with hooks in to bitbake like bitbake-layers, so I am not entirely certain what external means in this context. With the current patch, you could basically write the entire tool calling in to bitbake and pass uri's to the layer fetch code, and either pass in our pass out the path of the final destination. > Are you > planning to use external layer tooling at all or are you wanting > anything that external tooling can do exposed also by the bblayers.conf > URI? Well it is two fold. Right now, the actual form of the end layer-tooling doesn't exist. So no I really couldn't commit to using what ever is produced. However, the entire reason why we provided this patch and the content-tools, is so that we could nudge them in a direction were we would be much more likely to use them. More or less the entire point of the yocto project was to stop inventing the same tools over and over again for each distro, so we can concentrate on the actual value adds that each company/distro/group brings to the table. Basically what your implementing here has already been done, ergo in the spirit of the purpose of the project, we were trying to avoid reinventing the same tools again either here or in yocto/oe-core/bitbake. The path you guys are tracking, is more or less the same track we started on and we ended up with what we provided to you. > If this is some kind of stopgap solution for compatibility and is a > single blocking issue for Montavista that might be reasonable. If its > the start of a move to reverse the order of the stack and put bitbake at > the top and layer tooling secondary, I'm worried. Actually the remote fetching is just a mechanism to empower the layer-tooling and not duplicate code. I am not sure of you have looked at the content-tools I provided last month, but those are basically our "layer-tools". They create two things, collections(which at this point would be layers) and solutions, which are sets of collections/layers that work together. All of that is well above the bitbake level. Given that the layers need to exist/be defined prior to the first execution of bitbake, I wouldn't even begin to fathom how one would make layer-tooling secondary to bitbake. So I don't think I am trying to do that. > > Also looking at the patch at a technical level, is there any reason to > use _layerUnpack() instead of the fetch2 unpack method? The code is more or less, a minor reworking of OE's collection.inc from Chris. So it has just carried that a long for the ride. In addition, it would mean the code could only work with fetch2. fetch does not provide unpack. I originally put this together to work on both version of fetch(which it does). It shouldn't be an issue to call unpack if the fetcher is fetch2. > The patch is > much cleaner than it was, that much is good but needing to set so many > fetcher variables like that is a sign that worries me a little. I have not submitted an updated patch. What you are looking at would have to be the original. :) I got feed back and responded and never got any comment on those responses. > I guess > the solution there is to code them as fallback defaults into the > fetchers themselves. Well as I noted at the start, the setting of the fetcher stuff is primarily because, the fetchers don't start off with reasonable defaults so they are more or less broken until we parse conf/bitbake.conf. In the setup I/we use, we do not start our projects with any local layers/collections at all, so we would not have access to bitbake.conf, unless we went back to sourceing the mini one that used to be provided. We can move the setting of that stuff to the init of the fetchers, and that wouldn't be a big deal. But once you eliminate the stuff to make fetching work, the list would almost entirely go away. ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [PATCH] Add support for remote layering. 2011-07-02 0:33 ` Jeremy Puhlman @ 2011-07-04 11:34 ` Richard Purdie 2011-07-05 15:54 ` Jeremy Puhlman 2011-07-04 12:39 ` Paul Eggleton 1 sibling, 1 reply; 20+ messages in thread From: Richard Purdie @ 2011-07-04 11:34 UTC (permalink / raw) To: Jeremy Puhlman Cc: Paul Eggleton, Patches and discussions about the oe-core layer On Fri, 2011-07-01 at 17:33 -0700, Jeremy Puhlman wrote: > On 7/1/2011 2:37 PM, Richard Purdie wrote: > > On Fri, 2011-07-01 at 11:43 -0700, Jeremy Puhlman wrote: > >> Just a quick follow up, the part of this statement that to me is wrong, > >> is it appears you are dictating work flow. While, imho, the layer > >> management code which is above this level, is the place to dicate > >> workflow. The patch is to core bitbake and isolated at that. This patch > >> adds an additional work flow, and can be used in either state(the > >> results of a current BBLAYERS setting is exactly the same today as it as > >> after the patch). More or less the patch is useful in a scope larger > >> then just layer-tooling. > > > > Let me try and convey at least one of the things I'm worrying about with > > this patch. We're looking at establishing layer tooling which operate > > outside of or "above" bitbake. There are a variety of reasons for that, > > rightly or wrongly but this patch implies we're going to reverse that. > > I think that only is true if you view the patch in the scope of the only > purpose for the patch is an element of the layer tooling. Being able to > specify a uri as a layer is in and of it self a useful ability, > irrespective of whether there exists higher order tools to create that list. I agree the patch has a use in its own right, standalone. I am however asking us all to take a step back and consider the big picture which is what I need do with a lot of what we're doing. My point is that whilst in isolation its ok, I don't think that approach can scale to fulfil all the varying needs of our users. If it can't, we need to look at what can, and then how we could include equivalent functionality. > Well in this case, there is little that is interwoven. The remote layers > bit in the main thread of the execution is a single line in the cooker. > Paul was saying that he wanted to more or less still implement the tool > with in the bitbake code. Ironically, that will likely be more > interwoven then this patch. > > The patch on its own with out regard to layer tooling for this project, > can pretty much stand on its own(which is why I provided it a month or > two before we pushed up the content tools). The interwoven comment has nothing to do with how many lines of code it adds/removes. My concern is that we have a ton of different operations we really need to have around layers and they can't all work through the bitbake command itself as it simply wasn't designed for it. Ending up in a case where the bitbake command may do some things to layers some of the time isn't the consistent user experience I'd like to have. Would you like to explain to a user that yes, bitbake can fetch a remote layer but can't update it? Even with the code we're discussing, you need an external tool to create the list and to update it at appropriate points from upstream so you really might as well have that code actually do the fetch as well (calling bitbake's fetchers)? I'm much more worried about the workflow and its implications than I am about the actual code. > > As an example consider the case someone calls a script asking for new > > commits in layer Y to be added to flattened layer Z. We then have to add > > code checking if layer Y or layer X is a "bitbake remote layer" and then > > handle the updating of the layer accordingly. > > When ever you evaluate any of the layers, they are downloaded and > unpacked prior to examination of any of the meta data(stay > bblayers.conf). More or less any place where you would evalute > bblayers.conf you would evaluate the collection uri's. How you > update(i.e. if they are git for example) should be setable via the > already in place fetch mechanism. > > More or less you would just need to run the BBLAYERS var through the > remote layers class then the end result is the BBLAYERS is just a list > of directories like it is now, which is why there is only one line of > change in the current thread of execution. As I've said, I don't care about the single line of code change. The important thing is whether the work flow can consistently scale into what we as project all need. Unfortunately at the moment its looking like it doesn't. > > The abstraction in the remote layers code isn't strong enough to cope > > with that kind of interaction in its own right and I'm not sure its > > possible to do that simply with a line in a bblayers.conf file and still > > be readable. > > My problem with this is currently using the same uri mechanism used in > the rest of bitbake fetches. Why pick on it here? I think layers are a bit different to a lot of the source code we currently fetch. We might want to push things back, transfer commits between layers and perform a number of other operations. These operations all require "state" type data that currently we can't store in a SRC_URI. I'm not even sure we want to store it there. > Well paul more or less said he was going to be writing the remote > layering tools with hooks in to bitbake like bitbake-layers, so I am not > entirely certain what external means in this context. With the current > patch, you could basically write the entire tool calling in to bitbake > and pass uri's to the layer fetch code, and either pass in our pass out > the path of the final destination. I think Paul is planning something very similar in spirit to the code you have with the difference you'd use a different tool to actually manipulate the build environment and fetch the layers first rather than having bitbake do it itself. > > Are you > > planning to use external layer tooling at all or are you wanting > > anything that external tooling can do exposed also by the bblayers.conf > > URI? > > Well it is two fold. Right now, the actual form of the end layer-tooling > doesn't exist. So no I really couldn't commit to using what ever is > produced. There has been discussion about this in person at Collab/ELC and on the mailing lists/wikis but we probably need to do better at communicating this I guess :/ > However, the entire reason why we provided this patch and the > content-tools, is so that we could nudge them in a direction were we > would be much more likely to use them. > > More or less the entire point of the yocto project was to stop inventing > the same tools over and over again for each distro, so we can > concentrate on the actual value adds that each company/distro/group > brings to the table. > > Basically what your implementing here has already been done, ergo in the > spirit of the purpose of the project, we were trying to avoid > reinventing the same tools again either here or in > yocto/oe-core/bitbake. The path you guys are tracking, is more or less > the same track we started on and we ended up with what we provided to you. The whole point of Yocto is to collaborate. I'm pleased to have your input in the form of the patches and I really do appreciate that. This doesn't however guarantee we take them "as is" since we've also consulted with a number of other people about what their needs are and your solution whilst evidently perfect for you doesn't meet all the criteria we established during the planning process. We're trying to write some tools that should work for everyone. It may require some changes in process for some people, hopefully these will be logical and improve the user experience and understanding. There are a variety of ways I think we could even directly make your use case work (such as bitbake automatically calling the external update tool if some environment variable is set). Collaboration means there are more than just your requirements that need to be worked into this at this point. > > If this is some kind of stopgap solution for compatibility and is a > > single blocking issue for Montavista that might be reasonable. If its > > the start of a move to reverse the order of the stack and put bitbake at > > the top and layer tooling secondary, I'm worried. > > Actually the remote fetching is just a mechanism to empower the > layer-tooling and not duplicate code. I am not sure of you have looked > at the content-tools I provided last month, but those are basically our > "layer-tools". They create two things, collections(which at this point > would be layers) and solutions, which are sets of collections/layers > that work together. All of that is well above the bitbake level. Given > that the layers need to exist/be defined prior to the first execution of > bitbake, I wouldn't even begin to fathom how one would make > layer-tooling secondary to bitbake. > > So I don't think I am trying to do that. So why put all the work into that content-tool code and then have the workflow so bitbake does the actual fetching? It shouldn't be hard to write a "bitbake-layer-fetch" script that does this and it would make a clearer separation of responsibility for the different scripts. What is the reason for not doing that? > > Also looking at the patch at a technical level, is there any reason to > > use _layerUnpack() instead of the fetch2 unpack method? > > The code is more or less, a minor reworking of OE's collection.inc from > Chris. So it has just carried that a long for the ride. In addition, it > would mean the code could only work with fetch2. fetch does not provide > unpack. I originally put this together to work on both version of > fetch(which it does). It shouldn't be an issue to call unpack if the > fetcher is fetch2. What's the opinion of fetch2 at this point? I'm hoping we can rely on that for all future development work at this point. > > The patch is > > much cleaner than it was, that much is good but needing to set so many > > fetcher variables like that is a sign that worries me a little. > > I have not submitted an updated patch. What you are looking at would > have to be the original. :) I got feed back and responded and never got > any comment on those responses. I'm probably confusing this with the first version you sent out I guess, sorry. > > I guess > > the solution there is to code them as fallback defaults into the > > fetchers themselves. > > Well as I noted at the start, the setting of the fetcher stuff is > primarily because, the fetchers don't start off with reasonable defaults > so they are more or less broken until we parse conf/bitbake.conf. > > In the setup I/we use, we do not start our projects with any local > layers/collections at all, so we would not have access to bitbake.conf, > unless we went back to sourceing the mini one that used to be provided. > We can move the setting of that stuff to the init of the fetchers, and > that wouldn't be a big deal. I understand that. You're not thinking about what I'm suggesting ;-) > But once you eliminate the stuff to make fetching work, the list would > almost entirely go away. For example, what I mean is that instead of: fetchcmd = data.getVar("FETCHCOMMAND", d, True) in wget.py, we could update this to something like: defaultcmd = "/usr/bin/env wget -t 5 -q --passive-ftp -P ${DL_DIR} ${URI}" fetchcmd = data.getVar("FETCHCOMMAND", d, True) or defaultcmd Cheers, Richard ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [PATCH] Add support for remote layering. 2011-07-04 11:34 ` Richard Purdie @ 2011-07-05 15:54 ` Jeremy Puhlman 0 siblings, 0 replies; 20+ messages in thread From: Jeremy Puhlman @ 2011-07-05 15:54 UTC (permalink / raw) To: Richard Purdie Cc: Paul Eggleton, Patches and discussions about the oe-core layer > I agree the patch has a use in its own right, standalone. I am however > asking us all to take a step back and consider the big picture which is > what I need do with a lot of what we're doing. > > My point is that whilst in isolation its ok, I don't think that approach > can scale to fulfil all the varying needs of our users. If it can't, we > need to look at what can, and then how we could include equivalent > functionality. Okay. > Even with the code we're discussing, you need an external tool to create > the list and to update it at appropriate points from upstream so you > really might as well have that code actually do the fetch as well > (calling bitbake's fetchers)? Well in its current state, you don't need one, but it makes it a lot easier. OTOH all this discussion more or less centers around BBLAYERS which is already easy to use. > I'm much more worried about the workflow and its implications than I am > about the actual code. Okay. More ore less this really only expands the options for workflow. > I think layers are a bit different to a lot of the source code we > currently fetch. We might want to push things back, transfer commits > between layers and perform a number of other operations. Unless we are writing a new set of fetchers(internal or otherwise), the limitation is the fetchers, since they don't do this. > These > operations all require "state" type data that currently we can't store > in a SRC_URI. I'm not even sure we want to store it there. Wouldn't you have the same issue with a git repo tracking recipe? > > There has been discussion about this in person at Collab/ELC and on > the mailing lists/wikis but we probably need to do better at > communicating this I guess :/ Yes. Been following a long, trying to contribute where it made sense. OTOH, in the end, if it is insufficient for our needs we can't use it. > The whole point of Yocto is to collaborate. I'm pleased to have your > input in the form of the patches and I really do appreciate that. > > This doesn't however guarantee we take them "as is" since we've also > consulted with a number of other people about what their needs are and > your solution whilst evidently perfect for you doesn't meet all the > criteria we established during the planning process. I never suggested that anything be taken as is. In the irc discussions with Paul, we talked about the bits that we wanted to change. > We're trying to > write some tools that should work for everyone. It may require some > changes in process for some people, hopefully these will be logical and > improve the user experience and understanding. Understood, otherwise I wouldn't be having this e-mail exchange with you at all. > There are a variety of ways I think we could even directly make your use > case work (such as bitbake automatically calling the external update > tool if some environment variable is set). Okay. > Collaboration means there are > more than just your requirements that need to be worked into this at > this point. Nor have I suggested that you guys were. > What's the opinion of fetch2 at this point? I'm hoping we can rely on > that for all future development work at this point. From the I am just using fetchers in a more or less rudimentary manner, it feels more or less the same(which kind a was the point). I have not really used any of the additional stuff that has been added in, but more powerful and functionally rich code is always a positive thing. Also I am currently supporting fetch with our older product so when I wrote the patch, making it work with fetch seemed a good idea. > > I'm probably confusing this with the first version you sent out I guess, > sorry. > That was my point, there was only one. Really doesn't matter. >> We can move the setting of that stuff to the init of the fetchers, and >> that wouldn't be a big deal. > > I understand that. You're not thinking about what I'm suggesting ;-) > I caught what you were striving for there. -- Jeremy Puhlman Montavista Sofware, LLC. ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [PATCH] Add support for remote layering. 2011-07-02 0:33 ` Jeremy Puhlman 2011-07-04 11:34 ` Richard Purdie @ 2011-07-04 12:39 ` Paul Eggleton 2011-07-05 23:38 ` Jeremy Puhlman 1 sibling, 1 reply; 20+ messages in thread From: Paul Eggleton @ 2011-07-04 12:39 UTC (permalink / raw) To: Jeremy Puhlman; +Cc: openembedded-core On Saturday 02 July 2011 01:33:58 Jeremy Puhlman wrote: > On 7/1/2011 2:37 PM, Richard Purdie wrote: > > If we go forward with the patch and don't reverse that decision it means > > that layers can sometimes be a thing that are "above" bitbake, sometimes > > not depending on the circumstances. > > This kind of goes back to work flow issue. Unless you plan to make it so > you cannot use the layers with out the specific layering tools that are > in development, bitbake will always need to support different work flows. We aren't going to tie bitbake layer support of local layers to the layer tooling, no. We're expressly trying to avoid that. You suggest that I'm trying to "dictate workflow" however I'd argue that I'm doing the opposite (see below regarding updating). > > I'm trying hard to avoid that kind > > of interwoven complexity as it makes code changes very hard to make in > > the future and leads to frequent breakage where multiple usage methods > > exist. > > Well in this case, there is little that is interwoven. The remote layers > bit in the main thread of the execution is a single line in the cooker. > Paul was saying that he wanted to more or less still implement the tool > with in the bitbake code. Ironically, that will likely be more > interwoven then this patch. I don't think that they will. Firstly, no hooks are needed to fetch a layer remotely - it's just a fetch; you call cooker to run through a parse and then run the fetch operations you need. No changes to the bitbake core should be necessary, as far as I can see. (I don't object to reworking fetcher initialisation so that they are set up correctly "out of the box", though, that may be helpful to both approaches.) Secondly, the changes I have previously implemented within bitbake for use by bitbake-layers are generic and have negligible impact on bitbake operation. If you have any technical concerns with these, in all seriousness please reply to the patches on bitbake-devel; there's always room for improvement. > When ever you evaluate any of the layers, they are downloaded and > unpacked prior to examination of any of the meta data(stay > bblayers.conf). More or less any place where you would evalute > bblayers.conf you would evaluate the collection uri's. How you > update(i.e. if they are git for example) should be setable via the > already in place fetch mechanism. Updating is something I would like to allow to be a conscious action. That way, if you want to fetch down the metadata, then disconnect and carry on building, you can do so easily. Also, if you want to stay with the current version you have on disk, you know bitbake is not going to update the metadata in the course of doing the build, because you didn't explicitly ask it to. You're also free to make any additional changes on top of the fetched metadata before running the build. If bitbake is doing all the fetching/updating of metadata then immediately jumping into the build, there's no room for that - unless you use Ctrl+C to break out, which isn't really ideal. I think the piece I am missing at the moment is why having it as an external tool that largely replicates the same outward functionality as having it within bitbake presents a problem for the use case that you have. Is there some technical capability that we couldn't have using this approach - other than the fact that you're just calling bitbake and it does everything? If that's the only objection, would it not achieve the same thing if you had a small shell script that ran the fetch/update then bitbake? > More or less you would just need to run the BBLAYERS var through the > remote layers class then the end result is the BBLAYERS is just a list > of directories like it is now, which is why there is only one line of > change in the current thread of execution. > > > Can something external to > > the bitbake code do that? > > Unless your planning on creating tooling that doesn't use any of the > bitbake internals, not unless you want to duplicate a whole bunch of code. I don't plan to duplicate any significant amount of code. I'm not seeing that you couldn't do what was described using an external tool. > > When we extend the later tooling are we > > expected to extend this code to match functionality? > > Well if the end results of the layer tooling is a BBLAYERS with a list > of directories as it is now, there is nothing to extend. The code > handles that now. More or less if unless you are going in a drastically > different direction with the BBLAYERS list, I am not sure what would > need to be updated. Well, with my solution I would prefer to see BBLAYERS unchanged, so it just points to the local checkout of the remote layer. > > The abstraction in the remote layers code isn't strong enough to cope > > with that kind of interaction in its own right and I'm not sure its > > possible to do that simply with a line in a bblayers.conf file and still > > be readable. > > My problem with this is currently using the same uri mechanism used in > the rest of bitbake fetches. Why pick on it here? Fetching metadata and fetching source code are two different things. You don't really expect to have to do anything to the source code before bitbake builds it - bitbake and whatever metadata you have takes care of that. However with the metadata you may wish to make modifications prior to building. > > So I guess my question is how do you see this moving forwards? > > Well paul more or less said he was going to be writing the remote > layering tools with hooks in to bitbake like bitbake-layers, so I am not > entirely certain what external means in this context. With the current > patch, you could basically write the entire tool calling in to bitbake > and pass uri's to the layer fetch code, and either pass in our pass out > the path of the final destination. External means it's not part of the normal bitbake execution flow and there's nothing that needs to be changed within the bitbake core, unless it's of a generic nature (and right now I don't see any needed changes with my solution). No hooks should be necessary. > > Are you planning to use external layer tooling at all or are you wanting > > anything that external tooling can do exposed also by the bblayers.conf > > URI? > > Well it is two fold. Right now, the actual form of the end layer-tooling > doesn't exist. So no I really couldn't commit to using what ever is > produced. However, the entire reason why we provided this patch and the > content-tools, is so that we could nudge them in a direction were we > would be much more likely to use them. And that's a good thing, agreed. > More or less the entire point of the yocto project was to stop inventing > the same tools over and over again for each distro, so we can > concentrate on the actual value adds that each company/distro/group > brings to the table. Right. To me, that means going from a list of requirements (and I'll admit, it has been an informal list for this particular feature) and trying to come up with an implementation that fulfils those requirements. > Basically what your implementing here has already been done, ergo in the > spirit of the purpose of the project, we were trying to avoid > reinventing the same tools again either here or in > yocto/oe-core/bitbake. The path you guys are tracking, is more or less > the same track we started on and we ended up with what we provided to you. You have come up with an implementation that works well for you, and you've published that implementation. Thankyou for that. However, there are some aspects of that implementation that concern Richard and I so I'm trying to come up with an alternative implementation (using some of the same code) that meets the requirements. If we're not going to meet some important requirements that you have then I'm more than happy to stop and re-evaluate the situation. > > Also looking at the patch at a technical level, is there any reason to > > use _layerUnpack() instead of the fetch2 unpack method? > > The code is more or less, a minor reworking of OE's collection.inc from > Chris. So it has just carried that a long for the ride. In addition, it > would mean the code could only work with fetch2. fetch does not provide > unpack. I originally put this together to work on both version of > fetch(which it does). It shouldn't be an issue to call unpack if the > fetcher is fetch2. Understood. However, I'd hope we can soon move to a state where we only have one set of fetchers. Is there anything preventing Montavista from using fetch2 currently? > > The patch is > > much cleaner than it was, that much is good but needing to set so many > > fetcher variables like that is a sign that worries me a little. > > I have not submitted an updated patch. What you are looking at would > have to be the original. :) I got feed back and responded and never got > any comment on those responses. Well, we definitely would need the "layer" fetcher code to be refactored out; Richard made that comment earlier. I would also hope that as you suggested we can move to having the initialisation within the fetchers themselves so that the layer fetching isn't doing anything special in that regard. We need to establish the direction we want to go in however. Cheers, Paul -- Paul Eggleton Intel Open Source Technology Centre ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [PATCH] Add support for remote layering. 2011-07-04 12:39 ` Paul Eggleton @ 2011-07-05 23:38 ` Jeremy Puhlman 2011-07-11 17:46 ` Paul Eggleton 0 siblings, 1 reply; 20+ messages in thread From: Jeremy Puhlman @ 2011-07-05 23:38 UTC (permalink / raw) To: Paul Eggleton; +Cc: openembedded-core > > We aren't going to tie bitbake layer support of local layers to the layer > tooling, no. We're expressly trying to avoid that. > > You suggest that I'm trying to "dictate workflow" however I'd argue that I'm > doing the opposite (see below regarding updating). The line that led me to dicate: Fetching/updating layers should be (able to be) a conscious action rather than something that happens implicitly as part of the build process. Was not as firm a statement as was thinking it was. Apologies. > I don't think that they will. Firstly, no hooks are needed to fetch a layer > remotely - it's just a fetch; you call cooker to run through a parse and then > run the fetch operations you need. No changes to the bitbake core should be > necessary, as far as I can see. (I don't object to reworking fetcher > initialisation so that they are set up correctly "out of the box", though, > that may be helpful to both approaches.) I think this is kinda where I get hung up. The run through parse part. Our projects start off with zero meta data other then a local.conf and bblayers.conf. In earlier days, we provided collections.inc and bitbake.conf in a local conf directory provided with bitbake. That is not done anymore(ergo why the fetchers are broken from go, previously nothing happened until at lease one bitbake.conf was loaded). > Secondly, the changes I have previously implemented within bitbake for use by > bitbake-layers are generic and have negligible impact on bitbake operation. > If you have any technical concerns with these, in all seriousness please reply > to the patches on bitbake-devel; there's always room for improvement. Ill dig in to these, I have not done much digging, but I did see the flurry of recent patches. > Updating is something I would like to allow to be a conscious action. That > way, if you want to fetch down the metadata, then disconnect and carry on > building, you can do so easily. Also, if you want to stay with the current > version you have on disk, you know bitbake is not going to update the metadata > in the course of doing the build, because you didn't explicitly ask it to. > You're also free to make any additional changes on top of the fetched metadata > before running the build. If bitbake is doing all the fetching/updating of > metadata then immediately jumping into the build, there's no room for that - > unless you use Ctrl+C to break out, which isn't really ideal. So not to defend the patch again here, but this is not a problem with the patch either. Currently if you run "bitbake" and provide no other options, the patch will just fetch everything. You wouldn't need to Ctrl+C to make modifications. Now you can just jump off in to a build if it is what you want to do. Actually anything that calls cooker basically causes them to be fetched. Running bitbake-layers for example does the fetch. > I think the piece I am missing at the moment is why having it as an external > tool that largely replicates the same outward functionality as having it > within bitbake presents a problem for the use case that you have. Is there > some technical capability that we couldn't have using this approach - other > than the fact that you're just calling bitbake and it does everything? If > that's the only objection, would it not achieve the same thing if you had a > small shell script that ran the fetch/update then bitbake? Pretty much everything is a small matter of code. It is a matter of complexity. More or less when you boil it down, bitbakewith out content is fantastic at a few basic tasks, one of them being fetching. Its a kinda straight line to let bitbake be bitbake. I get it you don't want it to be automatic. Do you have something I can look at that addresses the remote layering? If your not saving the layers data in BBLAYERS where are you saving it? > > Fetching metadata and fetching source code are two different things. You don't > really expect to have to do anything to the source code before bitbake builds > it - bitbake and whatever metadata you have takes care of that. However with > the metadata you may wish to make modifications prior to building. It really comes down to use case. Since we started out with a bitbake/oe derived product, our collections have been always been stowed in tmp. Basically keep the upstream pristine make changes with mechanisms like amend.inc/.bbappend in more private layers. Now we are more or less working with non-oe developers, and folks that are more or less complete novices when it comes to this type of stuff. I accept this is only one use case, however I actually think it is more important then the standard oe-developer case. Most oe-developers could probably take the layers wiki page listing all the disparate layers and be good. Its the folks that are not as oe/bitbake sophicticated that this helps. Richard was commenting about moving checkin's from layer to layer. While for me in my personal work, that is fantastic. However, I can already do that with out the aid of a separate tool. Most people who use git in there day to day work could as well, whether they have ever heard of bitbake or oe or not. While that functionality is "neat", it shouldn't, imho, be added to complicate basic functionality. >> Basically what your implementing here has already been done, ergo in the >> spirit of the purpose of the project, we were trying to avoid >> reinventing the same tools again either here or in >> yocto/oe-core/bitbake. The path you guys are tracking, is more or less >> the same track we started on and we ended up with what we provided to you. > > You have come up with an implementation that works well for you, and you've > published that implementation. Thankyou for that. However, there are some > aspects of that implementation that concern Richard and I so I'm trying to > come up with an alternative implementation (using some of the same code) that > meets the requirements. If we're not going to meet some important requirements > that you have then I'm more than happy to stop and re-evaluate the situation. Well what we have more or less been able to accomplish with what we have provided is the ability for a user to generate a project with nothing locally stored but the uri's. The end user sources a script and runs bitbake. The layers/collections are retrieved and it is off and running. The content-tools more or less shuffle around uri's, source mirrors and prebuilt mirrors. The project lacking more or less any metadata other the configuration options can get dropped right in to scm, especially when using remote uris. Some of the above may not fit in the scope of what we are trying to do here, which is why I have been pushing some of the lower functionality that can be reused. If it comes down to running an extra command prior to running bitbake then will work. The one thing I would prefer is that the whole set of layers(including oe-core) be able to be checked out, rather then checking out oe-core first then checkout any additional layers. In that same vain supporting archives as valid layer types, would be helpful. > Understood. However, I'd hope we can soon move to a state where we only > have one set of fetchers. Is there anything preventing Montavista from using > fetch2 currently? Backwards compatibility. Currently we are supporting legacy code, that would require some rework to go with fetch2. For those products I could halt the forward progression on bitbake, but using a single version of bitbake would be better for us, though I completely understand the removal of the older fetch. Its more or less my problem then anything you need to deal with. > Well, we definitely would need the "layer" fetcher code to be refactored out; > Richard made that comment earlier. Well the code that lives there now would need to move in to __init__.py or in to the remotealyers bit with if else bits for fetch/fetch2. OTOH, it sounds like the decision is not to use the patch at all. > I would also hope that as you suggested we > can move to having the initialisation within the fetchers themselves so that > the layer fetching isn't doing anything special in that regard. We need to > establish the direction we want to go in however. Well yeah this is general goodness. -- Jeremy Puhlman Montavista Sofware, LLC. ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [PATCH] Add support for remote layering. 2011-07-05 23:38 ` Jeremy Puhlman @ 2011-07-11 17:46 ` Paul Eggleton 2011-07-11 18:45 ` Jeremy Puhlman 2011-07-18 15:59 ` Paul Eggleton 0 siblings, 2 replies; 20+ messages in thread From: Paul Eggleton @ 2011-07-11 17:46 UTC (permalink / raw) To: Jeremy Puhlman; +Cc: openembedded-core On Wednesday 06 July 2011 00:38:41 Jeremy Puhlman wrote: > Our projects start off with zero meta data other then a local.conf and > bblayers.conf. In earlier days, we provided collections.inc and > bitbake.conf in a local conf directory provided with bitbake. That is > not done anymore(ergo why the fetchers are broken from go, previously > nothing happened until at lease one bitbake.conf was loaded). I am aiming for this use case to be supported with my utility. What I would also like to handle though is that if you have chosen to set up some configuration, then it will be able to be read in before the fetching starts. > I get it you don't want it to be automatic. Do you have something I can look > at that addresses the remote layering? I've thrown together a proof-of-concept "bitbake-fetchlayers" in the "paule/remotelayers" contrib branch: http://git.yoctoproject.org/cgit/cgit.cgi/poky- contrib/log/?h=paule/remotelayers This is by no means a finished utility, may have hideous bugs, etc. This requires nothing more than vanilla bitbake to operate. Some notes: * BBPATH needs to be set, LAYER_UNPACKDIR also. * Currently it requires conf/bitbake.conf, classes/ etc., which are shipped with bitbake master but are not present with the copy of bitbake that's in Poky; I hope to be able to address this in such a way that the utility does not require these. * "init" is the command used to fetch multiple layers. I've also provided "fetch" as a way to test a single fetch operation; I would expect the latter to be removed at some point. (Also, none of these command names are final.) * Update is not yet implemented. For your use case, update is no more than a re-fetch; however where you are intending to interact with the layers as SCM working directories it would be better to do an update in-place. I'm still thinking about how best to implement this. * Output is rather noisy, this needs to be fixed also. * I did have to patch the fetchers as Richard suggested so they have default values for the configurable fetch commands. (We'd have to have done this anyway.) > If your not saving the layers data in BBLAYERS where are you saving it? Well, this is not entirely clear to me yet. However if they are not in bblayers.conf and you wish to refer to them again (as in your use case you would for updating) they would naturally have to be in another file somewhere. FWIW bitbake-fetchlayers' "init" command currently takes a list consisting of uri,localdir,subdir items, with the last two parameters being optional, and outputs a new BBLAYERS definition. This list could alternatively come from a file. > It really comes down to use case. Since we started out with a bitbake/oe > derived product, our collections have been always been stowed in tmp. > Basically keep the upstream pristine make changes with mechanisms like > amend.inc/.bbappend in more private layers. Now we are more or less > working with non-oe developers, and folks that are more or less complete > novices when it comes to this type of stuff. > > I accept this is only one use case, however I actually think it is more > important then the standard oe-developer case. Most oe-developers could > probably take the layers wiki page listing all the disparate layers and > be good. Its the folks that are not as oe/bitbake sophicticated that > this helps. Ultimately anyone coming to bitbake/OE fresh is going to need to know something about setting it up and using it. However, any mechanical steps can be scripted away. For these kinds of users, running "bitbake <target>" or "startbuild.sh" is not going to be much different. > Richard was commenting about moving checkin's from layer to layer. While > for me in my personal work, that is fantastic. However, I can already do > that with out the aid of a separate tool. Most people who use git in > there day to day work could as well, whether they have ever heard of > bitbake or oe or not. While that functionality is "neat", it shouldn't, > imho, be added to complicate basic functionality. Richard's aim is simply to keep the core of bitbake free of complexities that may be painful to extend if they are in the core. > If it comes down to running an extra command prior to running bitbake > then will work. The one thing I would prefer is that the whole set of > layers(including oe-core) be able to be checked out, rather then > checking out oe-core first then checkout any additional layers. In that > same vain supporting archives as valid layer types, would be helpful. I think we can support this without any issues - any of bitbake's fetchers should be able to be used, including wget and local. You won't need to grab oe-core first. > Backwards compatibility. Currently we are supporting legacy code, that > would require some rework to go with fetch2. For those products I could > halt the forward progression on bitbake, but using a single version of > bitbake would be better for us, though I completely understand the > removal of the older fetch. Its more or less my problem then anything > you need to deal with. Hmm. For simplicity I've only supported fetch2 in bitbake-fetchlayers - in fact it forces it at startup. It would not be hard to support fetch also, however I hope we could avoid having to do so. Cheers, Paul -- Paul Eggleton Intel Open Source Technology Centre ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [PATCH] Add support for remote layering. 2011-07-11 17:46 ` Paul Eggleton @ 2011-07-11 18:45 ` Jeremy Puhlman 2011-07-18 15:59 ` Paul Eggleton 1 sibling, 0 replies; 20+ messages in thread From: Jeremy Puhlman @ 2011-07-11 18:45 UTC (permalink / raw) To: Paul Eggleton; +Cc: openembedded-core > I am aiming for this use case to be supported with my utility. What I would > also like to handle though is that if you have chosen to set up some > configuration, then it will be able to be read in before the fetching starts. Sounds reasonable. >> I get it you don't want it to be automatic. Do you have something I can look >> at that addresses the remote layering? > > I've thrown together a proof-of-concept "bitbake-fetchlayers" in the > "paule/remotelayers" contrib branch: > > http://git.yoctoproject.org/cgit/cgit.cgi/poky- > contrib/log/?h=paule/remotelayers > > This is by no means a finished utility, may have hideous bugs, etc. This > requires nothing more than vanilla bitbake to operate. Some notes: > > * BBPATH needs to be set, LAYER_UNPACKDIR also. > > * Currently it requires conf/bitbake.conf, classes/ etc., which are shipped > with bitbake master but are not present with the copy of bitbake that's in > Poky; I hope to be able to address this in such a way that the utility does > not require these. > > * "init" is the command used to fetch multiple layers. I've also provided > "fetch" as a way to test a single fetch operation; I would expect the latter > to be removed at some point. (Also, none of these command names are final.) > > * Update is not yet implemented. For your use case, update is no more than a > re-fetch; however where you are intending to interact with the layers as SCM > working directories it would be better to do an update in-place. I'm still > thinking about how best to implement this. I will try and check this out later this week. > * Output is rather noisy, this needs to be fixed also. k. > * I did have to patch the fetchers as Richard suggested so they have default > values for the configurable fetch commands. (We'd have to have done this > anyway.) Great. > > I think we can support this without any issues - any of bitbake's fetchers > should be able to be used, including wget and local. You won't need to grab > oe-core first. Okay. > Hmm. For simplicity I've only supported fetch2 in bitbake-fetchlayers - in > fact it forces it at startup. It would not be hard to support fetch also, > however I hope we could avoid having to do so. That is fine. The legacy code is likely not going to be moving over to this anyways. It was more of a response to why did you do it, rather then it needs to support it. Don't bother unless it is really needed by someone else. -- Jeremy Puhlman Montavista Sofware, LLC. ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [PATCH] Add support for remote layering. 2011-07-11 17:46 ` Paul Eggleton 2011-07-11 18:45 ` Jeremy Puhlman @ 2011-07-18 15:59 ` Paul Eggleton 1 sibling, 0 replies; 20+ messages in thread From: Paul Eggleton @ 2011-07-18 15:59 UTC (permalink / raw) To: openembedded-core On Monday 11 July 2011 18:46:26 Paul Eggleton wrote: > I've thrown together a proof-of-concept "bitbake-fetchlayers" in the > "paule/remotelayers" contrib branch: > > http://git.yoctoproject.org/cgit/cgit.cgi/poky- > contrib/log/?h=paule/remotelayers I spent some time looking at this again today, and the contrib branch has now been updated. Changes are as follows: * bitbake.conf and base.bbclass are no longer needed, so you really only need bitbake now. Some fixes to some of the fetchers other than git might still be necessary however. * Only BBPATH now needs to be set; LAYER_UNPACKDIR now defaults to ${TOPDIR}/layers. Cheers, Paul -- Paul Eggleton Intel Open Source Technology Centre ^ permalink raw reply [flat|nested] 20+ messages in thread
end of thread, other threads:[~2011-07-18 16:03 UTC | newest] Thread overview: 20+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- [not found] <RFC: Layer tooling brainstorming> 2011-04-28 18:09 ` [PATCH] Add support for remote layering Jeremy Puhlman 2011-04-28 18:09 ` Jeremy Puhlman 2011-04-28 18:20 ` [PATCH 0/1] " Jeremy Puhlman 2011-05-06 13:15 ` [PATCH] " Richard Purdie 2011-05-12 13:11 ` Jeremy Puhlman 2011-05-12 17:34 ` Jeremy Puhlman 2011-05-20 16:45 ` Paul Eggleton 2011-05-20 17:42 ` Jeremy Puhlman 2011-07-01 13:24 ` Paul Eggleton 2011-07-01 17:17 ` Jeremy Puhlman 2011-07-01 18:43 ` Jeremy Puhlman 2011-07-01 21:37 ` Richard Purdie 2011-07-02 0:33 ` Jeremy Puhlman 2011-07-04 11:34 ` Richard Purdie 2011-07-05 15:54 ` Jeremy Puhlman 2011-07-04 12:39 ` Paul Eggleton 2011-07-05 23:38 ` Jeremy Puhlman 2011-07-11 17:46 ` Paul Eggleton 2011-07-11 18:45 ` Jeremy Puhlman 2011-07-18 15:59 ` Paul Eggleton
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.