* [PATCH 1/6] fetch2/npm: explicitly specify workdir
2016-09-05 22:57 [PATCH 0/6] npm fetcher fixes Paul Eggleton
@ 2016-09-05 22:57 ` Paul Eggleton
2016-09-05 22:57 ` [PATCH 2/6] fetch2/npm: fix broken fetches if more than one npm URL fetched Paul Eggleton
` (4 subsequent siblings)
5 siblings, 0 replies; 7+ messages in thread
From: Paul Eggleton @ 2016-09-05 22:57 UTC (permalink / raw)
To: bitbake-devel
We were downloading into the current directory here, which is fine if
that current directory can be expected to be the right place - but
that's not true when called from recipetool within OE. We should
explicitly specify the directory to run the command in and then there
won't be a problem.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
lib/bb/fetch2/npm.py | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/lib/bb/fetch2/npm.py b/lib/bb/fetch2/npm.py
index b26ac22..65ccb0d 100644
--- a/lib/bb/fetch2/npm.py
+++ b/lib/bb/fetch2/npm.py
@@ -102,7 +102,8 @@ class Npm(FetchMethod):
def _runwget(self, ud, d, command, quiet):
logger.debug(2, "Fetching %s using command '%s'" % (ud.url, command))
bb.fetch2.check_network_access(d, command)
- runfetchcmd(command, d, quiet)
+ dldir = d.getVar("DL_DIR", True)
+ runfetchcmd(command, d, quiet, workdir=dldir)
def _unpackdep(self, ud, pkg, data, destdir, dldir, d):
file = data[pkg]['tgz']
--
2.5.5
^ permalink raw reply related [flat|nested] 7+ messages in thread
* [PATCH 2/6] fetch2/npm: fix broken fetches if more than one npm URL fetched
2016-09-05 22:57 [PATCH 0/6] npm fetcher fixes Paul Eggleton
2016-09-05 22:57 ` [PATCH 1/6] fetch2/npm: explicitly specify workdir Paul Eggleton
@ 2016-09-05 22:57 ` Paul Eggleton
2016-09-05 22:57 ` [PATCH 3/6] fetch2/npm: support subdir= parameter Paul Eggleton
` (3 subsequent siblings)
5 siblings, 0 replies; 7+ messages in thread
From: Paul Eggleton @ 2016-09-05 22:57 UTC (permalink / raw)
To: bitbake-devel
You cannot set a URL-specific value in an object-level variable on
the FetchMethod in urldata_init() or the result is the value specific to
the last URL will be the one that gets set. This prevented fetching more
than one npm:// URL correctly - the other tarballs would not download to
the correct location and do_unpack failed to find them as a result.
Fix required in order to support [YOCTO #9537].
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
lib/bb/fetch2/npm.py | 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)
diff --git a/lib/bb/fetch2/npm.py b/lib/bb/fetch2/npm.py
index 65ccb0d..9b2d9e1 100644
--- a/lib/bb/fetch2/npm.py
+++ b/lib/bb/fetch2/npm.py
@@ -88,7 +88,7 @@ class Npm(FetchMethod):
ud.localpath = d.expand("${DL_DIR}/npm/%s" % ud.bbnpmmanifest)
self.basecmd = d.getVar("FETCHCMD_wget", True) or "/usr/bin/env wget -O -t 2 -T 30 -nv --passive-ftp --no-check-certificate "
- self.basecmd += " --directory-prefix=%s " % prefixdir
+ ud.prefixdir = prefixdir
ud.write_tarballs = ((data.getVar("BB_GENERATE_MIRROR_TARBALLS", d, True) or "0") != "0")
ud.mirrortarball = 'npm_%s-%s.tar.xz' % (ud.pkgname, ud.version)
@@ -182,7 +182,7 @@ class Npm(FetchMethod):
outputurl = pdata['dist']['tarball']
data[pkg] = {}
data[pkg]['tgz'] = os.path.basename(outputurl)
- self._runwget(ud, d, "%s %s" % (self.basecmd, outputurl), False)
+ self._runwget(ud, d, "%s --directory-prefix=%s %s" % (self.basecmd, ud.prefixdir, outputurl), False)
dependencies = pdata.get('dependencies', {})
optionalDependencies = pdata.get('optionalDependencies', {})
@@ -209,7 +209,7 @@ class Npm(FetchMethod):
outputurl = runfetchcmd(fetchcmd, d, True)
else:
outputurl = data['resolved']
- self._runwget(ud, d, "%s %s" % (self.basecmd, outputurl), False)
+ self._runwget(ud, d, "%s --directory-prefix=%s %s" % (self.basecmd, ud.prefixdir, outputurl), False)
manifest[pkg] = {}
manifest[pkg]['tgz'] = os.path.basename(outputurl).rstrip()
manifest[pkg]['deps'] = {}
--
2.5.5
^ permalink raw reply related [flat|nested] 7+ messages in thread
* [PATCH 3/6] fetch2/npm: support subdir= parameter
2016-09-05 22:57 [PATCH 0/6] npm fetcher fixes Paul Eggleton
2016-09-05 22:57 ` [PATCH 1/6] fetch2/npm: explicitly specify workdir Paul Eggleton
2016-09-05 22:57 ` [PATCH 2/6] fetch2/npm: fix broken fetches if more than one npm URL fetched Paul Eggleton
@ 2016-09-05 22:57 ` Paul Eggleton
2016-09-05 22:57 ` [PATCH 4/6] fetch2/npm: handle top-level shrinkwrap file Paul Eggleton
` (2 subsequent siblings)
5 siblings, 0 replies; 7+ messages in thread
From: Paul Eggleton @ 2016-09-05 22:57 UTC (permalink / raw)
To: bitbake-devel
"npmpkg" can be a default, but it should respect the subdir parameter as
with other FetchMethods. This allows us to have more than one npm://
entry in SRC_URI without nasty hacks.
Fix required in order to support [YOCTO #9537].
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
lib/bb/fetch2/npm.py | 7 ++++++-
1 file changed, 6 insertions(+), 1 deletion(-)
diff --git a/lib/bb/fetch2/npm.py b/lib/bb/fetch2/npm.py
index 9b2d9e1..134bd35 100644
--- a/lib/bb/fetch2/npm.py
+++ b/lib/bb/fetch2/npm.py
@@ -138,7 +138,12 @@ class Npm(FetchMethod):
workobj = json.load(datafile)
dldir = "%s/%s" % (os.path.dirname(ud.localpath), ud.pkgname)
- self._unpackdep(ud, ud.pkgname, workobj, "%s/npmpkg" % destdir, dldir, d)
+ if 'subdir' in ud.parm:
+ unpackdir = '%s/%s' % (destdir, ud.parm.get('subdir'))
+ else:
+ unpackdir = '%s/npmpkg' % destdir
+
+ self._unpackdep(ud, ud.pkgname, workobj, unpackdir, dldir, d)
def _parse_view(self, output):
'''
--
2.5.5
^ permalink raw reply related [flat|nested] 7+ messages in thread
* [PATCH 4/6] fetch2/npm: handle top-level shrinkwrap file
2016-09-05 22:57 [PATCH 0/6] npm fetcher fixes Paul Eggleton
` (2 preceding siblings ...)
2016-09-05 22:57 ` [PATCH 3/6] fetch2/npm: support subdir= parameter Paul Eggleton
@ 2016-09-05 22:57 ` Paul Eggleton
2016-09-05 22:57 ` [PATCH 5/6] fetch2/npm: clarify comment Paul Eggleton
2016-09-05 22:57 ` [PATCH 6/6] lib/bb/utils: edit_metadata() comment tweaks Paul Eggleton
5 siblings, 0 replies; 7+ messages in thread
From: Paul Eggleton @ 2016-09-05 22:57 UTC (permalink / raw)
To: bitbake-devel
Allow using a top-level shrinkwrap file with one or more npm://
dependencies, i.e. if the module isn't found at the top level then look
one level down.
Part of the fix for [YOCTO #9537].
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
lib/bb/fetch2/npm.py | 11 +++++++++--
1 file changed, 9 insertions(+), 2 deletions(-)
diff --git a/lib/bb/fetch2/npm.py b/lib/bb/fetch2/npm.py
index 134bd35..08d6876 100644
--- a/lib/bb/fetch2/npm.py
+++ b/lib/bb/fetch2/npm.py
@@ -204,8 +204,15 @@ class Npm(FetchMethod):
for dep, version in depsfound.items():
self._getdependencies(dep, data[pkg]['deps'], version, d, ud)
- def _getshrinkeddependencies(self, pkg, data, version, d, ud, lockdown, manifest):
+ def _getshrinkeddependencies(self, pkg, data, version, d, ud, lockdown, manifest, toplevel=True):
logger.debug(2, "NPM shrinkwrap file is %s" % data)
+ if toplevel:
+ name = data.get('name', None)
+ if name and name != pkg:
+ for obj in data.get('dependencies', []):
+ if obj == pkg:
+ self._getshrinkeddependencies(obj, data['dependencies'][obj], data['dependencies'][obj]['version'], d, ud, lockdown, manifest, False)
+ return
outputurl = "invalid"
if ('resolved' not in data) or (not data['resolved'].startswith('http')):
# will be the case for ${PN}
@@ -231,7 +238,7 @@ class Npm(FetchMethod):
if 'dependencies' in data:
for obj in data['dependencies']:
logger.debug(2, "Found dep is %s" % str(obj))
- self._getshrinkeddependencies(obj, data['dependencies'][obj], data['dependencies'][obj]['version'], d, ud, lockdown, manifest[pkg]['deps'])
+ self._getshrinkeddependencies(obj, data['dependencies'][obj], data['dependencies'][obj]['version'], d, ud, lockdown, manifest[pkg]['deps'], False)
def download(self, ud, d):
"""Fetch url"""
--
2.5.5
^ permalink raw reply related [flat|nested] 7+ messages in thread
* [PATCH 5/6] fetch2/npm: clarify comment
2016-09-05 22:57 [PATCH 0/6] npm fetcher fixes Paul Eggleton
` (3 preceding siblings ...)
2016-09-05 22:57 ` [PATCH 4/6] fetch2/npm: handle top-level shrinkwrap file Paul Eggleton
@ 2016-09-05 22:57 ` Paul Eggleton
2016-09-05 22:57 ` [PATCH 6/6] lib/bb/utils: edit_metadata() comment tweaks Paul Eggleton
5 siblings, 0 replies; 7+ messages in thread
From: Paul Eggleton @ 2016-09-05 22:57 UTC (permalink / raw)
To: bitbake-devel
The correct name of the parameter is "version" not "ver" so ensure we
aren't misleading the user by giving the latter in an example.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
lib/bb/fetch2/npm.py | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/lib/bb/fetch2/npm.py b/lib/bb/fetch2/npm.py
index 08d6876..43929ce 100644
--- a/lib/bb/fetch2/npm.py
+++ b/lib/bb/fetch2/npm.py
@@ -13,7 +13,7 @@ Usage in the recipe:
- name
- version
- npm://registry.npmjs.org/${PN}/-/${PN}-${PV}.tgz would become npm://registry.npmjs.org;name=${PN};ver=${PV}
+ npm://registry.npmjs.org/${PN}/-/${PN}-${PV}.tgz would become npm://registry.npmjs.org;name=${PN};version=${PV}
The fetcher all triggers off the existence of ud.localpath. If that exists and has the ".done" stamp, its assumed the fetch is good/done
"""
--
2.5.5
^ permalink raw reply related [flat|nested] 7+ messages in thread
* [PATCH 6/6] lib/bb/utils: edit_metadata() comment tweaks
2016-09-05 22:57 [PATCH 0/6] npm fetcher fixes Paul Eggleton
` (4 preceding siblings ...)
2016-09-05 22:57 ` [PATCH 5/6] fetch2/npm: clarify comment Paul Eggleton
@ 2016-09-05 22:57 ` Paul Eggleton
5 siblings, 0 replies; 7+ messages in thread
From: Paul Eggleton @ 2016-09-05 22:57 UTC (permalink / raw)
To: bitbake-devel
No functional changes, just make a couple of minor tweaks to the
comments for edit_metadata():
* There are four elements to be returned by the callback function
* Add an example return statement for when you don't want to modify the
value
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
lib/bb/utils.py | 4 +++-
1 file changed, 3 insertions(+), 1 deletion(-)
diff --git a/lib/bb/utils.py b/lib/bb/utils.py
index a855d58..729848a 100644
--- a/lib/bb/utils.py
+++ b/lib/bb/utils.py
@@ -1081,7 +1081,7 @@ def edit_metadata(meta_lines, variables, varfunc, match_overrides=False):
newlines: list of lines up to this point. You can use
this to prepend lines before this variable setting
if you wish.
- and should return a three-element tuple:
+ and should return a four-element tuple:
newvalue: new value to substitute in, or None to drop
the variable setting entirely. (If the removal
results in two consecutive blank lines, one of the
@@ -1095,6 +1095,8 @@ def edit_metadata(meta_lines, variables, varfunc, match_overrides=False):
multi-line value to continue on the same line as
the assignment, False to indent before the first
element.
+ To clarify, if you wish not to change the value, then you
+ would return like this: return origvalue, None, 0, True
match_overrides: True to match items with _overrides on the end,
False otherwise
Returns a tuple:
--
2.5.5
^ permalink raw reply related [flat|nested] 7+ messages in thread