* [PATCH 0/1] package_manager.py: fix for "Argument list too long"
@ 2016-10-13 9:45 Robert Yang
2016-10-13 9:45 ` [PATCH 1/1] " Robert Yang
0 siblings, 1 reply; 7+ messages in thread
From: Robert Yang @ 2016-10-13 9:45 UTC (permalink / raw)
To: openembedded-core
The following changes since commit 1a2311c8fa9a3703568cd390c44639fe3335023e:
build-appliance-image: Update to master head revision (2016-10-11 23:43:20 +0100)
are available in the git repository at:
git://git.openembedded.org/openembedded-core-contrib rbt/long
http://cgit.openembedded.org/cgit.cgi/openembedded-core-contrib/log/?h=rbt/long
Robert Yang (1):
package_manager.py: fix for "Argument list too long"
meta/lib/oe/package_manager.py | 6 +++++-
1 file changed, 5 insertions(+), 1 deletion(-)
--
2.9.0
^ permalink raw reply [flat|nested] 7+ messages in thread
* [PATCH 1/1] package_manager.py: fix for "Argument list too long"
2016-10-13 9:45 [PATCH 0/1] package_manager.py: fix for "Argument list too long" Robert Yang
@ 2016-10-13 9:45 ` Robert Yang
2016-10-13 10:39 ` Burton, Ross
0 siblings, 1 reply; 7+ messages in thread
From: Robert Yang @ 2016-10-13 9:45 UTC (permalink / raw)
To: openembedded-core
Fixed when len(TMPDIR) = 410:
$ bitbake core-image-sato-sdk
[snip]
Subprocess output:
/bin/sh: /bin/cp: Argument list too long
ERROR: core-image-sato-sdk-1.0-r0 do_rootfs: Function failed: do_rootfs
[snip]
This is because "copyhardlinktree(deploy_arch_dir, arch_channel)" does:
"cp -afl deploy_arch_dir/* arch_channel", while the deploy_arch_dir/* is
expanded to "deploy_arch_dir/pkg1 deploy_arch_dir/pkg2
deploy_arch_dir/pkg3 ..." which causes the "Argument list too long",
change cwd to deploy_arch_dir can avoid the error.
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
---
meta/lib/oe/package_manager.py | 6 +++++-
1 file changed, 5 insertions(+), 1 deletion(-)
diff --git a/meta/lib/oe/package_manager.py b/meta/lib/oe/package_manager.py
index 3cee973..e0e203b 100644
--- a/meta/lib/oe/package_manager.py
+++ b/meta/lib/oe/package_manager.py
@@ -1024,7 +1024,11 @@ class RpmPM(PackageManager):
lockfilename = self.d.getVar('DEPLOY_DIR_RPM', True) + "/rpm.lock"
lf = bb.utils.lockfile(lockfilename, False)
- oe.path.copyhardlinktree(deploy_arch_dir, arch_channel)
+ # chdir() to deploy_arch_dir to avoid "Argument list too long" error
+ oldcwd = os.getcwd()
+ os.chdir(deploy_arch_dir)
+ oe.path.copyhardlinktree('.', arch_channel)
+ os.chdir(oldcwd)
bb.utils.unlockfile(lf)
if not arch in ch_already_added:
--
2.9.0
^ permalink raw reply related [flat|nested] 7+ messages in thread
* Re: [PATCH 1/1] package_manager.py: fix for "Argument list too long"
2016-10-13 9:45 ` [PATCH 1/1] " Robert Yang
@ 2016-10-13 10:39 ` Burton, Ross
2016-10-14 9:58 ` Robert Yang
0 siblings, 1 reply; 7+ messages in thread
From: Burton, Ross @ 2016-10-13 10:39 UTC (permalink / raw)
To: Robert Yang; +Cc: OE-core
[-- Attachment #1: Type: text/plain, Size: 535 bytes --]
On 13 October 2016 at 10:45, Robert Yang <liezhi.yang@windriver.com> wrote:
> This is because "copyhardlinktree(deploy_arch_dir, arch_channel)" does:
> "cp -afl deploy_arch_dir/* arch_channel", while the deploy_arch_dir/* is
> expanded to "deploy_arch_dir/pkg1 deploy_arch_dir/pkg2
> deploy_arch_dir/pkg3 ..." which causes the "Argument list too long",
> change cwd to deploy_arch_dir can avoid the error.
>
Would it be better to change the implementation of copyhardlinktree so that
it *can't* have this problem?
Ross
[-- Attachment #2: Type: text/html, Size: 1003 bytes --]
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [PATCH 1/1] package_manager.py: fix for "Argument list too long"
2016-10-13 10:39 ` Burton, Ross
@ 2016-10-14 9:58 ` Robert Yang
2016-10-14 10:18 ` Robert Yang
0 siblings, 1 reply; 7+ messages in thread
From: Robert Yang @ 2016-10-14 9:58 UTC (permalink / raw)
To: Burton, Ross; +Cc: OE-core
On 10/13/2016 06:39 PM, Burton, Ross wrote:
>
> On 13 October 2016 at 10:45, Robert Yang <liezhi.yang@windriver.com
> <mailto:liezhi.yang@windriver.com>> wrote:
>
> This is because "copyhardlinktree(deploy_arch_dir, arch_channel)" does:
> "cp -afl deploy_arch_dir/* arch_channel", while the deploy_arch_dir/* is
> expanded to "deploy_arch_dir/pkg1 deploy_arch_dir/pkg2
> deploy_arch_dir/pkg3 ..." which causes the "Argument list too long",
> change cwd to deploy_arch_dir can avoid the error.
>
>
> Would it be better to change the implementation of copyhardlinktree so that it
> *can't* have this problem?
Good idea, thanks, updated in the repo:
git://git.openembedded.org/openembedded-core-contrib rbt/long
http://cgit.openembedded.org/cgit.cgi/openembedded-core-contrib/log/?h=rbt/long
Author: Robert Yang <liezhi.yang@windriver.com>
Date: Thu Oct 13 01:28:41 2016 -0700
oe/path.py: fix for "Argument list too long"
Fixed when len(TMPDIR) = 410:
$ bitbake core-image-sato-sdk
[snip]
Subprocess output:
/bin/sh: /bin/cp: Argument list too long
ERROR: core-image-sato-sdk-1.0-r0 do_rootfs: Function failed: do_rootfs
[snip]
This is because "copyhardlinktree(src, dst)" does "cp -afl src/* dst",
while src/* is expanded to "src/file1 src/file2, src/file3..." which
causes the "Argument list too long", change cwd to src can fix the
problem.
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
diff --git a/meta/lib/oe/path.py b/meta/lib/oe/path.py
index 06a5af2..13821b3 100644
--- a/meta/lib/oe/path.py
+++ b/meta/lib/oe/path.py
@@ -80,14 +80,18 @@ def copyhardlinktree(src, dst):
cmd = "cd %s; find . -type d -print | tar --xattrs
--xattrs-include='*' -cf - -C %s -p --no-recursion --files-from - | tar --xattrs
--xattrs-include='*' -xf - -C %s" % (src, src, dst)
subprocess.check_output(cmd, shell=True, stderr=subprocess.STDOUT)
source = ''
+ # chdir() to src to avoid "Argument list too long" error
+ oldcwd = os.getcwd()
if os.path.isdir(src):
+ os.chdir(src)
import glob
- if len(glob.glob('%s/.??*' % src)) > 0:
- source = '%s/.??* ' % src
- source = source + '%s/*' % src
+ if len(glob.glob('.??*')) > 0:
+ source = '.??* '
+ source += '*'
else:
source = src
cmd = 'cp -afl --preserve=xattr %s %s' % (source, dst)
+ os.chdir(oldcwd)
subprocess.check_output(cmd, shell=True, stderr=subprocess.STDOUT)
else:
copytree(src, dst)
// Robert
>
> Ross
^ permalink raw reply related [flat|nested] 7+ messages in thread
* Re: [PATCH 1/1] package_manager.py: fix for "Argument list too long"
2016-10-14 9:58 ` Robert Yang
@ 2016-10-14 10:18 ` Robert Yang
2016-10-16 4:26 ` Christopher Larson
0 siblings, 1 reply; 7+ messages in thread
From: Robert Yang @ 2016-10-14 10:18 UTC (permalink / raw)
To: Burton, Ross; +Cc: OE-core
On 10/14/2016 05:58 PM, Robert Yang wrote:
>
>
> On 10/13/2016 06:39 PM, Burton, Ross wrote:
>>
>> On 13 October 2016 at 10:45, Robert Yang <liezhi.yang@windriver.com
>> <mailto:liezhi.yang@windriver.com>> wrote:
>>
>> This is because "copyhardlinktree(deploy_arch_dir, arch_channel)" does:
>> "cp -afl deploy_arch_dir/* arch_channel", while the deploy_arch_dir/* is
>> expanded to "deploy_arch_dir/pkg1 deploy_arch_dir/pkg2
>> deploy_arch_dir/pkg3 ..." which causes the "Argument list too long",
>> change cwd to deploy_arch_dir can avoid the error.
>>
>>
>> Would it be better to change the implementation of copyhardlinktree so that it
>> *can't* have this problem?
>
> Good idea, thanks, updated in the repo:
>
> git://git.openembedded.org/openembedded-core-contrib rbt/long
> http://cgit.openembedded.org/cgit.cgi/openembedded-core-contrib/log/?h=rbt/long
>
Sorry, this patch is not what I wanted to paste, it should be:
diff --git a/meta/lib/oe/path.py b/meta/lib/oe/path.py
index 06a5af2..1d9cca5 100644
--- a/meta/lib/oe/path.py
+++ b/meta/lib/oe/path.py
@@ -80,15 +80,20 @@ def copyhardlinktree(src, dst):
cmd = "cd %s; find . -type d -print | tar --xattrs
--xattrs-include='*' -cf - -C %s -p --no-recursion --files-from - | tar --xattrs
--xattrs-include='*' -xf - -C %s" % (src, src, dst)
subprocess.check_output(cmd, shell=True, stderr=subprocess.STDOUT)
source = ''
+ oldcwd = os.getcwd()
if os.path.isdir(src):
+ dst = os.path.realpath(dst)
+ # chdir() to src to avoid "Argument list too long" error
+ os.chdir(src)
import glob
- if len(glob.glob('%s/.??*' % src)) > 0:
- source = '%s/.??* ' % src
- source = source + '%s/*' % src
+ if len(glob.glob('./.??*')) > 0:
+ source = './.??* '
+ source += './*'
else:
source = src
cmd = 'cp -afl --preserve=xattr %s %s' % (source, dst)
subprocess.check_output(cmd, shell=True, stderr=subprocess.STDOUT)
+ os.chdir(oldcwd)
else:
copytree(src, dst)
// Robert
>
> Author: Robert Yang <liezhi.yang@windriver.com>
> Date: Thu Oct 13 01:28:41 2016 -0700
>
> oe/path.py: fix for "Argument list too long"
>
> Fixed when len(TMPDIR) = 410:
> $ bitbake core-image-sato-sdk
> [snip]
> Subprocess output:
> /bin/sh: /bin/cp: Argument list too long
>
> ERROR: core-image-sato-sdk-1.0-r0 do_rootfs: Function failed: do_rootfs
> [snip]
>
> This is because "copyhardlinktree(src, dst)" does "cp -afl src/* dst",
> while src/* is expanded to "src/file1 src/file2, src/file3..." which
> causes the "Argument list too long", change cwd to src can fix the
> problem.
>
> Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
>
>
> diff --git a/meta/lib/oe/path.py b/meta/lib/oe/path.py
> index 06a5af2..13821b3 100644
> --- a/meta/lib/oe/path.py
> +++ b/meta/lib/oe/path.py
> @@ -80,14 +80,18 @@ def copyhardlinktree(src, dst):
> cmd = "cd %s; find . -type d -print | tar --xattrs --xattrs-include='*'
> -cf - -C %s -p --no-recursion --files-from - | tar --xattrs --xattrs-include='*'
> -xf - -C %s" % (src, src, dst)
> subprocess.check_output(cmd, shell=True, stderr=subprocess.STDOUT)
> source = ''
> + # chdir() to src to avoid "Argument list too long" error
> + oldcwd = os.getcwd()
> if os.path.isdir(src):
> + os.chdir(src)
> import glob
> - if len(glob.glob('%s/.??*' % src)) > 0:
> - source = '%s/.??* ' % src
> - source = source + '%s/*' % src
> + if len(glob.glob('.??*')) > 0:
> + source = '.??* '
> + source += '*'
> else:
> source = src
> cmd = 'cp -afl --preserve=xattr %s %s' % (source, dst)
> + os.chdir(oldcwd)
> subprocess.check_output(cmd, shell=True, stderr=subprocess.STDOUT)
> else:
> copytree(src, dst)
>
> // Robert
>
>>
>> Ross
^ permalink raw reply related [flat|nested] 7+ messages in thread
* Re: [PATCH 1/1] package_manager.py: fix for "Argument list too long"
2016-10-14 10:18 ` Robert Yang
@ 2016-10-16 4:26 ` Christopher Larson
2016-10-17 3:45 ` Robert Yang
0 siblings, 1 reply; 7+ messages in thread
From: Christopher Larson @ 2016-10-16 4:26 UTC (permalink / raw)
To: Robert Yang; +Cc: OE-core
[-- Attachment #1: Type: text/plain, Size: 2651 bytes --]
On Fri, Oct 14, 2016 at 3:18 AM, Robert Yang <liezhi.yang@windriver.com>
wrote:
> On 10/14/2016 05:58 PM, Robert Yang wrote:
>
>>
>>
>> On 10/13/2016 06:39 PM, Burton, Ross wrote:
>>
>>>
>>> On 13 October 2016 at 10:45, Robert Yang <liezhi.yang@windriver.com
>>> <mailto:liezhi.yang@windriver.com>> wrote:
>>>
>>> This is because "copyhardlinktree(deploy_arch_dir, arch_channel)"
>>> does:
>>> "cp -afl deploy_arch_dir/* arch_channel", while the
>>> deploy_arch_dir/* is
>>> expanded to "deploy_arch_dir/pkg1 deploy_arch_dir/pkg2
>>> deploy_arch_dir/pkg3 ..." which causes the "Argument list too long",
>>> change cwd to deploy_arch_dir can avoid the error.
>>>
>>>
>>> Would it be better to change the implementation of copyhardlinktree so
>>> that it
>>> *can't* have this problem?
>>>
>>
>> Good idea, thanks, updated in the repo:
>>
>> git://git.openembedded.org/openembedded-core-contrib rbt/long
>> http://cgit.openembedded.org/cgit.cgi/openembedded-core-cont
>> rib/log/?h=rbt/long
>>
>>
> Sorry, this patch is not what I wanted to paste, it should be:
>
> diff --git a/meta/lib/oe/path.py b/meta/lib/oe/path.py
> index 06a5af2..1d9cca5 100644
> --- a/meta/lib/oe/path.py
> +++ b/meta/lib/oe/path.py
> @@ -80,15 +80,20 @@ def copyhardlinktree(src, dst):
> cmd = "cd %s; find . -type d -print | tar --xattrs
> --xattrs-include='*' -cf - -C %s -p --no-recursion --files-from - | tar
> --xattrs --xattrs-include='*' -xf - -C %s" % (src, src, dst)
> subprocess.check_output(cmd, shell=True, stderr=subprocess.STDOUT)
> source = ''
> + oldcwd = os.getcwd()
> if os.path.isdir(src):
> + dst = os.path.realpath(dst)
> + # chdir() to src to avoid "Argument list too long" error
> + os.chdir(src)
> import glob
> - if len(glob.glob('%s/.??*' % src)) > 0:
> - source = '%s/.??* ' % src
> - source = source + '%s/*' % src
> + if len(glob.glob('./.??*')) > 0:
> + source = './.??* '
> + source += './*'
> else:
> source = src
> cmd = 'cp -afl --preserve=xattr %s %s' % (source, dst)
> subprocess.check_output(cmd, shell=True, stderr=subprocess.STDOUT)
> + os.chdir(oldcwd)
Question, why not just pass cwd=src in the check_output() call, rather than
changing it in the parent python process?
--
Christopher Larson
clarson at kergoth dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Maintainer - Tslib
Senior Software Engineer, Mentor Graphics
[-- Attachment #2: Type: text/html, Size: 4096 bytes --]
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [PATCH 1/1] package_manager.py: fix for "Argument list too long"
2016-10-16 4:26 ` Christopher Larson
@ 2016-10-17 3:45 ` Robert Yang
0 siblings, 0 replies; 7+ messages in thread
From: Robert Yang @ 2016-10-17 3:45 UTC (permalink / raw)
To: Christopher Larson; +Cc: OE-core
Hi Christopher,
On 10/16/2016 12:26 PM, Christopher Larson wrote:
>
> On Fri, Oct 14, 2016 at 3:18 AM, Robert Yang <liezhi.yang@windriver.com
> <mailto:liezhi.yang@windriver.com>> wrote:
>
> On 10/14/2016 05:58 PM, Robert Yang wrote:
>
>
>
> On 10/13/2016 06:39 PM, Burton, Ross wrote:
>
>
> On 13 October 2016 at 10:45, Robert Yang <liezhi.yang@windriver.com
> <mailto:liezhi.yang@windriver.com>
> <mailto:liezhi.yang@windriver.com
> <mailto:liezhi.yang@windriver.com>>> wrote:
>
> This is because "copyhardlinktree(deploy_arch_dir,
> arch_channel)" does:
> "cp -afl deploy_arch_dir/* arch_channel", while the
> deploy_arch_dir/* is
> expanded to "deploy_arch_dir/pkg1 deploy_arch_dir/pkg2
> deploy_arch_dir/pkg3 ..." which causes the "Argument list too long",
> change cwd to deploy_arch_dir can avoid the error.
>
>
> Would it be better to change the implementation of copyhardlinktree
> so that it
> *can't* have this problem?
>
>
> Good idea, thanks, updated in the repo:
>
> git://git.openembedded.org/openembedded-core-contrib
> <http://git.openembedded.org/openembedded-core-contrib> rbt/long
>
> http://cgit.openembedded.org/cgit.cgi/openembedded-core-contrib/log/?h=rbt/long
> <http://cgit.openembedded.org/cgit.cgi/openembedded-core-contrib/log/?h=rbt/long>
>
>
> Sorry, this patch is not what I wanted to paste, it should be:
>
> diff --git a/meta/lib/oe/path.py b/meta/lib/oe/path.py
> index 06a5af2..1d9cca5 100644
> --- a/meta/lib/oe/path.py
> +++ b/meta/lib/oe/path.py
> @@ -80,15 +80,20 @@ def copyhardlinktree(src, dst):
> cmd = "cd %s; find . -type d -print | tar --xattrs
> --xattrs-include='*' -cf - -C %s -p --no-recursion --files-from - | tar
> --xattrs --xattrs-include='*' -xf - -C %s" % (src, src, dst)
> subprocess.check_output(cmd, shell=True, stderr=subprocess.STDOUT)
> source = ''
> + oldcwd = os.getcwd()
> if os.path.isdir(src):
> + dst = os.path.realpath(dst)
> + # chdir() to src to avoid "Argument list too long" error
> + os.chdir(src)
> import glob
> - if len(glob.glob('%s/.??*' % src)) > 0:
> - source = '%s/.??* ' % src
> - source = source + '%s/*' % src
> + if len(glob.glob('./.??*')) > 0:
> + source = './.??* '
> + source += './*'
> else:
> source = src
> cmd = 'cp -afl --preserve=xattr %s %s' % (source, dst)
> subprocess.check_output(cmd, shell=True, stderr=subprocess.STDOUT)
> + os.chdir(oldcwd)
>
>
> Question, why not just pass cwd=src in the check_output() call, rather than
> changing it in the parent python process?
src can be a file or directory, so we can't simply use cwd=src, I will use
os.getcwd() as cwd when it is not a directory. Updated patch in the repo.
git://git.openembedded.org/openembedded-core-contrib rbt/long
Author: Robert Yang <liezhi.yang@windriver.com>
Date: Thu Oct 13 01:28:41 2016 -0700
oe/path.py: fix for "Argument list too long"
Fixed when len(TMPDIR) = 410:
$ bitbake core-image-sato-sdk
[snip]
Subprocess output:
/bin/sh: /bin/cp: Argument list too long
ERROR: core-image-sato-sdk-1.0-r0 do_rootfs: Function failed: do_rootfs
[snip]
This is because "copyhardlinktree(src, dst)" does "cp -afl src/* dst",
while src/* is expanded to "src/file1 src/file2, src/file3..." which
causes the "Argument list too long", use ./* as src and change cwd in
subprocess.check_output() to fix the problem.
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
diff --git a/meta/lib/oe/path.py b/meta/lib/oe/path.py
index 06a5af2..ed7fd1e 100644
--- a/meta/lib/oe/path.py
+++ b/meta/lib/oe/path.py
@@ -83,12 +83,14 @@ def copyhardlinktree(src, dst):
if os.path.isdir(src):
import glob
if len(glob.glob('%s/.??*' % src)) > 0:
- source = '%s/.??* ' % src
- source = source + '%s/*' % src
+ source = './.??* '
+ source += './*'
+ s_dir = src
else:
source = src
- cmd = 'cp -afl --preserve=xattr %s %s' % (source, dst)
- subprocess.check_output(cmd, shell=True, stderr=subprocess.STDOUT)
+ s_dir = os.getcwd()
+ cmd = 'cp -afl --preserve=xattr %s %s' % (source, os.path.realpath(dst))
+ subprocess.check_output(cmd, shell=True, cwd=s_dir,
stderr=subprocess.STDOUT)
else:
copytree(src, dst)
// Robert
> --
> Christopher Larson
> clarson at kergoth dot com
> Founder - BitBake, OpenEmbedded, OpenZaurus
> Maintainer - Tslib
> Senior Software Engineer, Mentor Graphics
^ permalink raw reply related [flat|nested] 7+ messages in thread
end of thread, other threads:[~2016-10-17 3:45 UTC | newest]
Thread overview: 7+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2016-10-13 9:45 [PATCH 0/1] package_manager.py: fix for "Argument list too long" Robert Yang
2016-10-13 9:45 ` [PATCH 1/1] " Robert Yang
2016-10-13 10:39 ` Burton, Ross
2016-10-14 9:58 ` Robert Yang
2016-10-14 10:18 ` Robert Yang
2016-10-16 4:26 ` Christopher Larson
2016-10-17 3:45 ` Robert Yang
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.