[PATCH] devtool: prevent starting git gc in background

[PATCH] devtool: prevent starting git gc in background

[Taras Kondratiuk]

Devtool creates a git repository for extracted sources in a temporary
directory and then moves it to a final destination after patching is
done. Unfortunately devtool is not aware that some of its git operations
may have caused git garbage collector to start in background. If timing
is just right a repository move fails because GC is removing git objects
while they are being moved.

The issue was hit on Krogoth release, but the code that moves created
git repository is still the same.

Disable gc.autodetach to make GC run synchronously and block git until
it is done.

Traceback (most recent call last):
  File "<snip>/openembedded-core/scripts/devtool", line 342, in <module>
    ret = main()
  File "<snip>/openembedded-core/scripts/devtool", line 329, in main
    ret = args.func(args, config, basepath, workspace)
  File "<snip>/openembedded-core/scripts/lib/devtool/standard.py", line 352, in extract
    initial_rev = _extract_source(srctree, args.keep_temp, args.branch, False, rd)
  File "<snip>/openembedded-core/scripts/lib/devtool/standard.py", line 644, in _extract_source
    shutil.move(srcsubdir, srctree)
  File "/usr/lib/python2.7/shutil.py", line 299, in move
    copytree(src, real_dst, symlinks=True)
  File "/usr/lib/python2.7/shutil.py", line 208, in copytree
    raise Error, errors
Error: [('/tmp/devtool5RXkuX/workdir/grpc-1.2.5/.git/objects/5e', '<snip>/build/grpc/grpc_src/.git/objects/5e', "[Errno 2] No such file or directory: '/tmp/devtool5RXkuX/workdir/grpc-1.2.5/.git/objects/5e'"), ('/tmp/devtool5RXkuX/workdir/grpc-1.2.5/.git/objects/57', '<snip>/build/grpc/grpc_src/.git/objects/57', "[Errno 2] No such file or directory: '/tmp/devtool5RXkuX/workdir/grpc-1.2.5/.git/objects/57'"), many git objects ... ]

Signed-off-by: Taras Kondratiuk <takondra@cisco.com>
---
 scripts/lib/devtool/__init__.py | 1 +
 1 file changed, 1 insertion(+)

diff --git a/scripts/lib/devtool/__init__.py b/scripts/lib/devtool/__init__.py
index 89f098a912a6..7aff31231a2d 100644
--- a/scripts/lib/devtool/__init__.py
+++ b/scripts/lib/devtool/__init__.py
@@ -205,6 +205,7 @@ def setup_git_repo(repodir, version, devbranch, basetag='devtool-base', d=None):
     import oe.patch
     if not os.path.exists(os.path.join(repodir, '.git')):
         bb.process.run('git init', cwd=repodir)
+        bb.process.run('git config gc.autodetach 0', cwd=repodir)
         bb.process.run('git add .', cwd=repodir)
         commit_cmd = ['git']
         oe.patch.GitApplyTree.gitCommandUserOptions(commit_cmd, d=d)
-- 
2.7.4

Re: [PATCH] devtool: prevent starting git gc in background

[Paul Eggleton]

Hi Taras

On Friday, 22 March 2019 12:08:25 PM NZDT Taras Kondratiuk wrote:
> Devtool creates a git repository for extracted sources in a temporary
> directory and then moves it to a final destination after patching is
> done. Unfortunately devtool is not aware that some of its git operations
> may have caused git garbage collector to start in background. If timing
> is just right a repository move fails because GC is removing git objects
> while they are being moved.
> 
> The issue was hit on Krogoth release, but the code that moves created
> git repository is still the same.
> 
> Disable gc.autodetach to make GC run synchronously and block git until
> it is done.

The fix is OK, but I'd like it to be explicit and say "git config --local"
rather than just "git config" just in case of any future behavioral
changes in git config. Could you please send a v2 with this change?

Thanks,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre

[PATCH v2] devtool: prevent starting git gc in background

[Taras Kondratiuk]

Devtool creates a git repository for extracted sources in a temporary
directory and then moves it to a final destination after patching is
done. Unfortunately devtool is not aware that some of its git operations
may have caused git garbage collector to start in background. If timing
is just right a repository move fails because GC is removing git objects
while they are being moved.

The issue was hit on Krogoth release, but the code that moves created
git repository is still the same.

Disable gc.autodetach to make GC run synchronously and block git until
it is done.

Traceback (most recent call last):
  File "<snip>/openembedded-core/scripts/devtool", line 342, in <module>
    ret = main()
  File "<snip>/openembedded-core/scripts/devtool", line 329, in main
    ret = args.func(args, config, basepath, workspace)
  File "<snip>/openembedded-core/scripts/lib/devtool/standard.py", line 352, in extract
    initial_rev = _extract_source(srctree, args.keep_temp, args.branch, False, rd)
  File "<snip>/openembedded-core/scripts/lib/devtool/standard.py", line 644, in _extract_source
    shutil.move(srcsubdir, srctree)
  File "/usr/lib/python2.7/shutil.py", line 299, in move
    copytree(src, real_dst, symlinks=True)
  File "/usr/lib/python2.7/shutil.py", line 208, in copytree
    raise Error, errors
Error: [('/tmp/devtool5RXkuX/workdir/grpc-1.2.5/.git/objects/5e', '<snip>/build/grpc/grpc_src/.git/objects/5e', "[Errno 2] No such file or directory: '/tmp/devtool5RXkuX/workdir/grpc-1.2.5/.git/objects/5e'"), ('/tmp/devtool5RXkuX/workdir/grpc-1.2.5/.git/objects/57', '<snip>/build/grpc/grpc_src/.git/objects/57', "[Errno 2] No such file or directory: '/tmp/devtool5RXkuX/workdir/grpc-1.2.5/.git/objects/57'"), many git objects ... ]

Signed-off-by: Taras Kondratiuk <takondra@cisco.com>
---
v2: Added "--local".

 scripts/lib/devtool/__init__.py | 1 +
 1 file changed, 1 insertion(+)

diff --git a/scripts/lib/devtool/__init__.py b/scripts/lib/devtool/__init__.py
index 89f098a912a6..8fc7fffcd672 100644
--- a/scripts/lib/devtool/__init__.py
+++ b/scripts/lib/devtool/__init__.py
@@ -205,6 +205,7 @@ def setup_git_repo(repodir, version, devbranch, basetag='devtool-base', d=None):
     import oe.patch
     if not os.path.exists(os.path.join(repodir, '.git')):
         bb.process.run('git init', cwd=repodir)
+        bb.process.run('git config --local gc.autodetach 0', cwd=repodir)
         bb.process.run('git add .', cwd=repodir)
         commit_cmd = ['git']
         oe.patch.GitApplyTree.gitCommandUserOptions(commit_cmd, d=d)
-- 
2.7.4

Re: [PATCH v2] devtool: prevent starting git gc in background

[Paul Eggleton]

Acked-by: Paul Eggleton <paul.eggleton@linux.intel.com>

On Friday, 22 March 2019 1:56:59 PM NZDT Taras Kondratiuk via Openembedded-core wrote:
> Devtool creates a git repository for extracted sources in a temporary
> directory and then moves it to a final destination after patching is
> done. Unfortunately devtool is not aware that some of its git operations
> may have caused git garbage collector to start in background. If timing
> is just right a repository move fails because GC is removing git objects
> while they are being moved.
> 
> The issue was hit on Krogoth release, but the code that moves created
> git repository is still the same.
> 
> Disable gc.autodetach to make GC run synchronously and block git until
> it is done.
> 
> Traceback (most recent call last):
>   File "<snip>/openembedded-core/scripts/devtool", line 342, in <module>
>     ret = main()
>   File "<snip>/openembedded-core/scripts/devtool", line 329, in main
>     ret = args.func(args, config, basepath, workspace)
>   File "<snip>/openembedded-core/scripts/lib/devtool/standard.py", line 352, in extract
>     initial_rev = _extract_source(srctree, args.keep_temp, args.branch, False, rd)
>   File "<snip>/openembedded-core/scripts/lib/devtool/standard.py", line 644, in _extract_source
>     shutil.move(srcsubdir, srctree)
>   File "/usr/lib/python2.7/shutil.py", line 299, in move
>     copytree(src, real_dst, symlinks=True)
>   File "/usr/lib/python2.7/shutil.py", line 208, in copytree
>     raise Error, errors
> Error: [('/tmp/devtool5RXkuX/workdir/grpc-1.2.5/.git/objects/5e', '<snip>/build/grpc/grpc_src/.git/objects/5e', "[Errno 2] No such file or directory: '/tmp/devtool5RXkuX/workdir/grpc-1.2.5/.git/objects/5e'"), ('/tmp/devtool5RXkuX/workdir/grpc-1.2.5/.git/objects/57', '<snip>/build/grpc/grpc_src/.git/objects/57', "[Errno 2] No such file or directory: '/tmp/devtool5RXkuX/workdir/grpc-1.2.5/.git/objects/57'"), many git objects ... ]
> 
> Signed-off-by: Taras Kondratiuk <takondra@cisco.com>
> ---
> v2: Added "--local".
> 
>  scripts/lib/devtool/__init__.py | 1 +
>  1 file changed, 1 insertion(+)
> 
> diff --git a/scripts/lib/devtool/__init__.py b/scripts/lib/devtool/__init__.py
> index 89f098a912a6..8fc7fffcd672 100644
> --- a/scripts/lib/devtool/__init__.py
> +++ b/scripts/lib/devtool/__init__.py
> @@ -205,6 +205,7 @@ def setup_git_repo(repodir, version, devbranch, basetag='devtool-base', d=None):
>      import oe.patch
>      if not os.path.exists(os.path.join(repodir, '.git')):
>          bb.process.run('git init', cwd=repodir)
> +        bb.process.run('git config --local gc.autodetach 0', cwd=repodir)
>          bb.process.run('git add .', cwd=repodir)
>          commit_cmd = ['git']
>          oe.patch.GitApplyTree.gitCommandUserOptions(commit_cmd, d=d)
> 


-- 

Paul Eggleton
Intel Open Source Technology Centre