* [PATCH-v2] Allow git-filter-branch to process large repositories with lots of branches.
@ 2013-09-07 21:03 Lee Carver
2013-09-07 23:06 ` Stefano Lattarini
0 siblings, 1 reply; 7+ messages in thread
From: Lee Carver @ 2013-09-07 21:03 UTC (permalink / raw)
To: Andreas Schwab, gitster, git; +Cc: Lee Carver
As noted in several forums, a recommended way to move trees between
repositories
is to use git-filter-branch to revise the history for a single tree:
http://gbayer.com/development/moving-files-from-one-git-repository-to-anoth
er-preserving-history/
http://stackoverflow.com/questions/1365541/how-to-move-files-from-one-git-r
epo-to-another-not-a-clone-preserving-history
However, this can lead to argument list too long errors when the original
repository has many retained branches (>6k)
/usr/local/git/libexec/git-core/git-filter-branch: line 270:
/usr/local/git/libexec/git-core/git: Argument list too long
Could not get the commits
Piping the saved output from git rev-parse into git rev-list avoids this
problem, since the rev-parse output is not processed as a command line
argument.
---
git-filter-branch.sh | 5 +++--
1 file changed, 3 insertions(+), 2 deletions(-)
diff --git a/git-filter-branch.sh b/git-filter-branch.sh
index ac2a005..60d239b 100755
--- a/git-filter-branch.sh
+++ b/git-filter-branch.sh
@@ -255,7 +255,7 @@ else
remap_to_ancestor=t
fi
-rev_args=$(git rev-parse --revs-only "$@")
+git rev-parse --revs-only "$@" > ../parse
case "$filter_subdir" in
"")
@@ -267,8 +267,9 @@ case "$filter_subdir" in
;;
esac
+cat ../parse | \
git rev-list --reverse --topo-order --default HEAD \
- --parents --simplify-merges $rev_args "$@" > ../revs ||
+ --parents --simplify-merges --stdin "$@" > ../revs ||
die "Could not get the commits"
commits=$(wc -l <../revs | tr -d " ")
--
1.8.3.2
^ permalink raw reply related [flat|nested] 7+ messages in thread
* Re: [PATCH-v2] Allow git-filter-branch to process large repositories with lots of branches.
2013-09-07 21:03 [PATCH-v2] Allow git-filter-branch to process large repositories with lots of branches Lee Carver
@ 2013-09-07 23:06 ` Stefano Lattarini
2013-09-10 22:55 ` [PATCH-v3] " Lee Carver
0 siblings, 1 reply; 7+ messages in thread
From: Stefano Lattarini @ 2013-09-07 23:06 UTC (permalink / raw)
To: Lee Carver; +Cc: Andreas Schwab, gitster, git, Lee Carver
On 07/09/13 22:03, Lee Carver wrote:
> As noted in several forums, a recommended way to move trees between
> repositories
> is to use git-filter-branch to revise the history for a single tree:
>
> http://gbayer.com/development/moving-files-from-one-git-repository-to-anoth
> er-preserving-history/
> http://stackoverflow.com/questions/1365541/how-to-move-files-from-one-git-r
> epo-to-another-not-a-clone-preserving-history
>
> However, this can lead to argument list too long errors when the original
> repository has many retained branches (>6k)
>
> /usr/local/git/libexec/git-core/git-filter-branch: line 270:
> /usr/local/git/libexec/git-core/git: Argument list too long
> Could not get the commits
>
> Piping the saved output from git rev-parse into git rev-list avoids this
> problem, since the rev-parse output is not processed as a command line
> argument.
> ---
> git-filter-branch.sh | 5 +++--
> 1 file changed, 3 insertions(+), 2 deletions(-)
>
> diff --git a/git-filter-branch.sh b/git-filter-branch.sh
> index ac2a005..60d239b 100755
> --- a/git-filter-branch.sh
> +++ b/git-filter-branch.sh
> @@ -255,7 +255,7 @@ else
> remap_to_ancestor=t
> fi
>
> -rev_args=$(git rev-parse --revs-only "$@")
> +git rev-parse --revs-only "$@" > ../parse
>
> case "$filter_subdir" in
> "")
> @@ -267,8 +267,9 @@ case "$filter_subdir" in
> ;;
> esac
>
> +cat ../parse | \
> git rev-list --reverse --topo-order --default HEAD \
> - --parents --simplify-merges $rev_args "$@" > ../revs ||
> + --parents --simplify-merges --stdin "$@" > ../revs ||
>
Useless use of cat IMO. I'd suggest using a redirection instead:
git rev-list --reverse --topo-order --default HEAD \
- --parents --simplify-merges $rev_args "$@" > ../revs ||
+ --parents --simplify-merges --stdin "$@" > ../revs < ../parse ||
> die "Could not get the commits"
> commits=$(wc -l <../revs | tr -d " ")
>
>
Regards,
Stefano
^ permalink raw reply [flat|nested] 7+ messages in thread
* [PATCH-v3] Allow git-filter-branch to process large repositories with lots of branches.
2013-09-07 23:06 ` Stefano Lattarini
@ 2013-09-10 22:55 ` Lee Carver
2013-09-10 23:20 ` Junio C Hamano
0 siblings, 1 reply; 7+ messages in thread
From: Lee Carver @ 2013-09-10 22:55 UTC (permalink / raw)
To: git, gitster; +Cc: Andreas Schwab, Lee Carver, Stefano Lattarini
As noted in several forums, a recommended way to move trees between
repositories
is to use git-filter-branch to revise the history for a single tree:
http://gbayer.com/development/moving-files-from-one-git-repository-to-anoth
er-preserving-history/
http://stackoverflow.com/questions/1365541/how-to-move-files-from-one-git-r
epo-to-another-not-a-clone-preserving-history
However, this can lead to argument list too long errors when the original
repository has many retained branches (>6k)
/usr/local/git/libexec/git-core/git-filter-branch: line 270:
/usr/local/git/libexec/git-core/git: Argument list too long
Could not get the commits
Piping the saved output from git rev-parse into git rev-list avoids this
problem, since the rev-parse output is not processed as a command line
argument.
---
git-filter-branch.sh | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/git-filter-branch.sh b/git-filter-branch.sh
index ac2a005..2091885 100755
--- a/git-filter-branch.shgit
+++ b/git-filter-branch.sh
@@ -255,7 +255,7 @@ else
remap_to_ancestor=t
fi
-rev_args=$(git rev-parse --revs-only "$@")
+git rev-parse --revs-only "$@" > ../parse
case "$filter_subdir" in
"")
@@ -268,7 +268,7 @@ case "$filter_subdir" in
esac
git rev-list --reverse --topo-order --default HEAD \
- --parents --simplify-merges $rev_args "$@" > ../revs ||
+ --parents --simplify-merges --stdin "$@" < ../parse > ../revs ||
die "Could not get the commits"
commits=$(wc -l <../revs | tr -d " ")
--
1.8.3.2
^ permalink raw reply related [flat|nested] 7+ messages in thread
* Re: [PATCH-v3] Allow git-filter-branch to process large repositories with lots of branches.
2013-09-10 22:55 ` [PATCH-v3] " Lee Carver
@ 2013-09-10 23:20 ` Junio C Hamano
2013-09-11 0:21 ` Lee Carver
0 siblings, 1 reply; 7+ messages in thread
From: Junio C Hamano @ 2013-09-10 23:20 UTC (permalink / raw)
To: Lee Carver; +Cc: git, Andreas Schwab, Lee Carver, Stefano Lattarini
Lee Carver <Lee.Carver@servicenow.com> writes:
> As noted in several forums, a recommended way to move trees between
> repositories
> is to use git-filter-branch to revise the history for a single tree:
>
> http://gbayer.com/development/moving-files-from-one-git-repository-to-anoth
> er-preserving-history/
> http://stackoverflow.com/questions/1365541/how-to-move-files-from-one-git-r
> epo-to-another-not-a-clone-preserving-history
>
> However, this can lead to argument list too long errors when the original
> repository has many retained branches (>6k)
>
> /usr/local/git/libexec/git-core/git-filter-branch: line 270:
> /usr/local/git/libexec/git-core/git: Argument list too long
> Could not get the commits
>
> Piping the saved output from git rev-parse into git rev-list avoids this
> problem, since the rev-parse output is not processed as a command line
> argument.
> ---
> git-filter-branch.sh | 4 ++--
> 1 file changed, 2 insertions(+), 2 deletions(-)
>
> diff --git a/git-filter-branch.sh b/git-filter-branch.sh
> index ac2a005..2091885 100755
> --- a/git-filter-branch.shgit
> +++ b/git-filter-branch.sh
> @@ -255,7 +255,7 @@ else
> remap_to_ancestor=t
> fi
>
> -rev_args=$(git rev-parse --revs-only "$@")
> +git rev-parse --revs-only "$@" > ../parse
Where is this "rev-parse" command running? Is it always safe to
clobber the file "../parse" like this?
>
> case "$filter_subdir" in
> "")
> @@ -268,7 +268,7 @@ case "$filter_subdir" in
> esac
>
> git rev-list --reverse --topo-order --default HEAD \
> - --parents --simplify-merges $rev_args "$@" > ../revs ||
> + --parents --simplify-merges --stdin "$@" < ../parse > ../revs ||
> die "Could not get the commits"
> commits=$(wc -l <../revs | tr -d " ")
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [PATCH-v3] Allow git-filter-branch to process large repositories with lots of branches.
2013-09-10 23:20 ` Junio C Hamano
@ 2013-09-11 0:21 ` Lee Carver
2013-09-11 16:06 ` Junio C Hamano
0 siblings, 1 reply; 7+ messages in thread
From: Lee Carver @ 2013-09-11 0:21 UTC (permalink / raw)
To: Junio C Hamano; +Cc: git, Andreas Schwab, Lee Carver, Stefano Lattarini
On 9/10/13 4:20 PM, "Junio C Hamano" <gitster@pobox.com> wrote:
>Lee Carver <Lee.Carver@servicenow.com> writes:
>
>> As noted in several forums, a recommended way to move trees between
>> repositories
>> is to use git-filter-branch to revise the history for a single tree:
>>
>>
>>http://gbayer.com/development/moving-files-from-one-git-repository-to-ano
>>th
>> er-preserving-history/
>>
>>http://stackoverflow.com/questions/1365541/how-to-move-files-from-one-git
>>-r
>> epo-to-another-not-a-clone-preserving-history
>>
>> However, this can lead to argument list too long errors when the
>>original
>> repository has many retained branches (>6k)
>>
>> /usr/local/git/libexec/git-core/git-filter-branch: line 270:
>> /usr/local/git/libexec/git-core/git: Argument list too long
>> Could not get the commits
>>
>> Piping the saved output from git rev-parse into git rev-list avoids this
>> problem, since the rev-parse output is not processed as a command line
>> argument.
>> ---
>> git-filter-branch.sh | 4 ++--
>> 1 file changed, 2 insertions(+), 2 deletions(-)
>>
>> diff --git a/git-filter-branch.sh b/git-filter-branch.sh
>> index ac2a005..2091885 100755
>> --- a/git-filter-branch.shgit
>> +++ b/git-filter-branch.sh
>> @@ -255,7 +255,7 @@ else
>> remap_to_ancestor=t
>> fi
>>
>> -rev_args=$(git rev-parse --revs-only "$@")
>> +git rev-parse --revs-only "$@" > ../parse
>
>Where is this "rev-parse" command running? Is it always safe to
>clobber the file "../parse" like this?
It is using the same ${tempdir} working directory that git rev-list uses
below for the ../revs file
It's normally .git-rewrite/t, following the normal working directory setup
near line 205.
>
>>
>> case "$filter_subdir" in
>> "")
>> @@ -268,7 +268,7 @@ case "$filter_subdir" in
>> esac
>>
>> git rev-list --reverse --topo-order --default HEAD \
>> - --parents --simplify-merges $rev_args "$@" > ../revs ||
>> + --parents --simplify-merges --stdin "$@" < ../parse > ../revs ||
>> die "Could not get the commits"
>> commits=$(wc -l <../revs | tr -d " ")
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [PATCH-v3] Allow git-filter-branch to process large repositories with lots of branches.
2013-09-11 0:21 ` Lee Carver
@ 2013-09-11 16:06 ` Junio C Hamano
2013-09-12 17:46 ` [PATCH-v4] " Lee Carver
0 siblings, 1 reply; 7+ messages in thread
From: Junio C Hamano @ 2013-09-11 16:06 UTC (permalink / raw)
To: Lee Carver; +Cc: git, Andreas Schwab, Lee Carver, Stefano Lattarini
Lee Carver <Lee.Carver@servicenow.com> writes:
> It is using the same ${tempdir} working directory that git rev-list uses
> below for the ../revs file
Ah, I missed that; then that should be safe. The patch looks sane.
Can we have your sign-off, too, please?
>
> It's normally .git-rewrite/t, following the normal working directory setup
> near line 205.
>
>>
>>>
>>> case "$filter_subdir" in
>>> "")
>>> @@ -268,7 +268,7 @@ case "$filter_subdir" in
>>> esac
>>>
>>> git rev-list --reverse --topo-order --default HEAD \
>>> - --parents --simplify-merges $rev_args "$@" > ../revs ||
>>> + --parents --simplify-merges --stdin "$@" < ../parse > ../revs ||
>>> die "Could not get the commits"
>>> commits=$(wc -l <../revs | tr -d " ")
^ permalink raw reply [flat|nested] 7+ messages in thread
* [PATCH-v4] Allow git-filter-branch to process large repositories with lots of branches.
2013-09-11 16:06 ` Junio C Hamano
@ 2013-09-12 17:46 ` Lee Carver
0 siblings, 0 replies; 7+ messages in thread
From: Lee Carver @ 2013-09-12 17:46 UTC (permalink / raw)
To: git, gitster
Cc: Andreas Schwab, Lee Carver, Stefano Lattarini, Junio C Hamano
As noted in several forums, a recommended way to move trees between
repositories
is to use git-filter-branch to revise the history for a single tree:
http://gbayer.com/development/moving-files-from-one-git-repository-to-anoth
er-preserving-history/
http://stackoverflow.com/questions/1365541/how-to-move-files-from-one-git-r
epo-to-another-not-a-clone-preserving-history
However, this can lead to argument list too long errors when the original
repository has many retained branches (>6k)
/usr/local/git/libexec/git-core/git-filter-branch: line 270:
/usr/local/git/libexec/git-core/git: Argument list too long
Could not get the commits
Piping the saved output from git rev-parse into git rev-list avoids this
problem, since the rev-parse output is not processed as a command line
argument.
Signed-off-by: Lee Carver <lee.carver@servicenow.com>
---
git-filter-branch.sh | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/git-filter-branch.sh b/git-filter-branch.sh
index ac2a005..2091885 100755
--- a/git-filter-branch.sh
+++ b/git-filter-branch.sh
@@ -255,7 +255,7 @@ else
remap_to_ancestor=t
fi
-rev_args=$(git rev-parse --revs-only "$@")
+git rev-parse --revs-only "$@" > ../parse
case "$filter_subdir" in
"")
@@ -268,7 +268,7 @@ case "$filter_subdir" in
esac
git rev-list --reverse --topo-order --default HEAD \
- --parents --simplify-merges $rev_args "$@" > ../revs ||
+ --parents --simplify-merges --stdin "$@" < ../parse > ../revs ||
die "Could not get the commits"
commits=$(wc -l <../revs | tr -d " ")
--
1.8.3.2
^ permalink raw reply related [flat|nested] 7+ messages in thread
end of thread, other threads:[~2013-09-12 17:46 UTC | newest]
Thread overview: 7+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2013-09-07 21:03 [PATCH-v2] Allow git-filter-branch to process large repositories with lots of branches Lee Carver
2013-09-07 23:06 ` Stefano Lattarini
2013-09-10 22:55 ` [PATCH-v3] " Lee Carver
2013-09-10 23:20 ` Junio C Hamano
2013-09-11 0:21 ` Lee Carver
2013-09-11 16:06 ` Junio C Hamano
2013-09-12 17:46 ` [PATCH-v4] " Lee Carver
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.