Fwd: git p4: feature request - branch check filtering

Fwd: git p4: feature request - branch check filtering

[Dan Porter]

Hi,

I'm unable to find a similar issue, and if it's raised on the mailing
list I apologize.

I work at a company that has recently moved all CVS, SVN, and git
repositories to Perforce.  Depots have not been setup correctly in
every case, and there is one depot that contains literally hundreds of
projects under commercial development (and hundreds of branches as a
result)

My project may be in //stupid_depot/commercial/teamporter/rok.  This
is the path I clone with git-p4.  The only branches in this depot that
contain files at this path are titled as
'rok_porter_branch/release_1.x' or similar.

When using '--detect-branches' git-p4 checks each key of branches to
see if any of them have files in the path I've cloned.  Whilst this is
good in practice there is unfortunately 6,809 branches, git-p4
processes about 2 a second and just under an hour to perform any
git-p4 rebase, submit, or similar operation.

I propose the addition of a branch list filtering option
(--filter-branches) that takes either a regular expression or list of
branches it should check.  This may be useful in sane situations where
you don't want to scan every branch in a Perforce repository, or
blacklist branches that have undesirable content (for example, one of
the branches is called 'svn-backup'.  It contains a single, multi-GB
tarball.)

It would be ideal to have this information (after initial clone or
sync) stored somewhere in the git config where is appropriate so that
future submit/rebase operations adhere to this list.

Has something like this been worked on, or has been considered in the
past?  If not I will consider implementing this after reading up on
the Git code guidelines.

Thanks for keeping the Git workflow accessible in painful areas.

Dan

Re: Fwd: git p4: feature request - branch check filtering

[Pete Wyckoff]

dpreid@gmail.com wrote on Tue, 18 Feb 2014 12:42 +0000:
> I work at a company that has recently moved all CVS, SVN, and git
> repositories to Perforce.  Depots have not been setup correctly in
> every case, and there is one depot that contains literally hundreds of
> projects under commercial development (and hundreds of branches as a
> result)

My condolences.

> My project may be in //stupid_depot/commercial/teamporter/rok.  This
> is the path I clone with git-p4.  The only branches in this depot that
> contain files at this path are titled as
> 'rok_porter_branch/release_1.x' or similar.
> 
> When using '--detect-branches' git-p4 checks each key of branches to
> see if any of them have files in the path I've cloned.  Whilst this is
> good in practice there is unfortunately 6,809 branches, git-p4
> processes about 2 a second and just under an hour to perform any
> git-p4 rebase, submit, or similar operation.

This is in getBranchMapping() presumably.  Where it loops
over each branch doing "p4 branch -o".  Yuk.

You could always avoid the --detect-branches if you don't really
need it, instead doing, say, multiple "git p4 sync" for the
different areas of the repo that interest you, each with its own
destination branch in git ("p4/depot-part1", "p4/depot-part3",
...).  Or --use-client-spec to cobble together an exact mapping
of where p4 files should land in git, all in a single git branch
then.

> I propose the addition of a branch list filtering option
> (--filter-branches) that takes either a regular expression or list of
> branches it should check.  This may be useful in sane situations where
> you don't want to scan every branch in a Perforce repository, or
> blacklist branches that have undesirable content (for example, one of
> the branches is called 'svn-backup'.  It contains a single, multi-GB
> tarball.)

There is the existing git-p4.branchList option that explicitly
adds (or overrides) branch information, beyond the ones auto-discovered.

You might be able to use that option, but change its behavior
to avoid the scan.  So that if that option is set in the config,
p4 is not asked anything about its branches.  Not sure if this
would break anyone's setup though.

Another approach would be to add a config option
git-p4.branchScan that defaults to True.  You could turn it off
and use branchList.

> It would be ideal to have this information (after initial clone or
> sync) stored somewhere in the git config where is appropriate so that
> future submit/rebase operations adhere to this list.
> 
> Has something like this been worked on, or has been considered in the
> past?  If not I will consider implementing this after reading up on
> the Git code guidelines.
> 
> Thanks for keeping the Git workflow accessible in painful areas.

It would be great if you could get something like this to work.
Start in getBranchMapping() and don't forget to write up your
work in Documentation/git-p4.txt.  Also, this is sort of a messy
area of the code, unfortunately.  t/t9801 tries to make sure some
of it keeps working.

		-- Pete

Re: Fwd: git p4: feature request - branch check filtering

[Dan Porter]

Hi Pete,

I should have updated on this earlier, but I wished to refine my work
on this feature before submitting.  With 2.0 looming I'll submit
what's there so far.

There is a patch viewable at this link:
https://github.com/Stealthii/git/commit/f7a2e611262fd977ac99e066872d3d0743b7df3c

For the use case this works perfectly - if I define branch mappings
with git config, followed by setting 'git-p4.skipBranchScan' to true,
git-p4 will skip scanning of all remote branches and limit to what's
defined in the map.  An example config:

[git-p4]
        skipBranchScan = true
        branchList = release_1.0.0:release_1.1.0
        branchList = release_1.1.0:release_1.2.0

If there is any more information I need to provide let me know. I have
been using this patch for over two months, testing both use cases with
and without git-p4.skipBranchScan and I have noticed no issues.  Logic
of git-p4 is not changed from default behaviour, unless the user
explicitly sets the boolean flag to skip scanning.

Dan


This email and any files transmitted with it are confidential and
intended solely for the use of the individual or entity to whom they
are addressed. If you have received this email in error please notify
the system manager. This message contains confidential information and
is intended only for the individual named. If you are not the named
addressee you should not disseminate, distribute or copy this e-mail.
Please notify the sender immediately by e-mail if you have received
this e-mail by mistake and delete this e-mail from your system. If you
are not the intended recipient you are notified that disclosing,
copying, distributing or taking any action in reliance on the contents
of this information is strictly prohibited.


On 23 February 2014 15:12, Pete Wyckoff <pw@padd.com> wrote:
> dpreid@gmail.com wrote on Tue, 18 Feb 2014 12:42 +0000:
>> I work at a company that has recently moved all CVS, SVN, and git
>> repositories to Perforce.  Depots have not been setup correctly in
>> every case, and there is one depot that contains literally hundreds of
>> projects under commercial development (and hundreds of branches as a
>> result)
>
> My condolences.
>
>> My project may be in //stupid_depot/commercial/teamporter/rok.  This
>> is the path I clone with git-p4.  The only branches in this depot that
>> contain files at this path are titled as
>> 'rok_porter_branch/release_1.x' or similar.
>>
>> When using '--detect-branches' git-p4 checks each key of branches to
>> see if any of them have files in the path I've cloned.  Whilst this is
>> good in practice there is unfortunately 6,809 branches, git-p4
>> processes about 2 a second and just under an hour to perform any
>> git-p4 rebase, submit, or similar operation.
>
> This is in getBranchMapping() presumably.  Where it loops
> over each branch doing "p4 branch -o".  Yuk.
>
> You could always avoid the --detect-branches if you don't really
> need it, instead doing, say, multiple "git p4 sync" for the
> different areas of the repo that interest you, each with its own
> destination branch in git ("p4/depot-part1", "p4/depot-part3",
> ...).  Or --use-client-spec to cobble together an exact mapping
> of where p4 files should land in git, all in a single git branch
> then.
>
>> I propose the addition of a branch list filtering option
>> (--filter-branches) that takes either a regular expression or list of
>> branches it should check.  This may be useful in sane situations where
>> you don't want to scan every branch in a Perforce repository, or
>> blacklist branches that have undesirable content (for example, one of
>> the branches is called 'svn-backup'.  It contains a single, multi-GB
>> tarball.)
>
> There is the existing git-p4.branchList option that explicitly
> adds (or overrides) branch information, beyond the ones auto-discovered.
>
> You might be able to use that option, but change its behavior
> to avoid the scan.  So that if that option is set in the config,
> p4 is not asked anything about its branches.  Not sure if this
> would break anyone's setup though.
>
> Another approach would be to add a config option
> git-p4.branchScan that defaults to True.  You could turn it off
> and use branchList.
>
>> It would be ideal to have this information (after initial clone or
>> sync) stored somewhere in the git config where is appropriate so that
>> future submit/rebase operations adhere to this list.
>>
>> Has something like this been worked on, or has been considered in the
>> past?  If not I will consider implementing this after reading up on
>> the Git code guidelines.
>>
>> Thanks for keeping the Git workflow accessible in painful areas.
>
> It would be great if you could get something like this to work.
> Start in getBranchMapping() and don't forget to write up your
> work in Documentation/git-p4.txt.  Also, this is sort of a messy
> area of the code, unfortunately.  t/t9801 tries to make sure some
> of it keeps working.
>
>                 -- Pete
>

Re: Fwd: git p4: feature request - branch check filtering

[Junio C Hamano]

Dan Porter <dpreid@gmail.com> writes:

> I should have updated on this earlier, but I wished to refine my work
> on this feature before submitting.  With 2.0 looming I'll submit
> what's there so far.

I am not Pete, but...

The pre-release time is to find and fix regressions that may have
been introduced since the last release while we tried to add new
features and fixes to 2.0 until now.

"With 2.0 looming" is not a good reason to send out a new work.  In
fact, if 2.0 is "looming", it is already too late for the upcoming
release.  Of course "With 2.0 looming" does not mean that you must
not work on things that are regression fixes---it is your own time
and effort, and it will be great if that can help later releases.

Thanks for contributing anyway ;-)

Re: Fwd: git p4: feature request - branch check filtering

[Pete Wyckoff]

dpreid@gmail.com wrote on Tue, 22 Apr 2014 10:29 +0100:
> There is a patch viewable at this link:
> https://github.com/Stealthii/git/commit/f7a2e611262fd977ac99e066872d3d0743b7df3c
> 
> For the use case this works perfectly - if I define branch mappings
> with git config, followed by setting 'git-p4.skipBranchScan' to true,
> git-p4 will skip scanning of all remote branches and limit to what's
> defined in the map.  An example config:
> 
> [git-p4]
>         skipBranchScan = true
>         branchList = release_1.0.0:release_1.1.0
>         branchList = release_1.1.0:release_1.2.0
> 
> If there is any more information I need to provide let me know. I have
> been using this patch for over two months, testing both use cases with
> and without git-p4.skipBranchScan and I have noticed no issues.  Logic
> of git-p4 is not changed from default behaviour, unless the user
> explicitly sets the boolean flag to skip scanning.

Thanks, Dan.  This looks good and is a fine compromise
considering the various choices we discussed earlier.

Junio's comments about 2.0 non-withstanding, I think this change
should go into the next convenient release.  So 2.1 or 2.0.1;
however the numbers end up working post-2.0.

If you could take a look at Documentation/SubmittingPatches,
and do a few things:

    1.  Write a nice commit message, say:

	git p4: add skipBranchScan to avoid p4 branch scan

	Some more useful text.

    2.  Include at the bottom of that message:

	Acked-by: Pete Wyckoff <pw@padd.com>

    3.  Inline the text of your patch, not just a link to github.

    4.  Consider adding a t98xx test.  This isn't required for
	a fairly minor change like yours, but if you think TDD
	is fun, have at it.  Might protect your feature against
	future hackers who would try to break it.  :)

Then send it to vger, cc junio (and me), and he will be kind
enough to queue it up appropriately.

Thanks!

		-- Pete