[PATCH 0/1] [RFC]git fetch support for rebaseable repo (a.k.a crazy git fetcher)

[PATCH 0/1] [RFC]git fetch support for rebaseable repo (a.k.a crazy git fetcher)

[Yu Ke]

From: Yu Ke <ke.yu@intel.com>


Pull URL: git://git.pokylinux.org/poky-contrib.git
  Branch: kyu3/crazygit
  Browse: http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=kyu3/crazygit

Thanks,
    Yu Ke <ke.yu@intel.com>
---


Yu Ke (1):
  git fetcher: add support for rebaseable git repo

 bitbake/lib/bb/fetch2/git.py |   28 ++++++++++++++++++++--------
 1 files changed, 20 insertions(+), 8 deletions(-)

[PATCH 1/1] git fetcher: add support for rebaseable git repo

[Yu Ke]

From: Yu Ke <ke.yu@intel.com>

Some upstream git repo may rebase in the future, which means current
revision may disappear from the upstream repo after the rebase.

current git fetcher can not handle this case, because the git mirror
tar ball is per repo, and may also change in the rebase and lost the
current revision info.

To fix this issue, this patch
- add rebaseable tag in the SRC_URI
- for rebaseable repo, make git mirror tar ball per revision, in this
  case, even upstream rebase, the git mirror still has the current
  revision info.
- for rebaseable repo, generate mirror tar ball by default, since the
  repo may change in the future.

Signed-off-by: Yu Ke <ke.yu@intel.com>
---
 bitbake/lib/bb/fetch2/git.py |   28 ++++++++++++++++++++--------
 1 files changed, 20 insertions(+), 8 deletions(-)

diff --git a/bitbake/lib/bb/fetch2/git.py b/bitbake/lib/bb/fetch2/git.py
index 811acbf..82721c6 100644
--- a/bitbake/lib/bb/fetch2/git.py
+++ b/bitbake/lib/bb/fetch2/git.py
@@ -57,6 +57,11 @@ class Git(FetchMethod):
         if 'nocheckout' in ud.parm:
             ud.nocheckout = True
 
+        # rebaseable means the upstream git repo may rebase in the future,
+        # and current revision may disappear from upstream repo
+        # rebaseable is false by default. set rebaseable=1 in SRC_URI if rebaseable.
+        ud.rebaseable = ud.parm.get("rebaseable","0") == "1"
+
         branches = ud.parm.get("branch", "master").split(',')
         if len(branches) != len(ud.names):
             raise bb.fetch2.ParameterError("The number of name and branch parameters is not balanced", ud.url)
@@ -65,16 +70,9 @@ class Git(FetchMethod):
             branch = branches[ud.names.index(name)]
             ud.branches[name] = branch
 
-        gitsrcname = '%s%s' % (ud.host, ud.path.replace('/', '.'))
-        ud.mirrortarball = 'git2_%s.tar.gz' % (gitsrcname)
-        ud.fullmirror = os.path.join(data.getVar("DL_DIR", d, True), ud.mirrortarball)
-        ud.clonedir = os.path.join(data.expand('${GITDIR}', d), gitsrcname)
-
         ud.basecmd = data.getVar("FETCHCMD_git", d, True) or "git"
 
-        ud.write_tarballs = (data.getVar("BB_GENERATE_MIRROR_TARBALLS", d, True) or "0") != "0"
-
-        ud.localfile = ud.clonedir
+        ud.write_tarballs = ((data.getVar("BB_GENERATE_MIRROR_TARBALLS", d, True) or "0") != "0") or ud.rebaseable
 
         ud.setup_revisons(d)
 
@@ -84,6 +82,20 @@ class Git(FetchMethod):
                 ud.branches[name] = ud.revisions[name]
                 ud.revisions[name] = self.latest_revision(ud.url, ud, d, name)
 
+        gitsrcname = '%s%s' % (ud.host, ud.path.replace('/', '.'))
+        # for rebaseable git repo, it is necessary to keep mirror tar ball
+        # per revision, so that even the revision disappears from the
+        # upstream repo in the future, the mirror will remain intact and still
+        # contains the revision
+        if ud.rebaseable:
+            for name in ud.names:
+                gitsrcname = gitsrcname + '_' + ud.revisions[name]
+        ud.mirrortarball = 'git2_%s.tar.gz' % (gitsrcname)
+        ud.fullmirror = os.path.join(data.getVar("DL_DIR", d, True), ud.mirrortarball)
+        ud.clonedir = os.path.join(data.expand('${GITDIR}', d), gitsrcname)
+
+        ud.localfile = ud.clonedir
+
     def localpath(self, url, ud, d):
         return ud.clonedir
 
-- 
1.7.0.4

Re: [PATCH 1/1] git fetcher: add support for rebaseable git repo

[Khem Raj]

On 5/15/2011 6:16 PM, Yu Ke wrote:
> From: Yu Ke<ke.yu@intel.com>
>
> Some upstream git repo may rebase in the future, which means current
> revision may disappear from the upstream repo after the rebase.
>
> current git fetcher can not handle this case, because the git mirror
> tar ball is per repo, and may also change in the rebase and lost the
> current revision info.
>
> To fix this issue, this patch
> - add rebaseable tag in the SRC_URI
> - for rebaseable repo, make git mirror tar ball per revision, in this
>    case, even upstream rebase, the git mirror still has the current
>    revision info.
> - for rebaseable repo, generate mirror tar ball by default, since the
>    repo may change in the future.

Do we have the options to SRC_URI documented somewhere ?
if yes then this should be added there too.

>
> Signed-off-by: Yu Ke<ke.yu@intel.com>
> ---
>   bitbake/lib/bb/fetch2/git.py |   28 ++++++++++++++++++++--------
>   1 files changed, 20 insertions(+), 8 deletions(-)
>
> diff --git a/bitbake/lib/bb/fetch2/git.py b/bitbake/lib/bb/fetch2/git.py
> index 811acbf..82721c6 100644
> --- a/bitbake/lib/bb/fetch2/git.py
> +++ b/bitbake/lib/bb/fetch2/git.py
> @@ -57,6 +57,11 @@ class Git(FetchMethod):
>           if 'nocheckout' in ud.parm:
>               ud.nocheckout = True
>
> +        # rebaseable means the upstream git repo may rebase in the future,
> +        # and current revision may disappear from upstream repo
> +        # rebaseable is false by default. set rebaseable=1 in SRC_URI if rebaseable.
> +        ud.rebaseable = ud.parm.get("rebaseable","0") == "1"
> +
>           branches = ud.parm.get("branch", "master").split(',')
>           if len(branches) != len(ud.names):
>               raise bb.fetch2.ParameterError("The number of name and branch parameters is not balanced", ud.url)
> @@ -65,16 +70,9 @@ class Git(FetchMethod):
>               branch = branches[ud.names.index(name)]
>               ud.branches[name] = branch
>
> -        gitsrcname = '%s%s' % (ud.host, ud.path.replace('/', '.'))
> -        ud.mirrortarball = 'git2_%s.tar.gz' % (gitsrcname)
> -        ud.fullmirror = os.path.join(data.getVar("DL_DIR", d, True), ud.mirrortarball)
> -        ud.clonedir = os.path.join(data.expand('${GITDIR}', d), gitsrcname)
> -
>           ud.basecmd = data.getVar("FETCHCMD_git", d, True) or "git"
>
> -        ud.write_tarballs = (data.getVar("BB_GENERATE_MIRROR_TARBALLS", d, True) or "0") != "0"
> -
> -        ud.localfile = ud.clonedir
> +        ud.write_tarballs = ((data.getVar("BB_GENERATE_MIRROR_TARBALLS", d, True) or "0") != "0") or ud.rebaseable
>
>           ud.setup_revisons(d)
>
> @@ -84,6 +82,20 @@ class Git(FetchMethod):
>                   ud.branches[name] = ud.revisions[name]
>                   ud.revisions[name] = self.latest_revision(ud.url, ud, d, name)
>
> +        gitsrcname = '%s%s' % (ud.host, ud.path.replace('/', '.'))
> +        # for rebaseable git repo, it is necessary to keep mirror tar ball
> +        # per revision, so that even the revision disappears from the
> +        # upstream repo in the future, the mirror will remain intact and still
> +        # contains the revision
> +        if ud.rebaseable:
> +            for name in ud.names:
> +                gitsrcname = gitsrcname + '_' + ud.revisions[name]
> +        ud.mirrortarball = 'git2_%s.tar.gz' % (gitsrcname)
> +        ud.fullmirror = os.path.join(data.getVar("DL_DIR", d, True), ud.mirrortarball)
> +        ud.clonedir = os.path.join(data.expand('${GITDIR}', d), gitsrcname)
> +
> +        ud.localfile = ud.clonedir
> +
>       def localpath(self, url, ud, d):
>           return ud.clonedir
>

Re: [PATCH 1/1] git fetcher: add support for rebaseable git repo

[Darren Hart]

On 05/20/2011 09:30 AM, Khem Raj wrote:
> On 5/15/2011 6:16 PM, Yu Ke wrote:
>> From: Yu Ke<ke.yu@intel.com>
>>
>> Some upstream git repo may rebase in the future, which means current
>> revision may disappear from the upstream repo after the rebase.
>>
>> current git fetcher can not handle this case, because the git mirror
>> tar ball is per repo, and may also change in the rebase and lost the
>> current revision info.
>>
>> To fix this issue, this patch
>> - add rebaseable tag in the SRC_URI
>> - for rebaseable repo, make git mirror tar ball per revision, in this
>>    case, even upstream rebase, the git mirror still has the current
>>    revision info.
>> - for rebaseable repo, generate mirror tar ball by default, since the
>>    repo may change in the future.
> 
> Do we have the options to SRC_URI documented somewhere ?
> if yes then this should be added there too.

I was looking and didn't find them. At the very least these should be
documented in the fetch2/git.py file for reference when
reviewing/maintaining the source.

-- 
Darren Hart
Intel Open Source Technology Center
Yocto Project - Linux Kernel

Re: [PATCH 1/1] git fetcher: add support for rebaseable git repo

[Yu Ke]

on 2011-5-21 8:07, Darren Hart wrote:
>
>
> On 05/20/2011 09:30 AM, Khem Raj wrote:
>> On 5/15/2011 6:16 PM, Yu Ke wrote:
>>> From: Yu Ke<ke.yu@intel.com>
>>>
>>> Some upstream git repo may rebase in the future, which means current
>>> revision may disappear from the upstream repo after the rebase.
>>>
>>> current git fetcher can not handle this case, because the git mirror
>>> tar ball is per repo, and may also change in the rebase and lost the
>>> current revision info.
>>>
>>> To fix this issue, this patch
>>> - add rebaseable tag in the SRC_URI
>>> - for rebaseable repo, make git mirror tar ball per revision, in this
>>>     case, even upstream rebase, the git mirror still has the current
>>>     revision info.
>>> - for rebaseable repo, generate mirror tar ball by default, since the
>>>     repo may change in the future.
>>
>> Do we have the options to SRC_URI documented somewhere ?
>> if yes then this should be added there too.
>
> I was looking and didn't find them. At the very least these should be
> documented in the fetch2/git.py file for reference when
> reviewing/maintaining the source.
>

This is good reminder, Thanks.

There are several places that hold this info:

- bitbake manual: http://bitbake.berlios.de/manual/ch03s07.html
- openembedded manual: 
http://docs.openembedded.org/usermanual/usermanual.html#id572539
- poky reference manual: 
http://www.yoctoproject.org/docs/poky-ref-manual/poky-ref-manual.html#ref-bitbake-fetchers

It looks the most appropriate place would be bitbake manual, which is 
also refereed by poky reference manual. So does anyone know how can I 
update this mannual?

Also, it sounds good idea to add the SRC_URI options doc in the source, 
I will cook a patch to add this info.

Regards
Ke

Re: [PATCH 1/1] git fetcher: add support for rebaseable git repo

[Darren Hart]

On 05/23/2011 10:50 PM, Yu Ke wrote:
> on 2011-5-21 8:07, Darren Hart wrote:
>>
>>
>> On 05/20/2011 09:30 AM, Khem Raj wrote:
>>> On 5/15/2011 6:16 PM, Yu Ke wrote:
>>>> From: Yu Ke<ke.yu@intel.com>
>>>>
>>>> Some upstream git repo may rebase in the future, which means current
>>>> revision may disappear from the upstream repo after the rebase.
>>>>
>>>> current git fetcher can not handle this case, because the git mirror
>>>> tar ball is per repo, and may also change in the rebase and lost the
>>>> current revision info.
>>>>
>>>> To fix this issue, this patch
>>>> - add rebaseable tag in the SRC_URI
>>>> - for rebaseable repo, make git mirror tar ball per revision, in this
>>>>     case, even upstream rebase, the git mirror still has the current
>>>>     revision info.
>>>> - for rebaseable repo, generate mirror tar ball by default, since the
>>>>     repo may change in the future.
>>>
>>> Do we have the options to SRC_URI documented somewhere ?
>>> if yes then this should be added there too.
>>
>> I was looking and didn't find them. At the very least these should be
>> documented in the fetch2/git.py file for reference when
>> reviewing/maintaining the source.
>>
> 
> This is good reminder, Thanks.
> 
> There are several places that hold this info:
> 
> - bitbake manual: http://bitbake.berlios.de/manual/ch03s07.html
> - openembedded manual: 
> http://docs.openembedded.org/usermanual/usermanual.html#id572539
> - poky reference manual: 
> http://www.yoctoproject.org/docs/poky-ref-manual/poky-ref-manual.html#ref-bitbake-fetchers
> 
> It looks the most appropriate place would be bitbake manual, which is 
> also refereed by poky reference manual. So does anyone know how can I 
> update this mannual?

I've provided a couple minor fixes to the manual. I just edited:

poky.git/bitbake/doc/manual/usermanual.xml

And sent the patch to the poky list. It might be preferable to develop
against the actual bitbake repository, but nobody has complained yet :-)

> 
> Also, it sounds good idea to add the SRC_URI options doc in the source, 
> I will cook a patch to add this info.

Great! Thanks Ke,

> 
> Regards
> Ke

-- 
Darren Hart
Intel Open Source Technology Center
Yocto Project - Linux Kernel

Re: [PATCH 0/1] [RFC]git fetch support for rebaseable repo (a.k.a crazy git fetcher)

[Richard Purdie]

On Mon, 2011-05-16 at 09:16 +0800, Yu Ke wrote:
> From: Yu Ke <ke.yu@intel.com>
> 
> 
> Pull URL: git://git.pokylinux.org/poky-contrib.git
>   Branch: kyu3/crazygit
>   Browse: http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=kyu3/crazygit
> 
> Thanks,
>     Yu Ke <ke.yu@intel.com>
> ---
> 
> 
> Yu Ke (1):
>   git fetcher: add support for rebaseable git repo

Merged to bitbake upstream and master, thanks.

Richard