RFC: Support for R/W git in bitbake

RFC: Support for R/W git in bitbake

[Ulf Samuelsson]

Today bitbake supports read only git access in recipes.
For various reasons, I would like to be able to do recipes
which would check out in a read/write mode.

I.E: have bitbake do "git clone git@emagii.com:myproject"

Did a prototype extension to bitbake that assumes the following URI
     wgit://emagii.com/myproject

Any objections to such an extension?
Suggestions for an alternative implementation?

-- 
Best Regards
Ulf Samuelsson
ulf@emagii.com

Re: RFC: Support for R/W git in bitbake

[Paul Eggleton]

On Wednesday 18 April 2012 00:35:29 Ulf Samuelsson wrote:
> Today bitbake supports read only git access in recipes.
> For various reasons, I would like to be able to do recipes
> which would check out in a read/write mode.

Could you elaborate on "various reasons"?

Would the new externalsrc bbclass be useful in your case?

Cheers,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre

Re: RFC: Support for R/W git in bitbake

[Ulf Samuelsson]

On 2012-04-18 01:07, Paul Eggleton wrote:
> On Wednesday 18 April 2012 00:35:29 Ulf Samuelsson wrote:
>> Today bitbake supports read only git access in recipes.
>> For various reasons, I would like to be able to do recipes
>> which would check out in a read/write mode.
> Could you elaborate on "various reasons"?
>
> Would the new externalsrc bbclass be useful in your case?
>
> Cheers,
> Paul
>

1. If I am busy working on an application, then it simplifies the 
development process.
     I can modify the code in the tree and push.
     This is mainly for kernel development.

2. If I work on a prerelease of some S/W drivers/Applications under NDA,
     then I cannot make that code publicly available
     but I still want to put  that on my Internet accessible git server.
     Typicailly this is before the release of a new chip and info about 
the chip should not be
     made public before the chip is released.

3.  I want to be able to ship something similar to the Angstrom setup 
scripts
      to someone else, and have them build an image, but it should not 
be available
      to anyone not accepted (without public key at the git server).

There are other uses for such a functionality, but those are my 
immediate needs.

As you see, this is mostly for development.
Once the code is released, then the recipe would be changed to the 
normal git access.

Didn't know anything about the externalsrc bbclass, but after checking, 
I would say no.
It won't do the two things above. I do see the use of it though.


-- 
Best Regards
Ulf Samuelsson
ulf@emagii.com
+46 722 427437

Re: RFC: Support for R/W git in bitbake

[Antonio Ospite]

[-- Attachment #1: Type: text/plain, Size: 2446 bytes --]

On Wed, 18 Apr 2012 03:34:27 +0200
Ulf Samuelsson <openembedded@emagii.com> wrote:

> On 2012-04-18 01:07, Paul Eggleton wrote:
> > On Wednesday 18 April 2012 00:35:29 Ulf Samuelsson wrote:
> >> Today bitbake supports read only git access in recipes.
> >> For various reasons, I would like to be able to do recipes
> >> which would check out in a read/write mode.
> > Could you elaborate on "various reasons"?
> >
> > Would the new externalsrc bbclass be useful in your case?
> >
> > Cheers,
> > Paul
> >
> 
> 1. If I am busy working on an application, then it simplifies the 
> development process.
>      I can modify the code in the tree and push.
>      This is mainly for kernel development.
>

for that you can pass "protocol=file" to make the git fetcher use the
"file" transport, see

http://ao2.it/en/blog/2010/05/27/neat-compilerun-cycle-git-and-openembedded

basically, in the .bb recipe, you can fetch from a local clone where
you do your normal work.

> 2. If I work on a prerelease of some S/W drivers/Applications under NDA,
>      then I cannot make that code publicly available
>      but I still want to put  that on my Internet accessible git server.
>      Typicailly this is before the release of a new chip and info about 
> the chip should not be
>      made public before the chip is released.
> 
> 3.  I want to be able to ship something similar to the Angstrom setup 
> scripts
>       to someone else, and have them build an image, but it should not 
> be available
>       to anyone not accepted (without public key at the git server).
>

maybe you can pass "protocol=ssh" as well to the git fetcher as well,
but I haven't tried that, when you say "R/W mode" you are basically
saying "ssh transport" for git, right? Let us know if that works.

> There are other uses for such a functionality, but those are my 
> immediate needs.
> 
> As you see, this is mostly for development.
> Once the code is released, then the recipe would be changed to the 
> normal git access.
> 
> Didn't know anything about the externalsrc bbclass, but after checking, 
> I would say no.
> It won't do the two things above. I do see the use of it though.
>

Ciao,
   Antonio

-- 
Antonio Ospite
http://ao2.it

A: Because it messes up the order in which people normally read text.
   See http://en.wikipedia.org/wiki/Posting_style
Q: Why is top-posting such a bad thing?

[-- Attachment #2: Type: application/pgp-signature, Size: 205 bytes --]

Re: RFC: Support for R/W git in bitbake

[Paul Eggleton]

On Wednesday 18 April 2012 03:34:27 Ulf Samuelsson wrote:
> 1. If I am busy working on an application, then it simplifies the
> development process.
>      I can modify the code in the tree and push.
>      This is mainly for kernel development.
> 
> 2. If I work on a prerelease of some S/W drivers/Applications under NDA,
>      then I cannot make that code publicly available
>      but I still want to put  that on my Internet accessible git server.
>      Typicailly this is before the release of a new chip and info about
> the chip should not be
>      made public before the chip is released.
> 
> 3.  I want to be able to ship something similar to the Angstrom setup
> scripts
>       to someone else, and have them build an image, but it should not
> be available
>       to anyone not accepted (without public key at the git server).
> 
> There are other uses for such a functionality, but those are my
> immediate needs.
> 
> As you see, this is mostly for development.
> Once the code is released, then the recipe would be changed to the
> normal git access.
> 
> Didn't know anything about the externalsrc bbclass, but after checking,
> I would say no.
> It won't do the two things above. I do see the use of it though.

externalsrc should handle everything except automatically fetching the source; 
for that you need to have your own local git clone (which of course can be 
r/w). I guess it depends on whether you expect to be sending such recipes to 
non-developers; for developers it ought not to be too much of a hassle to have 
their own local git clone.

The only problem with having an r/w checkout you are doing development in 
under WORKDIR is that if you bitbake -c clean the recipe you will lose 
whatever you are working on - externalsrc avoids this.

Cheers,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre

Re: RFC: Support for R/W git in bitbake

[Ulf Samuelsson]

On 2012-04-18 09:28, Paul Eggleton wrote:
> On Wednesday 18 April 2012 03:34:27 Ulf Samuelsson wrote:
>> 1. If I am busy working on an application, then it simplifies the
>> development process.
>>       I can modify the code in the tree and push.
>>       This is mainly for kernel development.
>>
>> 2. If I work on a prerelease of some S/W drivers/Applications under NDA,
>>       then I cannot make that code publicly available
>>       but I still want to put  that on my Internet accessible git server.
>>       Typicailly this is before the release of a new chip and info about
>> the chip should not be
>>       made public before the chip is released.
>>
>> 3.  I want to be able to ship something similar to the Angstrom setup
>> scripts
>>        to someone else, and have them build an image, but it should not
>> be available
>>        to anyone not accepted (without public key at the git server).
>>
>> There are other uses for such a functionality, but those are my
>> immediate needs.
>>
>> As you see, this is mostly for development.
>> Once the code is released, then the recipe would be changed to the
>> normal git access.
>>
>> Didn't know anything about the externalsrc bbclass, but after checking,
>> I would say no.
>> It won't do the two things above. I do see the use of it though.
> externalsrc should handle everything except automatically fetching the source;
> for that you need to have your own local git clone (which of course can be
> r/w). I guess it depends on whether you expect to be sending such recipes to
> non-developers; for developers it ought not to be too much of a hassle to have
> their own local git clone.
>
> The only problem with having an r/w checkout you are doing development in
> under WORKDIR is that if you bitbake -c clean the recipe you will lose
> whatever you are working on - externalsrc avoids this.
>
> Cheers,
> Paul
>
I am working with a company, which OEMs their solution to other companies.
They get tons of questions from those customers, and based on that
I would like to avoid anything which requires manual intervention
of the developer.

As for cleaning out changes, it might happen, but I am prepared to take 
the risk.
Today I am initializing a git after the extract anyway.

I am sure I will use externalsrc for something.

-- 

Best Regards
Ulf Samuelsson
ulf@emagii.com
+46 722 427437

Re: RFC: Support for R/W git in bitbake

[Anders Darander]

On Wed, Apr 18, 2012 at 08:24, Antonio Ospite <ospite@studenti.unina.it> wrote:
> On Wed, 18 Apr 2012 03:34:27 +0200
> Ulf Samuelsson <openembedded@emagii.com> wrote:
>
>> On 2012-04-18 01:07, Paul Eggleton wrote:
>> > On Wednesday 18 April 2012 00:35:29 Ulf Samuelsson wrote:
>> >> Today bitbake supports read only git access in recipes.
>> >> For various reasons, I would like to be able to do recipes
>> >> which would check out in a read/write mode.
>> > Could you elaborate on "various reasons"?
>> >
>> > Would the new externalsrc bbclass be useful in your case?
>> >
>> > Cheers,
>> > Paul
>> >
>>
>> 1. If I am busy working on an application, then it simplifies the
>> development process.
>>      I can modify the code in the tree and push.
>>      This is mainly for kernel development.
>>
>
> for that you can pass "protocol=file" to make the git fetcher use the
> "file" transport, see
>
> http://ao2.it/en/blog/2010/05/27/neat-compilerun-cycle-git-and-openembedded
>
> basically, in the .bb recipe, you can fetch from a local clone where
> you do your normal work.
>
>> 2. If I work on a prerelease of some S/W drivers/Applications under NDA,
>>      then I cannot make that code publicly available
>>      but I still want to put  that on my Internet accessible git server.
>>      Typicailly this is before the release of a new chip and info about
>> the chip should not be
>>      made public before the chip is released.
>>
>> 3.  I want to be able to ship something similar to the Angstrom setup
>> scripts
>>       to someone else, and have them build an image, but it should not
>> be available
>>       to anyone not accepted (without public key at the git server).
>>
>
> maybe you can pass "protocol=ssh" as well to the git fetcher as well,
> but I haven't tried that, when you say "R/W mode" you are basically
> saying "ssh transport" for git, right? Let us know if that works.

Ulf, any comments on Antonios ideas?

I've just checked our own git repos and recipes; we're using SRC_URI's like:
SRC_URI = "git://gitosis@server-url/linux;protocol=ssh "
Then it is possible to work in that git clone as usual. Though, I've
to mention that this is using oe-core, so there might be some
differences in fetchers etc.

/Anders