can bitbake build offline ?

can bitbake build offline ?

[MOHAMMAD RASIM]

Hi, is there a way I can run a bitbake build offline? I have all the 
required sources of the target in the sources directory, and I have 
built the target multiple times with internet connection, but when I 
build offline it fails because it needs internet access during parsing 
recipes.
Can I tell bitbake to skip running `git ls-remote` during parsing 
recipes so I can build offline?

Thanks

Re: can bitbake build offline ?

[Alexander Kanavin]

This page has a tip on what might be causing 'git ls-remote':

https://wiki.yoctoproject.org/wiki/How_do_I#Q:_How_do_I_create_my_own_source_download_mirror_.3F

Alex

2018-07-21 10:32 GMT+02:00 MOHAMMAD RASIM <mohammad.rasim96@gmail.com>:
> Hi, is there a way I can run a bitbake build offline? I have all the
> required sources of the target in the sources directory, and I have built
> the target multiple times with internet connection, but when I build offline
> it fails because it needs internet access during parsing recipes.
> Can I tell bitbake to skip running `git ls-remote` during parsing recipes so
> I can build offline?
>
> Thanks
> --
> _______________________________________________
> yocto mailing list
> yocto@yoctoproject.org
> https://lists.yoctoproject.org/listinfo/yocto

Re: can bitbake build offline ?

[MOHAMMAD RASIM]

I tried that and bitbake still fails when disabling the network


On 07/21/2018 11:42 AM, Alexander Kanavin wrote:
> This page has a tip on what might be causing 'git ls-remote':
>
> https://wiki.yoctoproject.org/wiki/How_do_I#Q:_How_do_I_create_my_own_source_download_mirror_.3F
>
> Alex
>
> 2018-07-21 10:32 GMT+02:00 MOHAMMAD RASIM <mohammad.rasim96@gmail.com>:
>> Hi, is there a way I can run a bitbake build offline? I have all the
>> required sources of the target in the sources directory, and I have built
>> the target multiple times with internet connection, but when I build offline
>> it fails because it needs internet access during parsing recipes.
>> Can I tell bitbake to skip running `git ls-remote` during parsing recipes so
>> I can build offline?
>>
>> Thanks
>> --
>> _______________________________________________
>> yocto mailing list
>> yocto@yoctoproject.org
>> https://lists.yoctoproject.org/listinfo/yocto

Re: can bitbake build offline ?

[Paul Eggleton]

Hi Mohammad,

If it's failing during parsing that means you still have a recipe that does 
not have a proper SRCREV set, which you need to fix as described by the page 
that Alex linked. The error will be reporting which recipe that is.

Cheers,
Paul

On Thursday, 26 July 2018 5:44:26 AM CEST MOHAMMAD RASIM wrote:
> I tried that and bitbake still fails when disabling the network
> 
> 
> On 07/21/2018 11:42 AM, Alexander Kanavin wrote:
> > This page has a tip on what might be causing 'git ls-remote':
> >
> > https://wiki.yoctoproject.org/wiki/
How_do_I#Q:_How_do_I_create_my_own_source_download_mirror_.3F
> >
> > Alex
> >
> > 2018-07-21 10:32 GMT+02:00 MOHAMMAD RASIM <mohammad.rasim96@gmail.com>:
> >> Hi, is there a way I can run a bitbake build offline? I have all the
> >> required sources of the target in the sources directory, and I have built
> >> the target multiple times with internet connection, but when I build 
offline
> >> it fails because it needs internet access during parsing recipes.
> >> Can I tell bitbake to skip running `git ls-remote` during parsing recipes 
so
> >> I can build offline?
> >>
> >> Thanks
> >> --
> >> _______________________________________________
> >> yocto mailing list
> >> yocto@yoctoproject.org
> >> https://lists.yoctoproject.org/listinfo/yocto
> 
> 


-- 

Paul Eggleton
Intel Open Source Technology Centre

Re: can bitbake build offline ?

[MOHAMMAD RASIM]

Well, actually I used the find command in that wiki page to find the 
recipe that uses tag name to chose github revision and it found only one 
recipe

 >meta-openembedded/meta-initramfs/recipes-devtools/mtd/ubi-utils-klibc_1.5.1.bb

I masked this recipe in the local.conf file and bitbake failed at 
parsing other recipes, looking at those recipes I saw that they don't 
use tag names but they use

 >SRCREV = "${AUTOREV}"

which is logical since bitbake has to issue ls-remote to know what is 
the latest revision on the remote git repo (maybe the wiki page needs 
update to point this ?)

Now I can't remove this AUTOREV in these recipes since I need bitbake to 
fetch the latest updates ( I push updates regularly on that repo and I 
don't want to update the recipe file each time to add the latest revision)

So, here comes the hard part, Is it possible to tell bitbake to issue 
ls-remote if there is network and to use the latest fetched revision in 
the sources directory if the network is disabled ? that way I can use 
${AUTOREV} in my recipes and bitbake will build from the local repo in 
the sources directory unless there is network to update the local repo, 
probably there is no such thing :) .

Thanks


On 07/26/2018 09:26 AM, Paul Eggleton wrote:
> Hi Mohammad,
>
> If it's failing during parsing that means you still have a recipe that does
> not have a proper SRCREV set, which you need to fix as described by the page
> that Alex linked. The error will be reporting which recipe that is.
>
> Cheers,
> Paul
>
> On Thursday, 26 July 2018 5:44:26 AM CEST MOHAMMAD RASIM wrote:
>> I tried that and bitbake still fails when disabling the network
>>
>>
>> On 07/21/2018 11:42 AM, Alexander Kanavin wrote:
>>> This page has a tip on what might be causing 'git ls-remote':
>>>
>>> https://wiki.yoctoproject.org/wiki/
> How_do_I#Q:_How_do_I_create_my_own_source_download_mirror_.3F
>>> Alex
>>>
>>> 2018-07-21 10:32 GMT+02:00 MOHAMMAD RASIM <mohammad.rasim96@gmail.com>:
>>>> Hi, is there a way I can run a bitbake build offline? I have all the
>>>> required sources of the target in the sources directory, and I have built
>>>> the target multiple times with internet connection, but when I build
> offline
>>>> it fails because it needs internet access during parsing recipes.
>>>> Can I tell bitbake to skip running `git ls-remote` during parsing recipes
> so
>>>> I can build offline?
>>>>
>>>> Thanks
>>>> --
>>>> _______________________________________________
>>>> yocto mailing list
>>>> yocto@yoctoproject.org
>>>> https://lists.yoctoproject.org/listinfo/yocto
>>
>

Re: can bitbake build offline ?

[Paul Eggleton]

On Thursday, 26 July 2018 12:09:46 PM CEST MOHAMMAD RASIM wrote:
> Well, actually I used the find command in that wiki page to find the 
> recipe that uses tag name to chose github revision and it found only one 
> recipe
> 
>  >meta-openembedded/meta-initramfs/recipes-devtools/mtd/ubi-utils-
>  >klibc_1.5.1.bb

Hmm, right, it seems like you're using a previous branch - this got fixed
in a later update of that recipe.

> I masked this recipe in the local.conf file and bitbake failed at 
> parsing other recipes, looking at those recipes I saw that they don't 
> use tag names but they use
> 
>  >SRCREV = "${AUTOREV}"
> 
> which is logical since bitbake has to issue ls-remote to know what is 
> the latest revision on the remote git repo (maybe the wiki page needs 
> update to point this ?)
> 
> Now I can't remove this AUTOREV in these recipes since I need bitbake to 
> fetch the latest updates ( I push updates regularly on that repo and I 
> don't want to update the recipe file each time to add the latest revision)
> 
> So, here comes the hard part, Is it possible to tell bitbake to issue 
> ls-remote if there is network and to use the latest fetched revision in 
> the sources directory if the network is disabled ? that way I can use 
> ${AUTOREV} in my recipes and bitbake will build from the local repo in 
> the sources directory unless there is network to update the local repo, 
> probably there is no such thing :) .

So there isn't a mechanism to do exactly that, no, however you can sort of
achieve the same thing if you set up a .inc file that sets the SRCREV values
for each recipe and sets BB_NO_NETWORK at the same time (though the
latter could still be separate). e.g. let's call it no_network.inc:

BB_NO_NETWORK = "1"
SRCREV_pn-abc = "b4c2bd84ee6f699e348d602a82d2d0963384cdea"
SRCREV_pn-xyz = "e3b30def2cd1c9ede7630489c3949a45b6eba6ee"
..

Then to build offline you would just add the following to your config:

require no_network.inc

FYI you can enable buildhistory and use the buildhistory-collect-srcrevs 
script to generate all those SRCREV lines so you don't have to do that
by hand:

https://www.yoctoproject.org/docs/current/dev-manual/dev-manual.html#maintaining-build-output-quality

Cheers,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre

Re: can bitbake build offline ?

[MOHAMMAD RASIM]

Wow that's a cool trick and it worked for me so ,thank you.

Also after some digging I found a solution that does exactly what I 
described in my previous message(I think), and I'm gonna leave it here 
in case someone face the same issue and find this message:

You can use set BB_SRCREV_POLICY = "cache" the local.conf which will 
prevent the AUTOREV recipes from failing by using the local cache for 
revisions

https://www.yoctoproject.org/docs/2.1/bitbake-user-manual/bitbake-user-manual.html#var-BB_SRCREV_POLICY

But If there is a new commits on the remote repo you probably need to 
disable this and rebuild to let bitbake fetch the latest revision and 
you can re enable it afterwards.


Regards


On 07/26/2018 01:40 PM, Paul Eggleton wrote:
> On Thursday, 26 July 2018 12:09:46 PM CEST MOHAMMAD RASIM wrote:
>> Well, actually I used the find command in that wiki page to find the
>> recipe that uses tag name to chose github revision and it found only one
>> recipe
>>
>>   >meta-openembedded/meta-initramfs/recipes-devtools/mtd/ubi-utils-
>>   >klibc_1.5.1.bb
> Hmm, right, it seems like you're using a previous branch - this got fixed
> in a later update of that recipe.
>
>> I masked this recipe in the local.conf file and bitbake failed at
>> parsing other recipes, looking at those recipes I saw that they don't
>> use tag names but they use
>>
>>   >SRCREV = "${AUTOREV}"
>>
>> which is logical since bitbake has to issue ls-remote to know what is
>> the latest revision on the remote git repo (maybe the wiki page needs
>> update to point this ?)
>>
>> Now I can't remove this AUTOREV in these recipes since I need bitbake to
>> fetch the latest updates ( I push updates regularly on that repo and I
>> don't want to update the recipe file each time to add the latest revision)
>>
>> So, here comes the hard part, Is it possible to tell bitbake to issue
>> ls-remote if there is network and to use the latest fetched revision in
>> the sources directory if the network is disabled ? that way I can use
>> ${AUTOREV} in my recipes and bitbake will build from the local repo in
>> the sources directory unless there is network to update the local repo,
>> probably there is no such thing :) .
> So there isn't a mechanism to do exactly that, no, however you can sort of
> achieve the same thing if you set up a .inc file that sets the SRCREV values
> for each recipe and sets BB_NO_NETWORK at the same time (though the
> latter could still be separate). e.g. let's call it no_network.inc:
>
> BB_NO_NETWORK = "1"
> SRCREV_pn-abc = "b4c2bd84ee6f699e348d602a82d2d0963384cdea"
> SRCREV_pn-xyz = "e3b30def2cd1c9ede7630489c3949a45b6eba6ee"
> ..
>
> Then to build offline you would just add the following to your config:
>
> require no_network.inc
>
> FYI you can enable buildhistory and use the buildhistory-collect-srcrevs
> script to generate all those SRCREV lines so you don't have to do that
> by hand:
>
> https://www.yoctoproject.org/docs/current/dev-manual/dev-manual.html#maintaining-build-output-quality
>
> Cheers,
> Paul
>