[Buildroot] how to get buildroot to update a git repo that has changes? (ie building HEAD vs a specific SHA)

[Buildroot] how to get buildroot to update a git repo that has changes? (ie building HEAD vs a specific SHA)

[Tim Harvey]

Greetings,

I use buildroot to build a Linux kernel+ramdisk using:
BR2_LINUX_KERNEL_CUSTOM_GIT=y
BR2_LINUX_KERNEL_CUSTOM_REPO_URL=<gitrepo>
BR2_LINUX_KERNEL_CUSTOM_REPO_VERSION=<gitbranch>

When the upstream branch changes I would like to build that kernel but
I find that buildroot caches the repo in dl/linux/.git and doesn't
seem to know how to check to see if there are new commits.

I realize that this is atypical as typically you want to configure
buildroot for a specific package gitsha/tag.

How would I go about getting buildroot to update the repo? I don't
particularly want to rm dl/linux I want to instead somehow check to
see if the repo needs a refresh then refresh it without a massive
network download.

Best Regards,

Tim
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] how to get buildroot to update a git repo that has changes? (ie building HEAD vs a specific SHA)

[Matt Flax]

I think that if you update the repo version to the new git checkin 
number, rather then the branch name, then it should refresh the tarball 
to the chosen version.

Matt

On 6/5/22 09:17, Tim Harvey wrote:
> Greetings,
>
> I use buildroot to build a Linux kernel+ramdisk using:
> BR2_LINUX_KERNEL_CUSTOM_GIT=y
> BR2_LINUX_KERNEL_CUSTOM_REPO_URL=<gitrepo>
> BR2_LINUX_KERNEL_CUSTOM_REPO_VERSION=<gitbranch>
>
> When the upstream branch changes I would like to build that kernel but
> I find that buildroot caches the repo in dl/linux/.git and doesn't
> seem to know how to check to see if there are new commits.
>
> I realize that this is atypical as typically you want to configure
> buildroot for a specific package gitsha/tag.
>
> How would I go about getting buildroot to update the repo? I don't
> particularly want to rm dl/linux I want to instead somehow check to
> see if the repo needs a refresh then refresh it without a massive
> network download.
>
> Best Regards,
>
> Tim
> _______________________________________________
> buildroot mailing list
> buildroot@buildroot.org
> https://lists.buildroot.org/mailman/listinfo/buildroot
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] how to get buildroot to update a git repo that has changes? (ie building HEAD vs a specific SHA)

[Baruch Siach via buildroot]

Hi Tim,

On Thu, May 05 2022, Tim Harvey wrote:
> Greetings,
>
> I use buildroot to build a Linux kernel+ramdisk using:
> BR2_LINUX_KERNEL_CUSTOM_GIT=y
> BR2_LINUX_KERNEL_CUSTOM_REPO_URL=<gitrepo>
> BR2_LINUX_KERNEL_CUSTOM_REPO_VERSION=<gitbranch>
>
> When the upstream branch changes I would like to build that kernel but
> I find that buildroot caches the repo in dl/linux/.git and doesn't
> seem to know how to check to see if there are new commits.
>
> I realize that this is atypical as typically you want to configure
> buildroot for a specific package gitsha/tag.
>
> How would I go about getting buildroot to update the repo? I don't
> particularly want to rm dl/linux I want to instead somehow check to
> see if the repo needs a refresh then refresh it without a massive
> network download.

Using a branch name for _VERSION is not supported. See the rationale in
the manual section 18.6.2, note on LIBFOO_VERSION description.

  https://buildroot.org/downloads/manual/manual.html

You might want to consider using OVERRIDE_SRCDIR to make Buildroot build
your local kernel source tree. See section 8.14.6 in the manual.

Hope this helps,
baruch

-- 
                                                     ~. .~   Tk Open Systems
=}------------------------------------------------ooO--U--Ooo------------{=
   - baruch@tkos.co.il - tel: +972.52.368.4656, http://www.tkos.co.il -
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] how to get buildroot to update a git repo that has changes? (ie building HEAD vs a specific SHA)

[Tim Harvey]

On Thu, May 5, 2022 at 8:21 PM Baruch Siach <baruch@tkos.co.il> wrote:
>
> Hi Tim,
>
> On Thu, May 05 2022, Tim Harvey wrote:
> > Greetings,
> >
> > I use buildroot to build a Linux kernel+ramdisk using:
> > BR2_LINUX_KERNEL_CUSTOM_GIT=y
> > BR2_LINUX_KERNEL_CUSTOM_REPO_URL=<gitrepo>
> > BR2_LINUX_KERNEL_CUSTOM_REPO_VERSION=<gitbranch>
> >
> > When the upstream branch changes I would like to build that kernel but
> > I find that buildroot caches the repo in dl/linux/.git and doesn't
> > seem to know how to check to see if there are new commits.
> >
> > I realize that this is atypical as typically you want to configure
> > buildroot for a specific package gitsha/tag.
> >
> > How would I go about getting buildroot to update the repo? I don't
> > particularly want to rm dl/linux I want to instead somehow check to
> > see if the repo needs a refresh then refresh it without a massive
> > network download.
>
> Using a branch name for _VERSION is not supported. See the rationale in
> the manual section 18.6.2, note on LIBFOO_VERSION description.
>
>   https://buildroot.org/downloads/manual/manual.html
>
> You might want to consider using OVERRIDE_SRCDIR to make Buildroot build
> your local kernel source tree. See section 8.14.6 in the manual.
>
> Hope this helps,
> baruch

Baruch,

Thanks for the references. I do understand that it is not supported. I
guess I was looking for a few simple lines of shell that would tell me
if the cached repo was out of date so that I could remove it.
# get branch from BR2 config
KERNEL_BRANCH=$(grep BR2_LINUX_KERNEL_CUSTOM_REPO_VERSION .config |
cut -d= -f2 | tr -d \")
# get the HEAD commit from the BR2 cache
KERNEL_HEAD=$(git -C dl/linux/git/ rev-parse $KERNEL_BRANCH)
^^^ but I'm not sure how to check to see what the latest revision is
on the remote branch?

I will look at switching to OVERRIDE_SRCDIR instead.

Best Regards,

Tim
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] how to get buildroot to update a git repo that has changes? (ie building HEAD vs a specific SHA)

[Baruch Siach via buildroot]

Hi Tim,

On Fri, May 06 2022, Tim Harvey wrote:
> On Thu, May 5, 2022 at 8:21 PM Baruch Siach <baruch@tkos.co.il> wrote:
>> On Thu, May 05 2022, Tim Harvey wrote:
>> > Greetings,
>> >
>> > I use buildroot to build a Linux kernel+ramdisk using:
>> > BR2_LINUX_KERNEL_CUSTOM_GIT=y
>> > BR2_LINUX_KERNEL_CUSTOM_REPO_URL=<gitrepo>
>> > BR2_LINUX_KERNEL_CUSTOM_REPO_VERSION=<gitbranch>
>> >
>> > When the upstream branch changes I would like to build that kernel but
>> > I find that buildroot caches the repo in dl/linux/.git and doesn't
>> > seem to know how to check to see if there are new commits.
>> >
>> > I realize that this is atypical as typically you want to configure
>> > buildroot for a specific package gitsha/tag.
>> >
>> > How would I go about getting buildroot to update the repo? I don't
>> > particularly want to rm dl/linux I want to instead somehow check to
>> > see if the repo needs a refresh then refresh it without a massive
>> > network download.
>>
>> Using a branch name for _VERSION is not supported. See the rationale in
>> the manual section 18.6.2, note on LIBFOO_VERSION description.
>>
>>   https://buildroot.org/downloads/manual/manual.html
>>
>> You might want to consider using OVERRIDE_SRCDIR to make Buildroot build
>> your local kernel source tree. See section 8.14.6 in the manual.
>
> Thanks for the references. I do understand that it is not supported. I
> guess I was looking for a few simple lines of shell that would tell me
> if the cached repo was out of date so that I could remove it.
> # get branch from BR2 config
> KERNEL_BRANCH=$(grep BR2_LINUX_KERNEL_CUSTOM_REPO_VERSION .config |
> cut -d= -f2 | tr -d \")
> # get the HEAD commit from the BR2 cache
> KERNEL_HEAD=$(git -C dl/linux/git/ rev-parse $KERNEL_BRANCH)
> ^^^ but I'm not sure how to check to see what the latest revision is
> on the remote branch?

Check 'git ls-remote'.

baruch

-- 
                                                     ~. .~   Tk Open Systems
=}------------------------------------------------ooO--U--Ooo------------{=
   - baruch@tkos.co.il - tel: +972.52.368.4656, http://www.tkos.co.il -
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] how to get buildroot to update a git repo that has changes? (ie building HEAD vs a specific SHA)

[Tim Harvey]

On Sat, May 7, 2022 at 1:07 PM Baruch Siach <baruch@tkos.co.il> wrote:
>
> Hi Tim,
>
> On Fri, May 06 2022, Tim Harvey wrote:
> > On Thu, May 5, 2022 at 8:21 PM Baruch Siach <baruch@tkos.co.il> wrote:
> >> On Thu, May 05 2022, Tim Harvey wrote:
> >> > Greetings,
> >> >
> >> > I use buildroot to build a Linux kernel+ramdisk using:
> >> > BR2_LINUX_KERNEL_CUSTOM_GIT=y
> >> > BR2_LINUX_KERNEL_CUSTOM_REPO_URL=<gitrepo>
> >> > BR2_LINUX_KERNEL_CUSTOM_REPO_VERSION=<gitbranch>
> >> >
> >> > When the upstream branch changes I would like to build that kernel but
> >> > I find that buildroot caches the repo in dl/linux/.git and doesn't
> >> > seem to know how to check to see if there are new commits.
> >> >
> >> > I realize that this is atypical as typically you want to configure
> >> > buildroot for a specific package gitsha/tag.
> >> >
> >> > How would I go about getting buildroot to update the repo? I don't
> >> > particularly want to rm dl/linux I want to instead somehow check to
> >> > see if the repo needs a refresh then refresh it without a massive
> >> > network download.
> >>
> >> Using a branch name for _VERSION is not supported. See the rationale in
> >> the manual section 18.6.2, note on LIBFOO_VERSION description.
> >>
> >>   https://buildroot.org/downloads/manual/manual.html
> >>
> >> You might want to consider using OVERRIDE_SRCDIR to make Buildroot build
> >> your local kernel source tree. See section 8.14.6 in the manual.
> >
> > Thanks for the references. I do understand that it is not supported. I
> > guess I was looking for a few simple lines of shell that would tell me
> > if the cached repo was out of date so that I could remove it.
> > # get branch from BR2 config
> > KERNEL_BRANCH=$(grep BR2_LINUX_KERNEL_CUSTOM_REPO_VERSION .config |
> > cut -d= -f2 | tr -d \")
> > # get the HEAD commit from the BR2 cache
> > KERNEL_HEAD=$(git -C dl/linux/git/ rev-parse $KERNEL_BRANCH)
> > ^^^ but I'm not sure how to check to see what the latest revision is
> > on the remote branch?
>
> Check 'git ls-remote'.

Thanks - thats what I needed.

I was able to add this to my auto-builder:
KERNEL_BRANCH=$(grep BR2_LINUX_KERNEL_CUSTOM_REPO_VERSION .config |
cut -d= -f2 | tr -d \")
KERNEL_HEAD=$(git -C dl/linux/git/ rev-parse $KERNEL_BRANCH)
REMOTE_HEAD=$(git -C dl/linux/git/ ls-remote --heads | grep
$KERNEL_BRANCH | cut -f1)
[ "$KERNEL_HEAD" == "$REMOTE_HEAD" ] || {
   echo removing kernel cache
   rm -rf dl/linux # otherwise won't get kernel bumps
}

Best Regards,

Tim
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] how to get buildroot to update a git repo that has changes? (ie building HEAD vs a specific SHA)

[Arnout Vandecappelle]

On 11/05/2022 17:28, Tim Harvey wrote:
> On Sat, May 7, 2022 at 1:07 PM Baruch Siach <baruch@tkos.co.il> wrote:
>>
>> Hi Tim,
>>
>> On Fri, May 06 2022, Tim Harvey wrote:
>>> On Thu, May 5, 2022 at 8:21 PM Baruch Siach <baruch@tkos.co.il> wrote:
>>>> On Thu, May 05 2022, Tim Harvey wrote:
>>>>> Greetings,
>>>>>
>>>>> I use buildroot to build a Linux kernel+ramdisk using:
>>>>> BR2_LINUX_KERNEL_CUSTOM_GIT=y
>>>>> BR2_LINUX_KERNEL_CUSTOM_REPO_URL=<gitrepo>
>>>>> BR2_LINUX_KERNEL_CUSTOM_REPO_VERSION=<gitbranch>
>>>>>
>>>>> When the upstream branch changes I would like to build that kernel but
>>>>> I find that buildroot caches the repo in dl/linux/.git and doesn't
>>>>> seem to know how to check to see if there are new commits.
>>>>>
>>>>> I realize that this is atypical as typically you want to configure
>>>>> buildroot for a specific package gitsha/tag.
>>>>>
>>>>> How would I go about getting buildroot to update the repo? I don't
>>>>> particularly want to rm dl/linux I want to instead somehow check to
>>>>> see if the repo needs a refresh then refresh it without a massive
>>>>> network download.
>>>>
>>>> Using a branch name for _VERSION is not supported. See the rationale in
>>>> the manual section 18.6.2, note on LIBFOO_VERSION description.
>>>>
>>>>    https://buildroot.org/downloads/manual/manual.html
>>>>
>>>> You might want to consider using OVERRIDE_SRCDIR to make Buildroot build
>>>> your local kernel source tree. See section 8.14.6 in the manual.
>>>
>>> Thanks for the references. I do understand that it is not supported. I
>>> guess I was looking for a few simple lines of shell that would tell me
>>> if the cached repo was out of date so that I could remove it.
>>> # get branch from BR2 config
>>> KERNEL_BRANCH=$(grep BR2_LINUX_KERNEL_CUSTOM_REPO_VERSION .config |
>>> cut -d= -f2 | tr -d \")
>>> # get the HEAD commit from the BR2 cache
>>> KERNEL_HEAD=$(git -C dl/linux/git/ rev-parse $KERNEL_BRANCH)
>>> ^^^ but I'm not sure how to check to see what the latest revision is
>>> on the remote branch?
>>
>> Check 'git ls-remote'.
> 
> Thanks - thats what I needed.
> 
> I was able to add this to my auto-builder:
> KERNEL_BRANCH=$(grep BR2_LINUX_KERNEL_CUSTOM_REPO_VERSION .config |
> cut -d= -f2 | tr -d \")
> KERNEL_HEAD=$(git -C dl/linux/git/ rev-parse $KERNEL_BRANCH)
> REMOTE_HEAD=$(git -C dl/linux/git/ ls-remote --heads | grep
> $KERNEL_BRANCH | cut -f1)
> [ "$KERNEL_HEAD" == "$REMOTE_HEAD" ] || {
>     echo removing kernel cache
>     rm -rf dl/linux # otherwise won't get kernel bumps

  You don't actually need to remove the git repository, it's enough to remove 
the tarball. At least, I think it is. So:

	rm -f dl/linux/linux-*-br1.tar.gz

(the -br1 indicates that it's a tarball created from git).
That's going to save you some download time.

  Regards,
  Arnout

> }
> 
> Best Regards,
> 
> Tim
> _______________________________________________
> buildroot mailing list
> buildroot@buildroot.org
> https://lists.buildroot.org/mailman/listinfo/buildroot
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot