git fetcher and github pull requests

git fetcher and github pull requests

[Nicolas Dechesne]

hi there,

we've been debugging an issue these days on our builder which ended up
 do_fetch failing with

LANG=C git -c core.fsyncobjectfiles=0 fetch -f --prune --progress
git://github.com/OP-TEE/optee_test.git refs/*:refs/* failed with exit
code 128, output:
error: Could not read 48e440f5f8d033e1ace6e41f424ecf6e6d96e5f2
error: Could not read 019a8db54beb29388e1108831d2e2dc135c1cd73

It happens that these refs correspond to pull requests done on github
which existed at some point, but have been updated with newer commits,
and won't exist anymore.

the bitbake fetcher seems to be greedy and fetches refs/* which ends
up fetching pull request when fetching from github, e.g. in my
workspace:

in $DL_DIR/git2/

github.com.docker.containerd.git/refs/pull/66
github.com.docker.containerd.git/refs/pull/459
github.com.docker.containerd.git/refs/pull/551
github.com.docker.containerd.git/refs/pull/321
github.com.docker.containerd.git/refs/pull/60
github.com.docker.containerd.git/refs/pull/523
github.com.docker.containerd.git/refs/pull/40
github.com.docker.containerd.git/refs/pull/561

It looks inefficient to fetch and store on each builder pull requests.
I understand this is just because how PR are implemented in github,
but github is quite central, so many we should/could do something
about it?

Beyond the inefficiencies, we now are seeing unrelated build issues as well.

What do you think? Should we try to avoid fetching refs/pull/* from
github? or is it our git fetch command that needs to be improve to
work in this situation?

thanks
nico

git fetcher and github pull requests

[Nicolas Dechesne]

hi,

re-posting to oe-core list. While this is a bitbake question, I didn't
get any answer.. so I would like to get more attention from bitbake
users..

cheers
nico


---------- Forwarded message ----------
From: Nicolas Dechesne <nicolas.dechesne@linaro.org>
Date: Wed, Feb 7, 2018 at 4:04 PM
Subject: git fetcher and github pull requests
To: bitbake-devel <bitbake-devel@lists.openembedded.org>, Ryan Harkin
<ryan.harkin@linaro.org>


hi there,

we've been debugging an issue these days on our builder which ended up
 do_fetch failing with

LANG=C git -c core.fsyncobjectfiles=0 fetch -f --prune --progress
git://github.com/OP-TEE/optee_test.git refs/*:refs/* failed with exit
code 128, output:
error: Could not read 48e440f5f8d033e1ace6e41f424ecf6e6d96e5f2
error: Could not read 019a8db54beb29388e1108831d2e2dc135c1cd73

It happens that these refs correspond to pull requests done on github
which existed at some point, but have been updated with newer commits,
and won't exist anymore.

the bitbake fetcher seems to be greedy and fetches refs/* which ends
up fetching pull request when fetching from github, e.g. in my
workspace:

in $DL_DIR/git2/

github.com.docker.containerd.git/refs/pull/66
github.com.docker.containerd.git/refs/pull/459
github.com.docker.containerd.git/refs/pull/551
github.com.docker.containerd.git/refs/pull/321
github.com.docker.containerd.git/refs/pull/60
github.com.docker.containerd.git/refs/pull/523
github.com.docker.containerd.git/refs/pull/40
github.com.docker.containerd.git/refs/pull/561

It looks inefficient to fetch and store on each builder pull requests.
I understand this is just because how PR are implemented in github,
but github is quite central, so many we should/could do something
about it?

Beyond the inefficiencies, we now are seeing unrelated build issues as well.

What do you think? Should we try to avoid fetching refs/pull/* from
github? or is it our git fetch command that needs to be improve to
work in this situation?

thanks
nico

Re: git fetcher and github pull requests

[Alexander Kanavin]

On 02/15/2018 09:40 AM, Nicolas Dechesne wrote:
> re-posting to oe-core list. While this is a bitbake question, I didn't
> get any answer.. so I would like to get more attention from bitbake
> users..

I do realize this is not answering your specific question, but from 
'getting attention' perspective patches are always better than problem 
descriptions. If you can come up with something, please do.

Alex

Re: git fetcher and github pull requests

[Richard Purdie]

On Wed, 2018-02-07 at 16:04 +0100, Nicolas Dechesne wrote:
> we've been debugging an issue these days on our builder which ended
> up
>  do_fetch failing with
> 
> LANG=C git -c core.fsyncobjectfiles=0 fetch -f --prune --progress
> git://github.com/OP-TEE/optee_test.git refs/*:refs/* failed with exit
> code 128, output:
> error: Could not read 48e440f5f8d033e1ace6e41f424ecf6e6d96e5f2
> error: Could not read 019a8db54beb29388e1108831d2e2dc135c1cd73
> 
> It happens that these refs correspond to pull requests done on github
> which existed at some point, but have been updated with newer
> commits,
> and won't exist anymore.
> 
> the bitbake fetcher seems to be greedy and fetches refs/* which ends
> up fetching pull request when fetching from github, e.g. in my
> workspace:
> 
> in $DL_DIR/git2/
> 
> github.com.docker.containerd.git/refs/pull/66
> github.com.docker.containerd.git/refs/pull/459
> github.com.docker.containerd.git/refs/pull/551
> github.com.docker.containerd.git/refs/pull/321
> github.com.docker.containerd.git/refs/pull/60
> github.com.docker.containerd.git/refs/pull/523
> github.com.docker.containerd.git/refs/pull/40
> github.com.docker.containerd.git/refs/pull/561
> 
> It looks inefficient to fetch and store on each builder pull
> requests.
> I understand this is just because how PR are implemented in github,
> but github is quite central, so many we should/could do something
> about it?
> 
> Beyond the inefficiencies, we now are seeing unrelated build issues
> as well.
> 
> What do you think? Should we try to avoid fetching refs/pull/* from
> github? or is it our git fetch command that needs to be improve to
> work in this situation?

The git fetcher deals with a complicated set of problems since it needs
to handle offline mirroring (with mirror tarballs) and a variety of
different scenarios. The reason its pulling a complete set of
references is likely to because of the mirroring and other demands
placed upon it from the different scenarios. This isn't bad as such,
github really shouldn't be sharing repos with invalid references in
them.

I did read this when you sent it, I just don't have any good
recommendation for you :(

I dread "special casing" in the git fetcher as has burnt us a lot in
the past, equally, that may be the only way to avoid this kind of
problem...

Cheers,

Richard

Re: git fetcher and github pull requests

[Nicolas Dechesne]

On Thu, Feb 15, 2018 at 11:59 AM, Richard Purdie
<richard.purdie@linuxfoundation.org> wrote:
> On Wed, 2018-02-07 at 16:04 +0100, Nicolas Dechesne wrote:
>> we've been debugging an issue these days on our builder which ended
>> up
>>  do_fetch failing with
>>
>> LANG=C git -c core.fsyncobjectfiles=0 fetch -f --prune --progress
>> git://github.com/OP-TEE/optee_test.git refs/*:refs/* failed with exit
>> code 128, output:
>> error: Could not read 48e440f5f8d033e1ace6e41f424ecf6e6d96e5f2
>> error: Could not read 019a8db54beb29388e1108831d2e2dc135c1cd73
>>
>> It happens that these refs correspond to pull requests done on github
>> which existed at some point, but have been updated with newer
>> commits,
>> and won't exist anymore.
>>
>> the bitbake fetcher seems to be greedy and fetches refs/* which ends
>> up fetching pull request when fetching from github, e.g. in my
>> workspace:
>>
>> in $DL_DIR/git2/
>>
>> github.com.docker.containerd.git/refs/pull/66
>> github.com.docker.containerd.git/refs/pull/459
>> github.com.docker.containerd.git/refs/pull/551
>> github.com.docker.containerd.git/refs/pull/321
>> github.com.docker.containerd.git/refs/pull/60
>> github.com.docker.containerd.git/refs/pull/523
>> github.com.docker.containerd.git/refs/pull/40
>> github.com.docker.containerd.git/refs/pull/561
>>
>> It looks inefficient to fetch and store on each builder pull
>> requests.
>> I understand this is just because how PR are implemented in github,
>> but github is quite central, so many we should/could do something
>> about it?
>>
>> Beyond the inefficiencies, we now are seeing unrelated build issues
>> as well.
>>
>> What do you think? Should we try to avoid fetching refs/pull/* from
>> github? or is it our git fetch command that needs to be improve to
>> work in this situation?
>
> The git fetcher deals with a complicated set of problems since it needs
> to handle offline mirroring (with mirror tarballs) and a variety of
> different scenarios. The reason its pulling a complete set of
> references is likely to because of the mirroring and other demands
> placed upon it from the different scenarios. This isn't bad as such,
> github really shouldn't be sharing repos with invalid references in
> them.

the repos don't have any invalid references. But when pull requests
are updated by users on github, new references are pushed, and we seem
to be keeping old (invalid) and new references in our mirrors...

>
> I did read this when you sent it, I just don't have any good
> recommendation for you :(

the email is not a request for someone to fix.. just to get some
feedback, and at least mention this issue on the list in case someone
else hits it too...

>
> I dread "special casing" in the git fetcher as has burnt us a lot in
> the past, equally, that may be the only way to avoid this kind of
> problem...

I will try to reproduce the issue with a simpler build/config. It
could be that git is confused by github using refs as refs/pull..

>
> Cheers,
>
> Richard
>
>
>

Re: git fetcher and github pull requests

[Richard Purdie]

On Thu, 2018-02-15 at 12:19 +0100, Nicolas Dechesne wrote:
> On Thu, Feb 15, 2018 at 11:59 AM, Richard Purdie
> <richard.purdie@linuxfoundation.org> wrote:
> > 
> > On Wed, 2018-02-07 at 16:04 +0100, Nicolas Dechesne wrote:
> > > 
> > > we've been debugging an issue these days on our builder which
> > > ended
> > > up
> > >  do_fetch failing with
> > > 
> > > LANG=C git -c core.fsyncobjectfiles=0 fetch -f --prune --progress
> > > git://github.com/OP-TEE/optee_test.git refs/*:refs/* failed with
> > > exit
> > > code 128, output:
> > > error: Could not read 48e440f5f8d033e1ace6e41f424ecf6e6d96e5f2
> > > error: Could not read 019a8db54beb29388e1108831d2e2dc135c1cd73
> > > 
> > > It happens that these refs correspond to pull requests done on
> > > github
> > > which existed at some point, but have been updated with newer
> > > commits,
> > > and won't exist anymore.
> > > 
> > > the bitbake fetcher seems to be greedy and fetches refs/* which
> > > ends
> > > up fetching pull request when fetching from github, e.g. in my
> > > workspace:
> > > 
> > > in $DL_DIR/git2/
> > > 
> > > github.com.docker.containerd.git/refs/pull/66
> > > github.com.docker.containerd.git/refs/pull/459
> > > github.com.docker.containerd.git/refs/pull/551
> > > github.com.docker.containerd.git/refs/pull/321
> > > github.com.docker.containerd.git/refs/pull/60
> > > github.com.docker.containerd.git/refs/pull/523
> > > github.com.docker.containerd.git/refs/pull/40
> > > github.com.docker.containerd.git/refs/pull/561
> > > 
> > > It looks inefficient to fetch and store on each builder pull
> > > requests.
> > > I understand this is just because how PR are implemented in
> > > github,
> > > but github is quite central, so many we should/could do something
> > > about it?
> > > 
> > > Beyond the inefficiencies, we now are seeing unrelated build
> > > issues
> > > as well.
> > > 
> > > What do you think? Should we try to avoid fetching refs/pull/*
> > > from
> > > github? or is it our git fetch command that needs to be improve
> > > to
> > > work in this situation?
> > The git fetcher deals with a complicated set of problems since it
> > needs
> > to handle offline mirroring (with mirror tarballs) and a variety of
> > different scenarios. The reason its pulling a complete set of
> > references is likely to because of the mirroring and other demands
> > placed upon it from the different scenarios. This isn't bad as
> > such,
> > github really shouldn't be sharing repos with invalid references in
> > them.
> the repos don't have any invalid references. But when pull requests
> are updated by users on github, new references are pushed, and we
> seem
> to be keeping old (invalid) and new references in our mirrors...

This isn't making sense to me :/.

Can you share a tarball of a repo where the above command gives the
reference errors you mention? I'd like to have a closer look at what is
happening...

Or is there another way to reproduce the error?

Cheers,

Richard

Re: git fetcher and github pull requests

[Nicolas Dechesne]

On Thu, Feb 15, 2018 at 1:02 PM, Richard Purdie
<richard.purdie@linuxfoundation.org> wrote:
> On Thu, 2018-02-15 at 12:19 +0100, Nicolas Dechesne wrote:
>> On Thu, Feb 15, 2018 at 11:59 AM, Richard Purdie
>> <richard.purdie@linuxfoundation.org> wrote:
>> >
>> > On Wed, 2018-02-07 at 16:04 +0100, Nicolas Dechesne wrote:
>> > >
>> > > we've been debugging an issue these days on our builder which
>> > > ended
>> > > up
>> > >  do_fetch failing with
>> > >
>> > > LANG=C git -c core.fsyncobjectfiles=0 fetch -f --prune --progress
>> > > git://github.com/OP-TEE/optee_test.git refs/*:refs/* failed with
>> > > exit
>> > > code 128, output:
>> > > error: Could not read 48e440f5f8d033e1ace6e41f424ecf6e6d96e5f2
>> > > error: Could not read 019a8db54beb29388e1108831d2e2dc135c1cd73
>> > >
>> > > It happens that these refs correspond to pull requests done on
>> > > github
>> > > which existed at some point, but have been updated with newer
>> > > commits,
>> > > and won't exist anymore.
>> > >
>> > > the bitbake fetcher seems to be greedy and fetches refs/* which
>> > > ends
>> > > up fetching pull request when fetching from github, e.g. in my
>> > > workspace:
>> > >
>> > > in $DL_DIR/git2/
>> > >
>> > > github.com.docker.containerd.git/refs/pull/66
>> > > github.com.docker.containerd.git/refs/pull/459
>> > > github.com.docker.containerd.git/refs/pull/551
>> > > github.com.docker.containerd.git/refs/pull/321
>> > > github.com.docker.containerd.git/refs/pull/60
>> > > github.com.docker.containerd.git/refs/pull/523
>> > > github.com.docker.containerd.git/refs/pull/40
>> > > github.com.docker.containerd.git/refs/pull/561
>> > >
>> > > It looks inefficient to fetch and store on each builder pull
>> > > requests.
>> > > I understand this is just because how PR are implemented in
>> > > github,
>> > > but github is quite central, so many we should/could do something
>> > > about it?
>> > >
>> > > Beyond the inefficiencies, we now are seeing unrelated build
>> > > issues
>> > > as well.
>> > >
>> > > What do you think? Should we try to avoid fetching refs/pull/*
>> > > from
>> > > github? or is it our git fetch command that needs to be improve
>> > > to
>> > > work in this situation?
>> > The git fetcher deals with a complicated set of problems since it
>> > needs
>> > to handle offline mirroring (with mirror tarballs) and a variety of
>> > different scenarios. The reason its pulling a complete set of
>> > references is likely to because of the mirroring and other demands
>> > placed upon it from the different scenarios. This isn't bad as
>> > such,
>> > github really shouldn't be sharing repos with invalid references in
>> > them.
>> the repos don't have any invalid references. But when pull requests
>> are updated by users on github, new references are pushed, and we
>> seem
>> to be keeping old (invalid) and new references in our mirrors...
>
> This isn't making sense to me :/.

well. that's the reason why i sent this email in the first place, it
doesn't seem to make any sense!

>
> Can you share a tarball of a repo where the above command gives the
> reference errors you mention? I'd like to have a closer look at what is
> happening...
>
> Or is there another way to reproduce the error?

right now, i can't, it was on our builders, and the fix was to delete
the git tarballs in DL_DIR...

the commit mentioned in the do_fetch() error correspond to github pull
request that have been submitted and eventually merged using github
interface with 'rebase and merge' option. the sha1 that bitbake
complains about corresponds to the sha1 of the initial pull request
before it was rebased and merged in the tree.

I will try to reproduce with a fake github project.

>
> Cheers,
>
> Richard