* git fetcher and github pull requests @ 2018-02-07 15:04 Nicolas Dechesne 2018-02-15 7:40 ` Nicolas Dechesne 2018-02-15 10:59 ` Richard Purdie 0 siblings, 2 replies; 7+ messages in thread From: Nicolas Dechesne @ 2018-02-07 15:04 UTC (permalink / raw) To: bitbake-devel, Ryan Harkin hi there, we've been debugging an issue these days on our builder which ended up do_fetch failing with LANG=C git -c core.fsyncobjectfiles=0 fetch -f --prune --progress git://github.com/OP-TEE/optee_test.git refs/*:refs/* failed with exit code 128, output: error: Could not read 48e440f5f8d033e1ace6e41f424ecf6e6d96e5f2 error: Could not read 019a8db54beb29388e1108831d2e2dc135c1cd73 It happens that these refs correspond to pull requests done on github which existed at some point, but have been updated with newer commits, and won't exist anymore. the bitbake fetcher seems to be greedy and fetches refs/* which ends up fetching pull request when fetching from github, e.g. in my workspace: in $DL_DIR/git2/ github.com.docker.containerd.git/refs/pull/66 github.com.docker.containerd.git/refs/pull/459 github.com.docker.containerd.git/refs/pull/551 github.com.docker.containerd.git/refs/pull/321 github.com.docker.containerd.git/refs/pull/60 github.com.docker.containerd.git/refs/pull/523 github.com.docker.containerd.git/refs/pull/40 github.com.docker.containerd.git/refs/pull/561 It looks inefficient to fetch and store on each builder pull requests. I understand this is just because how PR are implemented in github, but github is quite central, so many we should/could do something about it? Beyond the inefficiencies, we now are seeing unrelated build issues as well. What do you think? Should we try to avoid fetching refs/pull/* from github? or is it our git fetch command that needs to be improve to work in this situation? thanks nico ^ permalink raw reply [flat|nested] 7+ messages in thread
* git fetcher and github pull requests 2018-02-07 15:04 git fetcher and github pull requests Nicolas Dechesne @ 2018-02-15 7:40 ` Nicolas Dechesne 2018-02-15 7:44 ` Alexander Kanavin 2018-02-15 10:59 ` Richard Purdie 1 sibling, 1 reply; 7+ messages in thread From: Nicolas Dechesne @ 2018-02-15 7:40 UTC (permalink / raw) To: Patches and discussions about the oe-core layer, Ryan Harkin hi, re-posting to oe-core list. While this is a bitbake question, I didn't get any answer.. so I would like to get more attention from bitbake users.. cheers nico ---------- Forwarded message ---------- From: Nicolas Dechesne <nicolas.dechesne@linaro.org> Date: Wed, Feb 7, 2018 at 4:04 PM Subject: git fetcher and github pull requests To: bitbake-devel <bitbake-devel@lists.openembedded.org>, Ryan Harkin <ryan.harkin@linaro.org> hi there, we've been debugging an issue these days on our builder which ended up do_fetch failing with LANG=C git -c core.fsyncobjectfiles=0 fetch -f --prune --progress git://github.com/OP-TEE/optee_test.git refs/*:refs/* failed with exit code 128, output: error: Could not read 48e440f5f8d033e1ace6e41f424ecf6e6d96e5f2 error: Could not read 019a8db54beb29388e1108831d2e2dc135c1cd73 It happens that these refs correspond to pull requests done on github which existed at some point, but have been updated with newer commits, and won't exist anymore. the bitbake fetcher seems to be greedy and fetches refs/* which ends up fetching pull request when fetching from github, e.g. in my workspace: in $DL_DIR/git2/ github.com.docker.containerd.git/refs/pull/66 github.com.docker.containerd.git/refs/pull/459 github.com.docker.containerd.git/refs/pull/551 github.com.docker.containerd.git/refs/pull/321 github.com.docker.containerd.git/refs/pull/60 github.com.docker.containerd.git/refs/pull/523 github.com.docker.containerd.git/refs/pull/40 github.com.docker.containerd.git/refs/pull/561 It looks inefficient to fetch and store on each builder pull requests. I understand this is just because how PR are implemented in github, but github is quite central, so many we should/could do something about it? Beyond the inefficiencies, we now are seeing unrelated build issues as well. What do you think? Should we try to avoid fetching refs/pull/* from github? or is it our git fetch command that needs to be improve to work in this situation? thanks nico ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: git fetcher and github pull requests 2018-02-15 7:40 ` Nicolas Dechesne @ 2018-02-15 7:44 ` Alexander Kanavin 0 siblings, 0 replies; 7+ messages in thread From: Alexander Kanavin @ 2018-02-15 7:44 UTC (permalink / raw) To: Nicolas Dechesne, Patches and discussions about the oe-core layer, Ryan Harkin On 02/15/2018 09:40 AM, Nicolas Dechesne wrote: > re-posting to oe-core list. While this is a bitbake question, I didn't > get any answer.. so I would like to get more attention from bitbake > users.. I do realize this is not answering your specific question, but from 'getting attention' perspective patches are always better than problem descriptions. If you can come up with something, please do. Alex ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: git fetcher and github pull requests 2018-02-07 15:04 git fetcher and github pull requests Nicolas Dechesne 2018-02-15 7:40 ` Nicolas Dechesne @ 2018-02-15 10:59 ` Richard Purdie 2018-02-15 11:19 ` Nicolas Dechesne 1 sibling, 1 reply; 7+ messages in thread From: Richard Purdie @ 2018-02-15 10:59 UTC (permalink / raw) To: Nicolas Dechesne, bitbake-devel, Ryan Harkin On Wed, 2018-02-07 at 16:04 +0100, Nicolas Dechesne wrote: > we've been debugging an issue these days on our builder which ended > up > do_fetch failing with > > LANG=C git -c core.fsyncobjectfiles=0 fetch -f --prune --progress > git://github.com/OP-TEE/optee_test.git refs/*:refs/* failed with exit > code 128, output: > error: Could not read 48e440f5f8d033e1ace6e41f424ecf6e6d96e5f2 > error: Could not read 019a8db54beb29388e1108831d2e2dc135c1cd73 > > It happens that these refs correspond to pull requests done on github > which existed at some point, but have been updated with newer > commits, > and won't exist anymore. > > the bitbake fetcher seems to be greedy and fetches refs/* which ends > up fetching pull request when fetching from github, e.g. in my > workspace: > > in $DL_DIR/git2/ > > github.com.docker.containerd.git/refs/pull/66 > github.com.docker.containerd.git/refs/pull/459 > github.com.docker.containerd.git/refs/pull/551 > github.com.docker.containerd.git/refs/pull/321 > github.com.docker.containerd.git/refs/pull/60 > github.com.docker.containerd.git/refs/pull/523 > github.com.docker.containerd.git/refs/pull/40 > github.com.docker.containerd.git/refs/pull/561 > > It looks inefficient to fetch and store on each builder pull > requests. > I understand this is just because how PR are implemented in github, > but github is quite central, so many we should/could do something > about it? > > Beyond the inefficiencies, we now are seeing unrelated build issues > as well. > > What do you think? Should we try to avoid fetching refs/pull/* from > github? or is it our git fetch command that needs to be improve to > work in this situation? The git fetcher deals with a complicated set of problems since it needs to handle offline mirroring (with mirror tarballs) and a variety of different scenarios. The reason its pulling a complete set of references is likely to because of the mirroring and other demands placed upon it from the different scenarios. This isn't bad as such, github really shouldn't be sharing repos with invalid references in them. I did read this when you sent it, I just don't have any good recommendation for you :( I dread "special casing" in the git fetcher as has burnt us a lot in the past, equally, that may be the only way to avoid this kind of problem... Cheers, Richard ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: git fetcher and github pull requests 2018-02-15 10:59 ` Richard Purdie @ 2018-02-15 11:19 ` Nicolas Dechesne 2018-02-15 12:02 ` Richard Purdie 0 siblings, 1 reply; 7+ messages in thread From: Nicolas Dechesne @ 2018-02-15 11:19 UTC (permalink / raw) To: Richard Purdie; +Cc: Ryan Harkin, bitbake-devel On Thu, Feb 15, 2018 at 11:59 AM, Richard Purdie <richard.purdie@linuxfoundation.org> wrote: > On Wed, 2018-02-07 at 16:04 +0100, Nicolas Dechesne wrote: >> we've been debugging an issue these days on our builder which ended >> up >> do_fetch failing with >> >> LANG=C git -c core.fsyncobjectfiles=0 fetch -f --prune --progress >> git://github.com/OP-TEE/optee_test.git refs/*:refs/* failed with exit >> code 128, output: >> error: Could not read 48e440f5f8d033e1ace6e41f424ecf6e6d96e5f2 >> error: Could not read 019a8db54beb29388e1108831d2e2dc135c1cd73 >> >> It happens that these refs correspond to pull requests done on github >> which existed at some point, but have been updated with newer >> commits, >> and won't exist anymore. >> >> the bitbake fetcher seems to be greedy and fetches refs/* which ends >> up fetching pull request when fetching from github, e.g. in my >> workspace: >> >> in $DL_DIR/git2/ >> >> github.com.docker.containerd.git/refs/pull/66 >> github.com.docker.containerd.git/refs/pull/459 >> github.com.docker.containerd.git/refs/pull/551 >> github.com.docker.containerd.git/refs/pull/321 >> github.com.docker.containerd.git/refs/pull/60 >> github.com.docker.containerd.git/refs/pull/523 >> github.com.docker.containerd.git/refs/pull/40 >> github.com.docker.containerd.git/refs/pull/561 >> >> It looks inefficient to fetch and store on each builder pull >> requests. >> I understand this is just because how PR are implemented in github, >> but github is quite central, so many we should/could do something >> about it? >> >> Beyond the inefficiencies, we now are seeing unrelated build issues >> as well. >> >> What do you think? Should we try to avoid fetching refs/pull/* from >> github? or is it our git fetch command that needs to be improve to >> work in this situation? > > The git fetcher deals with a complicated set of problems since it needs > to handle offline mirroring (with mirror tarballs) and a variety of > different scenarios. The reason its pulling a complete set of > references is likely to because of the mirroring and other demands > placed upon it from the different scenarios. This isn't bad as such, > github really shouldn't be sharing repos with invalid references in > them. the repos don't have any invalid references. But when pull requests are updated by users on github, new references are pushed, and we seem to be keeping old (invalid) and new references in our mirrors... > > I did read this when you sent it, I just don't have any good > recommendation for you :( the email is not a request for someone to fix.. just to get some feedback, and at least mention this issue on the list in case someone else hits it too... > > I dread "special casing" in the git fetcher as has burnt us a lot in > the past, equally, that may be the only way to avoid this kind of > problem... I will try to reproduce the issue with a simpler build/config. It could be that git is confused by github using refs as refs/pull.. > > Cheers, > > Richard > > > ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: git fetcher and github pull requests 2018-02-15 11:19 ` Nicolas Dechesne @ 2018-02-15 12:02 ` Richard Purdie 2018-02-15 13:11 ` Nicolas Dechesne 0 siblings, 1 reply; 7+ messages in thread From: Richard Purdie @ 2018-02-15 12:02 UTC (permalink / raw) To: Nicolas Dechesne; +Cc: Ryan Harkin, bitbake-devel On Thu, 2018-02-15 at 12:19 +0100, Nicolas Dechesne wrote: > On Thu, Feb 15, 2018 at 11:59 AM, Richard Purdie > <richard.purdie@linuxfoundation.org> wrote: > > > > On Wed, 2018-02-07 at 16:04 +0100, Nicolas Dechesne wrote: > > > > > > we've been debugging an issue these days on our builder which > > > ended > > > up > > > do_fetch failing with > > > > > > LANG=C git -c core.fsyncobjectfiles=0 fetch -f --prune --progress > > > git://github.com/OP-TEE/optee_test.git refs/*:refs/* failed with > > > exit > > > code 128, output: > > > error: Could not read 48e440f5f8d033e1ace6e41f424ecf6e6d96e5f2 > > > error: Could not read 019a8db54beb29388e1108831d2e2dc135c1cd73 > > > > > > It happens that these refs correspond to pull requests done on > > > github > > > which existed at some point, but have been updated with newer > > > commits, > > > and won't exist anymore. > > > > > > the bitbake fetcher seems to be greedy and fetches refs/* which > > > ends > > > up fetching pull request when fetching from github, e.g. in my > > > workspace: > > > > > > in $DL_DIR/git2/ > > > > > > github.com.docker.containerd.git/refs/pull/66 > > > github.com.docker.containerd.git/refs/pull/459 > > > github.com.docker.containerd.git/refs/pull/551 > > > github.com.docker.containerd.git/refs/pull/321 > > > github.com.docker.containerd.git/refs/pull/60 > > > github.com.docker.containerd.git/refs/pull/523 > > > github.com.docker.containerd.git/refs/pull/40 > > > github.com.docker.containerd.git/refs/pull/561 > > > > > > It looks inefficient to fetch and store on each builder pull > > > requests. > > > I understand this is just because how PR are implemented in > > > github, > > > but github is quite central, so many we should/could do something > > > about it? > > > > > > Beyond the inefficiencies, we now are seeing unrelated build > > > issues > > > as well. > > > > > > What do you think? Should we try to avoid fetching refs/pull/* > > > from > > > github? or is it our git fetch command that needs to be improve > > > to > > > work in this situation? > > The git fetcher deals with a complicated set of problems since it > > needs > > to handle offline mirroring (with mirror tarballs) and a variety of > > different scenarios. The reason its pulling a complete set of > > references is likely to because of the mirroring and other demands > > placed upon it from the different scenarios. This isn't bad as > > such, > > github really shouldn't be sharing repos with invalid references in > > them. > the repos don't have any invalid references. But when pull requests > are updated by users on github, new references are pushed, and we > seem > to be keeping old (invalid) and new references in our mirrors... This isn't making sense to me :/. Can you share a tarball of a repo where the above command gives the reference errors you mention? I'd like to have a closer look at what is happening... Or is there another way to reproduce the error? Cheers, Richard ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: git fetcher and github pull requests 2018-02-15 12:02 ` Richard Purdie @ 2018-02-15 13:11 ` Nicolas Dechesne 0 siblings, 0 replies; 7+ messages in thread From: Nicolas Dechesne @ 2018-02-15 13:11 UTC (permalink / raw) To: Richard Purdie; +Cc: Ryan Harkin, bitbake-devel On Thu, Feb 15, 2018 at 1:02 PM, Richard Purdie <richard.purdie@linuxfoundation.org> wrote: > On Thu, 2018-02-15 at 12:19 +0100, Nicolas Dechesne wrote: >> On Thu, Feb 15, 2018 at 11:59 AM, Richard Purdie >> <richard.purdie@linuxfoundation.org> wrote: >> > >> > On Wed, 2018-02-07 at 16:04 +0100, Nicolas Dechesne wrote: >> > > >> > > we've been debugging an issue these days on our builder which >> > > ended >> > > up >> > > do_fetch failing with >> > > >> > > LANG=C git -c core.fsyncobjectfiles=0 fetch -f --prune --progress >> > > git://github.com/OP-TEE/optee_test.git refs/*:refs/* failed with >> > > exit >> > > code 128, output: >> > > error: Could not read 48e440f5f8d033e1ace6e41f424ecf6e6d96e5f2 >> > > error: Could not read 019a8db54beb29388e1108831d2e2dc135c1cd73 >> > > >> > > It happens that these refs correspond to pull requests done on >> > > github >> > > which existed at some point, but have been updated with newer >> > > commits, >> > > and won't exist anymore. >> > > >> > > the bitbake fetcher seems to be greedy and fetches refs/* which >> > > ends >> > > up fetching pull request when fetching from github, e.g. in my >> > > workspace: >> > > >> > > in $DL_DIR/git2/ >> > > >> > > github.com.docker.containerd.git/refs/pull/66 >> > > github.com.docker.containerd.git/refs/pull/459 >> > > github.com.docker.containerd.git/refs/pull/551 >> > > github.com.docker.containerd.git/refs/pull/321 >> > > github.com.docker.containerd.git/refs/pull/60 >> > > github.com.docker.containerd.git/refs/pull/523 >> > > github.com.docker.containerd.git/refs/pull/40 >> > > github.com.docker.containerd.git/refs/pull/561 >> > > >> > > It looks inefficient to fetch and store on each builder pull >> > > requests. >> > > I understand this is just because how PR are implemented in >> > > github, >> > > but github is quite central, so many we should/could do something >> > > about it? >> > > >> > > Beyond the inefficiencies, we now are seeing unrelated build >> > > issues >> > > as well. >> > > >> > > What do you think? Should we try to avoid fetching refs/pull/* >> > > from >> > > github? or is it our git fetch command that needs to be improve >> > > to >> > > work in this situation? >> > The git fetcher deals with a complicated set of problems since it >> > needs >> > to handle offline mirroring (with mirror tarballs) and a variety of >> > different scenarios. The reason its pulling a complete set of >> > references is likely to because of the mirroring and other demands >> > placed upon it from the different scenarios. This isn't bad as >> > such, >> > github really shouldn't be sharing repos with invalid references in >> > them. >> the repos don't have any invalid references. But when pull requests >> are updated by users on github, new references are pushed, and we >> seem >> to be keeping old (invalid) and new references in our mirrors... > > This isn't making sense to me :/. well. that's the reason why i sent this email in the first place, it doesn't seem to make any sense! > > Can you share a tarball of a repo where the above command gives the > reference errors you mention? I'd like to have a closer look at what is > happening... > > Or is there another way to reproduce the error? right now, i can't, it was on our builders, and the fix was to delete the git tarballs in DL_DIR... the commit mentioned in the do_fetch() error correspond to github pull request that have been submitted and eventually merged using github interface with 'rebase and merge' option. the sha1 that bitbake complains about corresponds to the sha1 of the initial pull request before it was rebased and merged in the tree. I will try to reproduce with a fake github project. > > Cheers, > > Richard ^ permalink raw reply [flat|nested] 7+ messages in thread
end of thread, other threads:[~2018-02-15 13:11 UTC | newest] Thread overview: 7+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2018-02-07 15:04 git fetcher and github pull requests Nicolas Dechesne 2018-02-15 7:40 ` Nicolas Dechesne 2018-02-15 7:44 ` Alexander Kanavin 2018-02-15 10:59 ` Richard Purdie 2018-02-15 11:19 ` Nicolas Dechesne 2018-02-15 12:02 ` Richard Purdie 2018-02-15 13:11 ` Nicolas Dechesne
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.