* openpower.xyz updates
@ 2016-09-26 11:07 Joel Stanley
[not found] ` <CALLMt=rbR1usfKPt9G8kdUa2RaqRaaVEG3rV62wSBjaM59qi6Q@mail.gmail.com>
0 siblings, 1 reply; 16+ messages in thread
From: Joel Stanley @ 2016-09-26 11:07 UTC (permalink / raw)
To: OpenBMC Maillist
Today I performed some upgrades on openpower.xyz:
- Upgraded from Ubuntu 14.04 LTS to 16.04 LTS
- Moved from the one-year free TLS certificate to LetsEncrypt
- Updated Jenkins from 10 month old version to the latest (2.7.4)
Things appear to be working okay. Please let me know if you notice any
breakages.
Cheers,
Joel
^ permalink raw reply [flat|nested] 16+ messages in thread[parent not found: <CALLMt=rbR1usfKPt9G8kdUa2RaqRaaVEG3rV62wSBjaM59qi6Q@mail.gmail.com>]
* Re: openpower.xyz updates [not found] ` <CALLMt=rbR1usfKPt9G8kdUa2RaqRaaVEG3rV62wSBjaM59qi6Q@mail.gmail.com> @ 2016-09-30 1:07 ` Joel Stanley 2016-09-30 1:38 ` Patrick Williams 0 siblings, 1 reply; 16+ messages in thread From: Joel Stanley @ 2016-09-30 1:07 UTC (permalink / raw) To: Andrew Geissler; +Cc: OpenBMC Maillist On Wed, Sep 28, 2016 at 10:08 PM, Andrew Geissler <geissonator@gmail.com> wrote: > Hey Joel, late yesterday afternoon Patrick reported that the gerrit CI > jobs were not triggering. I took a look and found that our job that > does ID validation (which triggers on patch set creation) was still > working fine but the follow up job that would trigger on the > "Ok-To-Test" comment was no longer triggering. I noticed that many of the gerrit reviews I have pending do not have the OOT or Verified checked. Is the system still having issues? > I've seen this a few times and it's usually always incompatibilities > between the gerrit and git plugins and/or the jenkins version. Since > it was blocking all dev work and we're getting near the sprint end I > jumped on and just had to downlevel the gerrit and git plugins by one > level to get things working again. > > Specifically, I had to downgrade our gerrit plugin to 2.21.1 and our > git client plugin to 1.19.7 Thanks for letting me know. Do you have a plan for fixing the issue so we can run up to date versions of the gerrit plugins? Cheers, Joel > > Andrew > > On Mon, Sep 26, 2016 at 6:07 AM, Joel Stanley <joel@jms.id.au> wrote: >> Today I performed some upgrades on openpower.xyz: >> >> - Upgraded from Ubuntu 14.04 LTS to 16.04 LTS >> - Moved from the one-year free TLS certificate to LetsEncrypt >> - Updated Jenkins from 10 month old version to the latest (2.7.4) >> >> Things appear to be working okay. Please let me know if you notice any >> breakages. >> >> Cheers, >> >> Joel ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: openpower.xyz updates 2016-09-30 1:07 ` Joel Stanley @ 2016-09-30 1:38 ` Patrick Williams 2016-09-30 1:46 ` Joel Stanley 0 siblings, 1 reply; 16+ messages in thread From: Patrick Williams @ 2016-09-30 1:38 UTC (permalink / raw) To: Joel Stanley; +Cc: Andrew Geissler, OpenBMC Maillist [-- Attachment #1: Type: text/plain, Size: 871 bytes --] On Fri, Sep 30, 2016 at 10:37:42AM +0930, Joel Stanley wrote: > On Wed, Sep 28, 2016 at 10:08 PM, Andrew Geissler <geissonator@gmail.com> wrote: > > Hey Joel, late yesterday afternoon Patrick reported that the gerrit CI > > jobs were not triggering. I took a look and found that our job that > > does ID validation (which triggers on patch set creation) was still > > working fine but the follow up job that would trigger on the > > "Ok-To-Test" comment was no longer triggering. > > I noticed that many of the gerrit reviews I have pending do not have > the OOT or Verified checked. Is the system still having issues? Currently, the only testing is done on the openbmc/openbmc repository. According to this query they are all set with both OTT and V. https://gerrit.openbmc-project.xyz/#/q/project:openbmc/openbmc+status:open -- Patrick Williams [-- Attachment #2: Digital signature --] [-- Type: application/pgp-signature, Size: 819 bytes --] ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: openpower.xyz updates 2016-09-30 1:38 ` Patrick Williams @ 2016-09-30 1:46 ` Joel Stanley 2016-09-30 13:12 ` Patrick Williams 0 siblings, 1 reply; 16+ messages in thread From: Joel Stanley @ 2016-09-30 1:46 UTC (permalink / raw) To: Patrick Williams; +Cc: Andrew Geissler, OpenBMC Maillist On Fri, Sep 30, 2016 at 11:08 AM, Patrick Williams <patrick@stwcx.xyz> wrote: > On Fri, Sep 30, 2016 at 10:37:42AM +0930, Joel Stanley wrote: >> On Wed, Sep 28, 2016 at 10:08 PM, Andrew Geissler <geissonator@gmail.com> wrote: >> > Hey Joel, late yesterday afternoon Patrick reported that the gerrit CI >> > jobs were not triggering. I took a look and found that our job that >> > does ID validation (which triggers on patch set creation) was still >> > working fine but the follow up job that would trigger on the >> > "Ok-To-Test" comment was no longer triggering. >> >> I noticed that many of the gerrit reviews I have pending do not have >> the OOT or Verified checked. Is the system still having issues? > > Currently, the only testing is done on the openbmc/openbmc repository. > According to this query they are all set with both OTT and V. > > https://gerrit.openbmc-project.xyz/#/q/project:openbmc/openbmc+status:open Okay. Do we plan to test the individual projects as part of an openbmc/openbmc build in the future? Cheers, Joel ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: openpower.xyz updates 2016-09-30 1:46 ` Joel Stanley @ 2016-09-30 13:12 ` Patrick Williams 2016-10-10 16:09 ` Andrew Geissler 0 siblings, 1 reply; 16+ messages in thread From: Patrick Williams @ 2016-09-30 13:12 UTC (permalink / raw) To: Joel Stanley; +Cc: Andrew Geissler, OpenBMC Maillist [-- Attachment #1: Type: text/plain, Size: 253 bytes --] On Fri, Sep 30, 2016 at 11:16:08AM +0930, Joel Stanley wrote: > Okay. Do we plan to test the individual projects as part of an > openbmc/openbmc build in the future? > Yes. https://github.com/openbmc/openbmc/issues/382 -- Patrick Williams [-- Attachment #2: Digital signature --] [-- Type: application/pgp-signature, Size: 819 bytes --] ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: openpower.xyz updates 2016-09-30 13:12 ` Patrick Williams @ 2016-10-10 16:09 ` Andrew Geissler 2016-10-10 23:44 ` Andrew Jeffery 0 siblings, 1 reply; 16+ messages in thread From: Andrew Geissler @ 2016-10-10 16:09 UTC (permalink / raw) To: Patrick Williams; +Cc: Joel Stanley, OpenBMC Maillist I noticed our gerrit trigger job to verify userid's was broken again this morning. Looks like the trouble plugin (Git Client Plugin) was upgraded again to the latest version. Anyone out there know why/who did it? I'm going to put us back to 1.19.7 per my earlier comments. There's no fix per-se for this, jenkins is always a bit hit or miss with it's latest LTS version working with the latest plugins. Best thing we can do is when we schedule our next jenkins LTS upgrade, to try all the latest plugins again. Andrew On Fri, Sep 30, 2016 at 8:12 AM, Patrick Williams <patrick@stwcx.xyz> wrote: > On Fri, Sep 30, 2016 at 11:16:08AM +0930, Joel Stanley wrote: >> Okay. Do we plan to test the individual projects as part of an >> openbmc/openbmc build in the future? >> > > Yes. https://github.com/openbmc/openbmc/issues/382 > > -- > Patrick Williams ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: openpower.xyz updates 2016-10-10 16:09 ` Andrew Geissler @ 2016-10-10 23:44 ` Andrew Jeffery 2016-10-10 23:49 ` Joel Stanley 2016-10-11 1:36 ` Chris Smart 0 siblings, 2 replies; 16+ messages in thread From: Andrew Jeffery @ 2016-10-10 23:44 UTC (permalink / raw) To: Andrew Geissler, Patrick Williams Cc: OpenBMC Maillist, Joel Stanley, Chris Smart [-- Attachment #1: Type: text/plain, Size: 1594 bytes --] On Mon, 2016-10-10 at 11:09 -0500, Andrew Geissler wrote: > I noticed our gerrit trigger job to verify userid's was broken again > this morning. Looks like the trouble plugin (Git Client Plugin) was > upgraded again to the latest version. Anyone out there know why/who > did it? I'm going to put us back to 1.19.7 per my earlier comments. Joel and Chris Smart were looking at this yesterday. A job broke: https://openpower.xyz/job/openpower-op-build/184/console According to Chris it broke because of: https://bugs.eclipse.org/bugs/show_bug.cgi?id=428039 15:58 < csmart> joel, basically release() was replaced with close() and so any plugin that uses git needs to be updated Looks like we're stuck between a rock and a hard place. Cheers, Andrew > > There's no fix per-se for this, jenkins is always a bit hit or miss > with it's latest LTS version working with the latest plugins. Best > thing we can do is when we schedule our next jenkins LTS upgrade, to > try all the latest plugins again. > > Andrew > > On Fri, Sep 30, 2016 at 8:12 AM, Patrick Williams <patrick@stwcx.xyz> wrote: > > > > On Fri, Sep 30, 2016 at 11:16:08AM +0930, Joel Stanley wrote: > > > > > > Okay. Do we plan to test the individual projects as part of an > > > openbmc/openbmc build in the future? > > > > > Yes. https://github.com/openbmc/openbmc/issues/382 > > > > -- > > Patrick Williams > _______________________________________________ > openbmc mailing list > openbmc@lists.ozlabs.org > https://lists.ozlabs.org/listinfo/openbmc [-- Attachment #2: This is a digitally signed message part --] [-- Type: application/pgp-signature, Size: 819 bytes --] ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: openpower.xyz updates 2016-10-10 23:44 ` Andrew Jeffery @ 2016-10-10 23:49 ` Joel Stanley 2016-10-11 1:36 ` Chris Smart 1 sibling, 0 replies; 16+ messages in thread From: Joel Stanley @ 2016-10-10 23:49 UTC (permalink / raw) To: Andrew Jeffery Cc: Andrew Geissler, Patrick Williams, OpenBMC Maillist, Chris Smart On Tue, Oct 11, 2016 at 10:44 AM, Andrew Jeffery <andrew@aj.id.au> wrote: > On Mon, 2016-10-10 at 11:09 -0500, Andrew Geissler wrote: >> I noticed our gerrit trigger job to verify userid's was broken again >> this morning. Looks like the trouble plugin (Git Client Plugin) was >> upgraded again to the latest version. Anyone out there know why/who >> did it? I'm going to put us back to 1.19.7 per my earlier comments. > > Joel and Chris Smart were looking at this yesterday. A job broke: > > https://openpower.xyz/job/openpower-op-build/184/console > > According to Chris it broke because of: > > https://bugs.eclipse.org/bugs/show_bug.cgi?id=428039 > > 15:58 < csmart> joel, basically release() was replaced with close() > and so any plugin that uses git needs to be updated Yep. Andrew G, do you have a bug report for the issue you're having? We need to find a solution that lets us stay up to date with fixes from Jenkins. Cheers, Joel > > Looks like we're stuck between a rock and a hard place. > > Cheers, > > Andrew > >> >> There's no fix per-se for this, jenkins is always a bit hit or miss >> with it's latest LTS version working with the latest plugins. Best >> thing we can do is when we schedule our next jenkins LTS upgrade, to >> try all the latest plugins again. >> >> Andrew >> >> On Fri, Sep 30, 2016 at 8:12 AM, Patrick Williams <patrick@stwcx.xyz> wrote: >> > >> > On Fri, Sep 30, 2016 at 11:16:08AM +0930, Joel Stanley wrote: >> > > >> > > Okay. Do we plan to test the individual projects as part of an >> > > openbmc/openbmc build in the future? >> > > >> > Yes. https://github.com/openbmc/openbmc/issues/382 >> > >> > -- >> > Patrick Williams >> _______________________________________________ >> openbmc mailing list >> openbmc@lists.ozlabs.org >> https://lists.ozlabs.org/listinfo/openbmc ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: openpower.xyz updates 2016-10-10 23:44 ` Andrew Jeffery 2016-10-10 23:49 ` Joel Stanley @ 2016-10-11 1:36 ` Chris Smart 2016-10-11 1:43 ` Andrew Geissler 1 sibling, 1 reply; 16+ messages in thread From: Chris Smart @ 2016-10-11 1:36 UTC (permalink / raw) To: Andrew Jeffery Cc: Andrew Geissler, Patrick Williams, OpenBMC Maillist, Joel Stanley [-- Attachment #1: Type: text/plain, Size: 2031 bytes --] On Tue, Oct 11, 2016 at 10:14:18AM +1030, Andrew Jeffery wrote: >On Mon, 2016-10-10 at 11:09 -0500, Andrew Geissler wrote: >> I noticed our gerrit trigger job to verify userid's was broken again >> this morning. Looks like the trouble plugin (Git Client Plugin) was >> upgraded again to the latest version. Anyone out there know why/who >> did it? I'm going to put us back to 1.19.7 per my earlier comments. > >Joel and Chris Smart were looking at this yesterday. A job broke: > >https://openpower.xyz/job/openpower-op-build/184/console > >According to Chris it broke because of: > >https://bugs.eclipse.org/bugs/show_bug.cgi?id=428039 > > 15:58 < csmart> joel, basically release() was replaced with close() > and so any plugin that uses git needs to be updated > >Looks like we're stuck between a rock and a hard place. > Yep, I updated the git client plugin because openpower build couldn't perform any git operations. I didn't expect that to affect any comms with other Jenkins instances though, that is strange. Why does the gerrit userid plugin rely on on git client? Do you have a log or backtrace? Might help to try and track it down. -c >Cheers, > >Andrew > >> >> There's no fix per-se for this, jenkins is always a bit hit or miss >> with it's latest LTS version working with the latest plugins. Best >> thing we can do is when we schedule our next jenkins LTS upgrade, to >> try all the latest plugins again. >> >> Andrew >> >> On Fri, Sep 30, 2016 at 8:12 AM, Patrick Williams <patrick@stwcx.xyz> wrote: >> > >> > On Fri, Sep 30, 2016 at 11:16:08AM +0930, Joel Stanley wrote: >> > > >> > > Okay. Do we plan to test the individual projects as part of an >> > > openbmc/openbmc build in the future? >> > > >> > Yes. https://github.com/openbmc/openbmc/issues/382 >> > >> > -- >> > Patrick Williams >> _______________________________________________ >> openbmc mailing list >> openbmc@lists.ozlabs.org >> https://lists.ozlabs.org/listinfo/openbmc [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 819 bytes --] ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: openpower.xyz updates 2016-10-11 1:36 ` Chris Smart @ 2016-10-11 1:43 ` Andrew Geissler 2016-10-11 2:00 ` Chris Smart 0 siblings, 1 reply; 16+ messages in thread From: Andrew Geissler @ 2016-10-11 1:43 UTC (permalink / raw) To: Chris Smart Cc: Andrew Jeffery, Patrick Williams, OpenBMC Maillist, Joel Stanley On Mon, Oct 10, 2016 at 8:36 PM, Chris Smart <chris@distroguy.com> wrote: > On Tue, Oct 11, 2016 at 10:14:18AM +1030, Andrew Jeffery wrote: >> >> On Mon, 2016-10-10 at 11:09 -0500, Andrew Geissler wrote: >>> >>> I noticed our gerrit trigger job to verify userid's was broken again >>> this morning. Looks like the trouble plugin (Git Client Plugin) was >>> upgraded again to the latest version. Anyone out there know why/who >>> did it? I'm going to put us back to 1.19.7 per my earlier comments. >> >> >> Joel and Chris Smart were looking at this yesterday. A job broke: >> >> https://openpower.xyz/job/openpower-op-build/184/console >> >> According to Chris it broke because of: >> >> https://bugs.eclipse.org/bugs/show_bug.cgi?id=428039 >> >> 15:58 < csmart> joel, basically release() was replaced with close() >> and so any plugin that uses git needs to be updated >> >> Looks like we're stuck between a rock and a hard place. >> > > Yep, I updated the git client plugin because openpower build couldn't > perform any git operations. > > I didn't expect that to affect any comms with other Jenkins instances > though, that is strange. Why does the gerrit userid plugin rely on on > git client? Do you have a log or backtrace? Might help to try and track > it down. Here's the fail, it's a weird one: https://openpower.xyz/job/openbmc-userid-verify/795/console 07:05:18 > git rev-parse FETCH_HEAD^{commit} # timeout=10 07:05:18 FATAL: org.eclipse.jgit.revwalk.RevWalk.release()V 07:05:18 java.lang.NoSuchMethodError: org.eclipse.jgit.revwalk.RevWalk.release()V 07:05:18 at com.sonyericsson.hudson.plugins.gerrit.trigger.hudsontrigger.GerritTriggerBuildChooser$1.invoke(GerritTriggerBuildChooser.java:185) 07:05:18 at com.sonyericsson.hudson.plugins.gerrit.trigger.hudsontrigger.GerritTriggerBuildChooser$1.invoke(GerritTriggerBuildChooser.java:166) 07:05:18 at org.jenkinsci.plugins.gitclient.AbstractGitAPIImpl.withRepository(AbstractGitAPIImpl.java:27) 07:05:18 at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.withRepository(CliGitAPIImpl.java:64) 07:05:18 at com.sonyericsson.hudson.plugins.gerrit.trigger.hudsontrigger.GerritTriggerBuildChooser.getFirstParent(GerritTriggerBuildChooser.java:166) ... No bug report on my part since I didn't think it was that big an issue....I have been proven wrong. I thing we have an imminent call, lets touch base there and I'll get one open if needed. > > -c > > >> Cheers, >> >> Andrew >> >>> >>> There's no fix per-se for this, jenkins is always a bit hit or miss >>> with it's latest LTS version working with the latest plugins. Best >>> thing we can do is when we schedule our next jenkins LTS upgrade, to >>> try all the latest plugins again. >>> >>> Andrew >>> >>> On Fri, Sep 30, 2016 at 8:12 AM, Patrick Williams <patrick@stwcx.xyz> >>> wrote: >>> > >>> > On Fri, Sep 30, 2016 at 11:16:08AM +0930, Joel Stanley wrote: >>> > > >>> > > Okay. Do we plan to test the individual projects as part of an >>> > > openbmc/openbmc build in the future? >>> > > >>> > Yes. https://github.com/openbmc/openbmc/issues/382 >>> > >>> > -- >>> > Patrick Williams >>> _______________________________________________ >>> openbmc mailing list >>> openbmc@lists.ozlabs.org >>> https://lists.ozlabs.org/listinfo/openbmc > > > ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: openpower.xyz updates 2016-10-11 1:43 ` Andrew Geissler @ 2016-10-11 2:00 ` Chris Smart 2016-10-11 2:05 ` Chris Smart 2016-10-11 2:06 ` Andrew Geissler 0 siblings, 2 replies; 16+ messages in thread From: Chris Smart @ 2016-10-11 2:00 UTC (permalink / raw) To: Andrew Geissler Cc: Andrew Jeffery, Patrick Williams, OpenBMC Maillist, Joel Stanley On Mon, Oct 10, 2016 at 08:43:43PM -0500, Andrew Geissler wrote: >On Mon, Oct 10, 2016 at 8:36 PM, Chris Smart <chris@distroguy.com> wrote: >> On Tue, Oct 11, 2016 at 10:14:18AM +1030, Andrew Jeffery wrote: >>> >>> On Mon, 2016-10-10 at 11:09 -0500, Andrew Geissler wrote: >>>> >>>> I noticed our gerrit trigger job to verify userid's was broken again >>>> this morning. Looks like the trouble plugin (Git Client Plugin) was >>>> upgraded again to the latest version. Anyone out there know why/who >>>> did it? I'm going to put us back to 1.19.7 per my earlier comments. >>> >>> >>> Joel and Chris Smart were looking at this yesterday. A job broke: >>> >>> https://openpower.xyz/job/openpower-op-build/184/console >>> >>> According to Chris it broke because of: >>> >>> https://bugs.eclipse.org/bugs/show_bug.cgi?id=428039 >>> >>> 15:58 < csmart> joel, basically release() was replaced with close() >>> and so any plugin that uses git needs to be updated >>> >>> Looks like we're stuck between a rock and a hard place. >>> >> >> Yep, I updated the git client plugin because openpower build couldn't >> perform any git operations. >> >> I didn't expect that to affect any comms with other Jenkins instances >> though, that is strange. Why does the gerrit userid plugin rely on on >> git client? Do you have a log or backtrace? Might help to try and track >> it down. > >Here's the fail, it's a weird one: >https://openpower.xyz/job/openbmc-userid-verify/795/console > >07:05:18 > git rev-parse FETCH_HEAD^{commit} # timeout=10 >07:05:18 FATAL: org.eclipse.jgit.revwalk.RevWalk.release()V >07:05:18 java.lang.NoSuchMethodError: >org.eclipse.jgit.revwalk.RevWalk.release()V >07:05:18 at com.sonyericsson.hudson.plugins.gerrit.trigger.hudsontrigger.GerritTriggerBuildChooser$1.invoke(GerritTriggerBuildChooser.java:185) >07:05:18 at com.sonyericsson.hudson.plugins.gerrit.trigger.hudsontrigger.GerritTriggerBuildChooser$1.invoke(GerritTriggerBuildChooser.java:166) >07:05:18 at org.jenkinsci.plugins.gitclient.AbstractGitAPIImpl.withRepository(AbstractGitAPIImpl.java:27) >07:05:18 at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.withRepository(CliGitAPIImpl.java:64) >07:05:18 at com.sonyericsson.hudson.plugins.gerrit.trigger.hudsontrigger.GerritTriggerBuildChooser.getFirstParent(GerritTriggerBuildChooser.java:166) >... > >No bug report on my part since I didn't think it was that big an >issue....I have been proven wrong. I thing we have an imminent call, >lets touch base there and I'll get one open if needed. > Yeah, so I think that's the same issue; release() was replaced with close() in jgit and so any plugin that uses that needs to be updated. This is what was breaking the openpower build and was fixed by updating the git client plugin. The log snippet you provided suggests to me that the gerrit trigger plugin needs to be updated, you can see it's trying to call that release() here: 07:05:18 FATAL: org.eclipse.jgit.revwalk.RevWalk.release()V 07:05:18 java.lang.NoSuchMethodError: org.eclipse.jgit.revwalk.RevWalk.release()V 07:05:18 at com.sonyericsson.hudson.plugins.gerrit.trigger.hudsontrigger.GerritTriggerBuildChooser$1.invoke(GerritTriggerBuildChooser.java:185) -c >> >> -c >> >> >>> Cheers, >>> >>> Andrew >>> >>>> >>>> There's no fix per-se for this, jenkins is always a bit hit or miss >>>> with it's latest LTS version working with the latest plugins. Best >>>> thing we can do is when we schedule our next jenkins LTS upgrade, to >>>> try all the latest plugins again. >>>> >>>> Andrew >>>> >>>> On Fri, Sep 30, 2016 at 8:12 AM, Patrick Williams <patrick@stwcx.xyz> >>>> wrote: >>>> > >>>> > On Fri, Sep 30, 2016 at 11:16:08AM +0930, Joel Stanley wrote: >>>> > > >>>> > > Okay. Do we plan to test the individual projects as part of an >>>> > > openbmc/openbmc build in the future? >>>> > > >>>> > Yes. https://github.com/openbmc/openbmc/issues/382 >>>> > >>>> > -- >>>> > Patrick Williams >>>> _______________________________________________ >>>> openbmc mailing list >>>> openbmc@lists.ozlabs.org >>>> https://lists.ozlabs.org/listinfo/openbmc >> >> >> ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: openpower.xyz updates 2016-10-11 2:00 ` Chris Smart @ 2016-10-11 2:05 ` Chris Smart 2016-10-11 2:06 ` Andrew Geissler 1 sibling, 0 replies; 16+ messages in thread From: Chris Smart @ 2016-10-11 2:05 UTC (permalink / raw) To: Andrew Geissler Cc: Andrew Jeffery, Patrick Williams, OpenBMC Maillist, Joel Stanley On Tue, Oct 11, 2016 at 01:00:29PM +1100, Chris Smart wrote: >On Mon, Oct 10, 2016 at 08:43:43PM -0500, Andrew Geissler wrote: >>On Mon, Oct 10, 2016 at 8:36 PM, Chris Smart <chris@distroguy.com> wrote: >>>On Tue, Oct 11, 2016 at 10:14:18AM +1030, Andrew Jeffery wrote: >>>> >>>>On Mon, 2016-10-10 at 11:09 -0500, Andrew Geissler wrote: >>>>> >>>>>I noticed our gerrit trigger job to verify userid's was broken again >>>>>this morning. Looks like the trouble plugin (Git Client Plugin) was >>>>>upgraded again to the latest version. Anyone out there know why/who >>>>>did it? I'm going to put us back to 1.19.7 per my earlier comments. >>>> >>>> >>>>Joel and Chris Smart were looking at this yesterday. A job broke: >>>> >>>>https://openpower.xyz/job/openpower-op-build/184/console >>>> >>>>According to Chris it broke because of: >>>> >>>>https://bugs.eclipse.org/bugs/show_bug.cgi?id=428039 >>>> >>>> 15:58 < csmart> joel, basically release() was replaced with close() >>>> and so any plugin that uses git needs to be updated >>>> >>>>Looks like we're stuck between a rock and a hard place. >>>> >>> >>>Yep, I updated the git client plugin because openpower build couldn't >>>perform any git operations. >>> >>>I didn't expect that to affect any comms with other Jenkins instances >>>though, that is strange. Why does the gerrit userid plugin rely on on >>>git client? Do you have a log or backtrace? Might help to try and track >>>it down. >> >>Here's the fail, it's a weird one: >>https://openpower.xyz/job/openbmc-userid-verify/795/console >> >>07:05:18 > git rev-parse FETCH_HEAD^{commit} # timeout=10 >>07:05:18 FATAL: org.eclipse.jgit.revwalk.RevWalk.release()V >>07:05:18 java.lang.NoSuchMethodError: >>org.eclipse.jgit.revwalk.RevWalk.release()V >>07:05:18 at com.sonyericsson.hudson.plugins.gerrit.trigger.hudsontrigger.GerritTriggerBuildChooser$1.invoke(GerritTriggerBuildChooser.java:185) >>07:05:18 at com.sonyericsson.hudson.plugins.gerrit.trigger.hudsontrigger.GerritTriggerBuildChooser$1.invoke(GerritTriggerBuildChooser.java:166) >>07:05:18 at org.jenkinsci.plugins.gitclient.AbstractGitAPIImpl.withRepository(AbstractGitAPIImpl.java:27) >>07:05:18 at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.withRepository(CliGitAPIImpl.java:64) >>07:05:18 at com.sonyericsson.hudson.plugins.gerrit.trigger.hudsontrigger.GerritTriggerBuildChooser.getFirstParent(GerritTriggerBuildChooser.java:166) >>... >> >>No bug report on my part since I didn't think it was that big an >>issue....I have been proven wrong. I thing we have an imminent call, >>lets touch base there and I'll get one open if needed. >> > >Yeah, so I think that's the same issue; release() was replaced with >close() in jgit and so any plugin that uses that needs to be updated. > >This is what was breaking the openpower build and was fixed by updating >the git client plugin. > >The log snippet you provided suggests to me that the gerrit trigger >plugin needs to be updated, you can see it's trying to call that >release() here: > >07:05:18 FATAL: org.eclipse.jgit.revwalk.RevWalk.release()V >07:05:18 java.lang.NoSuchMethodError: org.eclipse.jgit.revwalk.RevWalk.release()V >07:05:18 at com.sonyericsson.hudson.plugins.gerrit.trigger.hudsontrigger.GerritTriggerBuildChooser$1.invoke(GerritTriggerBuildChooser.java:185) > And to that effect: Release 2.22.0 of GerritTrigger fixes this: https://wiki.jenkins-ci.org/display/JENKINS/Gerrit+Trigger#GerritTrigger-Version2.22.0%28releasedAug172016%29 Version 2.22.0 (released Aug 17 2016) * Prepare for git client plugin 2.0.0 coexistence with 1.x (pull #296) pull #296: https://github.com/jenkinsci/gerrit-trigger-plugin/pull/296 So I would expect that updating gitclient and also gerrit-trigger would solve these issues. I don't know about the userid thing though - if you have a log for that, feel free to send it through and I'll take a look. -c >-c > >>> >>>-c >>> >>> >>>>Cheers, >>>> >>>>Andrew >>>> >>>>> >>>>>There's no fix per-se for this, jenkins is always a bit hit or miss >>>>>with it's latest LTS version working with the latest plugins. Best >>>>>thing we can do is when we schedule our next jenkins LTS upgrade, to >>>>>try all the latest plugins again. >>>>> >>>>>Andrew >>>>> >>>>>On Fri, Sep 30, 2016 at 8:12 AM, Patrick Williams <patrick@stwcx.xyz> >>>>>wrote: >>>>>> >>>>>> On Fri, Sep 30, 2016 at 11:16:08AM +0930, Joel Stanley wrote: >>>>>> > >>>>>> > Okay. Do we plan to test the individual projects as part of an >>>>>> > openbmc/openbmc build in the future? >>>>>> > >>>>>> Yes. https://github.com/openbmc/openbmc/issues/382 >>>>>> >>>>>> -- >>>>>> Patrick Williams >>>>>_______________________________________________ >>>>>openbmc mailing list >>>>>openbmc@lists.ozlabs.org >>>>>https://lists.ozlabs.org/listinfo/openbmc >>> >>> >>> ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: openpower.xyz updates 2016-10-11 2:00 ` Chris Smart 2016-10-11 2:05 ` Chris Smart @ 2016-10-11 2:06 ` Andrew Geissler 2016-10-11 2:12 ` Joel Stanley 1 sibling, 1 reply; 16+ messages in thread From: Andrew Geissler @ 2016-10-11 2:06 UTC (permalink / raw) To: Chris Smart Cc: Andrew Jeffery, Patrick Williams, OpenBMC Maillist, Joel Stanley On Mon, Oct 10, 2016 at 9:00 PM, Chris Smart <chris@distroguy.com> wrote: > On Mon, Oct 10, 2016 at 08:43:43PM -0500, Andrew Geissler wrote: >> >> On Mon, Oct 10, 2016 at 8:36 PM, Chris Smart <chris@distroguy.com> wrote: >>> >>> On Tue, Oct 11, 2016 at 10:14:18AM +1030, Andrew Jeffery wrote: >>>> >>>> >>>> On Mon, 2016-10-10 at 11:09 -0500, Andrew Geissler wrote: >>>>> >>>>> >>>>> I noticed our gerrit trigger job to verify userid's was broken again >>>>> this morning. Looks like the trouble plugin (Git Client Plugin) was >>>>> upgraded again to the latest version. Anyone out there know why/who >>>>> did it? I'm going to put us back to 1.19.7 per my earlier comments. >>>> >>>> >>>> >>>> Joel and Chris Smart were looking at this yesterday. A job broke: >>>> >>>> https://openpower.xyz/job/openpower-op-build/184/console >>>> >>>> According to Chris it broke because of: >>>> >>>> https://bugs.eclipse.org/bugs/show_bug.cgi?id=428039 >>>> >>>> 15:58 < csmart> joel, basically release() was replaced with close() >>>> and so any plugin that uses git needs to be updated >>>> >>>> Looks like we're stuck between a rock and a hard place. >>>> >>> >>> Yep, I updated the git client plugin because openpower build couldn't >>> perform any git operations. >>> >>> I didn't expect that to affect any comms with other Jenkins instances >>> though, that is strange. Why does the gerrit userid plugin rely on on >>> git client? Do you have a log or backtrace? Might help to try and track >>> it down. >> >> >> Here's the fail, it's a weird one: >> https://openpower.xyz/job/openbmc-userid-verify/795/console >> >> 07:05:18 > git rev-parse FETCH_HEAD^{commit} # timeout=10 >> 07:05:18 FATAL: org.eclipse.jgit.revwalk.RevWalk.release()V >> 07:05:18 java.lang.NoSuchMethodError: >> org.eclipse.jgit.revwalk.RevWalk.release()V >> 07:05:18 at >> com.sonyericsson.hudson.plugins.gerrit.trigger.hudsontrigger.GerritTriggerBuildChooser$1.invoke(GerritTriggerBuildChooser.java:185) >> 07:05:18 at >> com.sonyericsson.hudson.plugins.gerrit.trigger.hudsontrigger.GerritTriggerBuildChooser$1.invoke(GerritTriggerBuildChooser.java:166) >> 07:05:18 at >> org.jenkinsci.plugins.gitclient.AbstractGitAPIImpl.withRepository(AbstractGitAPIImpl.java:27) >> 07:05:18 at >> org.jenkinsci.plugins.gitclient.CliGitAPIImpl.withRepository(CliGitAPIImpl.java:64) >> 07:05:18 at >> com.sonyericsson.hudson.plugins.gerrit.trigger.hudsontrigger.GerritTriggerBuildChooser.getFirstParent(GerritTriggerBuildChooser.java:166) >> ... >> >> No bug report on my part since I didn't think it was that big an >> issue....I have been proven wrong. I thing we have an imminent call, >> lets touch base there and I'll get one open if needed. >> > > Yeah, so I think that's the same issue; release() was replaced with > close() in jgit and so any plugin that uses that needs to be updated. > > This is what was breaking the openpower build and was fixed by updating > the git client plugin. > > The log snippet you provided suggests to me that the gerrit trigger > plugin needs to be updated, you can see it's trying to call that > release() here: > > 07:05:18 FATAL: org.eclipse.jgit.revwalk.RevWalk.release()V > 07:05:18 java.lang.NoSuchMethodError: > org.eclipse.jgit.revwalk.RevWalk.release()V > 07:05:18 at > com.sonyericsson.hudson.plugins.gerrit.trigger.hudsontrigger.GerritTriggerBuildChooser$1.invoke(GerritTriggerBuildChooser.java:185) > Yeah, sigh... for some reason, when we had the latest gerrit plugin, I couldn't trigger off of "Comment Added" anymore which is what we use for the CI build trigger. I didn't find any obvious bug reports for it when I looked. Andrew > -c > > >>> >>> -c >>> >>> >>>> Cheers, >>>> >>>> Andrew >>>> >>>>> >>>>> There's no fix per-se for this, jenkins is always a bit hit or miss >>>>> with it's latest LTS version working with the latest plugins. Best >>>>> thing we can do is when we schedule our next jenkins LTS upgrade, to >>>>> try all the latest plugins again. >>>>> >>>>> Andrew >>>>> >>>>> On Fri, Sep 30, 2016 at 8:12 AM, Patrick Williams <patrick@stwcx.xyz> >>>>> wrote: >>>>> > >>>>> > On Fri, Sep 30, 2016 at 11:16:08AM +0930, Joel Stanley wrote: >>>>> > > >>>>> > > Okay. Do we plan to test the individual projects as part of an >>>>> > > openbmc/openbmc build in the future? >>>>> > > >>>>> > Yes. https://github.com/openbmc/openbmc/issues/382 >>>>> > >>>>> > -- >>>>> > Patrick Williams >>>>> _______________________________________________ >>>>> openbmc mailing list >>>>> openbmc@lists.ozlabs.org >>>>> https://lists.ozlabs.org/listinfo/openbmc >>> >>> >>> >>> > ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: openpower.xyz updates 2016-10-11 2:06 ` Andrew Geissler @ 2016-10-11 2:12 ` Joel Stanley 2016-10-11 6:23 ` Joel Stanley 0 siblings, 1 reply; 16+ messages in thread From: Joel Stanley @ 2016-10-11 2:12 UTC (permalink / raw) To: Andrew Geissler Cc: Chris Smart, Andrew Jeffery, Patrick Williams, OpenBMC Maillist On Tue, Oct 11, 2016 at 1:06 PM, Andrew Geissler <geissonator@gmail.com> wrote: >> The log snippet you provided suggests to me that the gerrit trigger >> plugin needs to be updated, you can see it's trying to call that >> release() here: >> >> 07:05:18 FATAL: org.eclipse.jgit.revwalk.RevWalk.release()V >> 07:05:18 java.lang.NoSuchMethodError: >> org.eclipse.jgit.revwalk.RevWalk.release()V >> 07:05:18 at >> com.sonyericsson.hudson.plugins.gerrit.trigger.hudsontrigger.GerritTriggerBuildChooser$1.invoke(GerritTriggerBuildChooser.java:185) >> > > Yeah, sigh... for some reason, when we had the latest gerrit plugin, I > couldn't trigger off of "Comment Added" anymore which is what we use > for the CI build trigger. I didn't find any obvious bug reports for > it when I looked. Do you have the backtrace and log for that failure? Cheers, Joel ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: openpower.xyz updates 2016-10-11 2:12 ` Joel Stanley @ 2016-10-11 6:23 ` Joel Stanley 2016-10-12 5:36 ` Joel Stanley 0 siblings, 1 reply; 16+ messages in thread From: Joel Stanley @ 2016-10-11 6:23 UTC (permalink / raw) To: Andrew Geissler Cc: Chris Smart, Andrew Jeffery, Patrick Williams, OpenBMC Maillist On Tue, Oct 11, 2016 at 1:12 PM, Joel Stanley <joel@jms.id.au> wrote: > On Tue, Oct 11, 2016 at 1:06 PM, Andrew Geissler <geissonator@gmail.com> wrote: >>> The log snippet you provided suggests to me that the gerrit trigger >>> plugin needs to be updated, you can see it's trying to call that >>> release() here: >>> >>> 07:05:18 FATAL: org.eclipse.jgit.revwalk.RevWalk.release()V >>> 07:05:18 java.lang.NoSuchMethodError: >>> org.eclipse.jgit.revwalk.RevWalk.release()V >>> 07:05:18 at >>> com.sonyericsson.hudson.plugins.gerrit.trigger.hudsontrigger.GerritTriggerBuildChooser$1.invoke(GerritTriggerBuildChooser.java:185) >>> >> >> Yeah, sigh... for some reason, when we had the latest gerrit plugin, I >> couldn't trigger off of "Comment Added" anymore which is what we use >> for the CI build trigger. I didn't find any obvious bug reports for >> it when I looked. > > Do you have the backtrace and log for that failure? Patrick, can you look at the logs from the gerrit side? Is the connection succeeding? Andrew G, I took a look at the Gerrit plugin between 2.21.1 and 2.22. https://github.com/jenkinsci/gerrit-trigger-plugin/compare/gerrit-trigger-2.21.1...gerrit-trigger-2.22.0 https://issues.jenkins-ci.org/browse/JENKINS-35379 Perhaps we could try bisecting the changes made to the plugin to work out what went wrong? Cheers, Joel ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: openpower.xyz updates 2016-10-11 6:23 ` Joel Stanley @ 2016-10-12 5:36 ` Joel Stanley 0 siblings, 0 replies; 16+ messages in thread From: Joel Stanley @ 2016-10-12 5:36 UTC (permalink / raw) To: Andrew Geissler Cc: Chris Smart, Andrew Jeffery, Patrick Williams, OpenBMC Maillist On Tue, Oct 11, 2016 at 5:23 PM, Joel Stanley <joel@jms.id.au> wrote: > On Tue, Oct 11, 2016 at 1:12 PM, Joel Stanley <joel@jms.id.au> wrote: > Andrew G, I took a look at the Gerrit plugin between 2.21.1 and 2.22. > > https://github.com/jenkinsci/gerrit-trigger-plugin/compare/gerrit-trigger-2.21.1...gerrit-trigger-2.22.0 > > https://issues.jenkins-ci.org/browse/JENKINS-35379 > > Perhaps we could try bisecting the changes made to the plugin to work > out what went wrong? Overnight we upgraded to the latest LTS release of Jenkins to see if that fixed our issue. It did not. I then tried to downgrade the plugins, but this caused our github-oauth plugin to downgrade (I think the new Jenkins release tries to downgrade dependencies when downgrading a plugin). This broke login, which relies on oauth. I tried to re-upgrade the plugin, but it was still broken - an exception was thrown as the plugin was unable to parse the oauth config file. For reference, the signatures looked like this: Caused by: java.lang.ClassCastException: org.acegisecurity.providers.anonymous.AnonymousAuthenticationToken cannot be cast to org.jenkinsci.plugins.GithubAuthenticationToken com.google.common.util.concurrent.UncheckedExecutionException: java.lang.ClassCastException: org.acegisecurity.providers.anonymous.AnonymousAuthenticationToken cannot be cast to org.jenkinsci.plugins.GithubAuthenticationToken We tried removing the config file, modifying it, restoring from backup, to no avail. To our luck, the github-oauth plugin was updated over night. The new release Just Worked so we ran with that. Back to the core issue of the Gerrit - Jenkins failure to kick of build jobs. I found out more about the design of the Gerrit - Jenkins system today. We have a Gerrit Trigger plugin that listens to the event stream from Gerrit over ssh. When it sees a new patch it runs a job that checks the submitter against a list of whitelisted users (those in the OpenBMC team on Github). If the user is approved, it updates the commit with the ok-to-test label. Another job uses the Trigger plugin to look for the ok-to-test label. When it sees this label in the stream, it kicks off a build job. The reason Andrew separates out the two is so an admin can manually tick the ok-to-test box allowing drive by contributions to be tested. The bug is due to the newer version of the plugin not seeing these events. We're not sure why that happens, but put a workaround in place that causes the jobs to still trigger while we sort it out. Cheers, Joel ^ permalink raw reply [flat|nested] 16+ messages in thread
end of thread, other threads:[~2016-10-12 5:36 UTC | newest]
Thread overview: 16+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2016-09-26 11:07 openpower.xyz updates Joel Stanley
[not found] ` <CALLMt=rbR1usfKPt9G8kdUa2RaqRaaVEG3rV62wSBjaM59qi6Q@mail.gmail.com>
2016-09-30 1:07 ` Joel Stanley
2016-09-30 1:38 ` Patrick Williams
2016-09-30 1:46 ` Joel Stanley
2016-09-30 13:12 ` Patrick Williams
2016-10-10 16:09 ` Andrew Geissler
2016-10-10 23:44 ` Andrew Jeffery
2016-10-10 23:49 ` Joel Stanley
2016-10-11 1:36 ` Chris Smart
2016-10-11 1:43 ` Andrew Geissler
2016-10-11 2:00 ` Chris Smart
2016-10-11 2:05 ` Chris Smart
2016-10-11 2:06 ` Andrew Geissler
2016-10-11 2:12 ` Joel Stanley
2016-10-11 6:23 ` Joel Stanley
2016-10-12 5:36 ` Joel Stanley
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.