openpower.xyz updates

openpower.xyz updates

[Joel Stanley]

Today I performed some upgrades on openpower.xyz:

 - Upgraded from Ubuntu 14.04 LTS to 16.04 LTS
 - Moved from the one-year free TLS certificate to LetsEncrypt
 - Updated Jenkins from 10 month old version to the latest (2.7.4)

Things appear to be working okay. Please let me know if you notice any
breakages.

Cheers,

Joel

Re: openpower.xyz updates

[Joel Stanley]

On Wed, Sep 28, 2016 at 10:08 PM, Andrew Geissler <geissonator@gmail.com> wrote:
> Hey Joel, late yesterday afternoon Patrick reported that the gerrit CI
> jobs were not triggering.  I took a look and found that our job that
> does ID validation (which triggers on patch set creation) was still
> working fine but the follow up job that would trigger on the
> "Ok-To-Test" comment was no longer triggering.

I noticed that many of the gerrit reviews I have pending do not have
the OOT or Verified checked. Is the system still having issues?

> I've seen this a few times and it's usually always incompatibilities
> between the gerrit and git plugins and/or the jenkins version.  Since
> it was blocking all dev work and we're getting near the sprint end I
> jumped on and just had to downlevel the gerrit and git plugins by one
> level to get things working again.
>
> Specifically, I had to downgrade our gerrit plugin to 2.21.1 and our
> git client plugin to 1.19.7

Thanks for letting me know.

Do you have a plan for fixing the issue so we can run up to date
versions of the gerrit plugins?

Cheers,

Joel

>
> Andrew
>
> On Mon, Sep 26, 2016 at 6:07 AM, Joel Stanley <joel@jms.id.au> wrote:
>> Today I performed some upgrades on openpower.xyz:
>>
>>  - Upgraded from Ubuntu 14.04 LTS to 16.04 LTS
>>  - Moved from the one-year free TLS certificate to LetsEncrypt
>>  - Updated Jenkins from 10 month old version to the latest (2.7.4)
>>
>> Things appear to be working okay. Please let me know if you notice any
>> breakages.
>>
>> Cheers,
>>
>> Joel

Re: openpower.xyz updates

[Patrick Williams]

[-- Attachment #1: Type: text/plain, Size: 871 bytes --]

On Fri, Sep 30, 2016 at 10:37:42AM +0930, Joel Stanley wrote:
> On Wed, Sep 28, 2016 at 10:08 PM, Andrew Geissler <geissonator@gmail.com> wrote:
> > Hey Joel, late yesterday afternoon Patrick reported that the gerrit CI
> > jobs were not triggering.  I took a look and found that our job that
> > does ID validation (which triggers on patch set creation) was still
> > working fine but the follow up job that would trigger on the
> > "Ok-To-Test" comment was no longer triggering.
> 
> I noticed that many of the gerrit reviews I have pending do not have
> the OOT or Verified checked. Is the system still having issues?

Currently, the only testing is done on the openbmc/openbmc repository.
According to this query they are all set with both OTT and V.

https://gerrit.openbmc-project.xyz/#/q/project:openbmc/openbmc+status:open

-- 
Patrick Williams

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: openpower.xyz updates

[Joel Stanley]

On Fri, Sep 30, 2016 at 11:08 AM, Patrick Williams <patrick@stwcx.xyz> wrote:
> On Fri, Sep 30, 2016 at 10:37:42AM +0930, Joel Stanley wrote:
>> On Wed, Sep 28, 2016 at 10:08 PM, Andrew Geissler <geissonator@gmail.com> wrote:
>> > Hey Joel, late yesterday afternoon Patrick reported that the gerrit CI
>> > jobs were not triggering.  I took a look and found that our job that
>> > does ID validation (which triggers on patch set creation) was still
>> > working fine but the follow up job that would trigger on the
>> > "Ok-To-Test" comment was no longer triggering.
>>
>> I noticed that many of the gerrit reviews I have pending do not have
>> the OOT or Verified checked. Is the system still having issues?
>
> Currently, the only testing is done on the openbmc/openbmc repository.
> According to this query they are all set with both OTT and V.
>
> https://gerrit.openbmc-project.xyz/#/q/project:openbmc/openbmc+status:open

Okay. Do we plan to test the individual projects as part of an
openbmc/openbmc build in the future?

Cheers,

Joel

Re: openpower.xyz updates

[Patrick Williams]

[-- Attachment #1: Type: text/plain, Size: 253 bytes --]

On Fri, Sep 30, 2016 at 11:16:08AM +0930, Joel Stanley wrote:
> Okay. Do we plan to test the individual projects as part of an
> openbmc/openbmc build in the future?
> 

Yes.  https://github.com/openbmc/openbmc/issues/382

-- 
Patrick Williams

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: openpower.xyz updates

[Andrew Geissler]

I noticed our gerrit trigger job to verify userid's was broken again
this morning.  Looks like the trouble plugin (Git Client Plugin) was
upgraded again to the latest version.  Anyone out there know why/who
did it?  I'm going to put us back to 1.19.7 per my earlier comments.

There's no fix per-se for this, jenkins is always a bit hit or miss
with it's latest LTS version working with the latest plugins.  Best
thing we can do is when we schedule our next jenkins LTS upgrade, to
try all the latest plugins again.

Andrew

On Fri, Sep 30, 2016 at 8:12 AM, Patrick Williams <patrick@stwcx.xyz> wrote:
> On Fri, Sep 30, 2016 at 11:16:08AM +0930, Joel Stanley wrote:
>> Okay. Do we plan to test the individual projects as part of an
>> openbmc/openbmc build in the future?
>>
>
> Yes.  https://github.com/openbmc/openbmc/issues/382
>
> --
> Patrick Williams

Re: openpower.xyz updates

[Andrew Jeffery]

[-- Attachment #1: Type: text/plain, Size: 1594 bytes --]

On Mon, 2016-10-10 at 11:09 -0500, Andrew Geissler wrote:
> I noticed our gerrit trigger job to verify userid's was broken again
> this morning.  Looks like the trouble plugin (Git Client Plugin) was
> upgraded again to the latest version.  Anyone out there know why/who
> did it?  I'm going to put us back to 1.19.7 per my earlier comments.

Joel and Chris Smart were looking at this yesterday. A job broke:

https://openpower.xyz/job/openpower-op-build/184/console

According to Chris it broke because of:

https://bugs.eclipse.org/bugs/show_bug.cgi?id=428039

    15:58 < csmart> joel, basically release() was replaced with close()
    and so any plugin that uses git needs to be updated

Looks like we're stuck between a rock and a hard place.

Cheers,

Andrew

> 
> There's no fix per-se for this, jenkins is always a bit hit or miss
> with it's latest LTS version working with the latest plugins.  Best
> thing we can do is when we schedule our next jenkins LTS upgrade, to
> try all the latest plugins again.
> 
> Andrew
> 
> On Fri, Sep 30, 2016 at 8:12 AM, Patrick Williams <patrick@stwcx.xyz> wrote:
> > 
> > On Fri, Sep 30, 2016 at 11:16:08AM +0930, Joel Stanley wrote:
> > > 
> > > Okay. Do we plan to test the individual projects as part of an
> > > openbmc/openbmc build in the future?
> > > 
> > Yes.  https://github.com/openbmc/openbmc/issues/382
> > 
> > --
> > Patrick Williams
> _______________________________________________
> openbmc mailing list
> openbmc@lists.ozlabs.org
> https://lists.ozlabs.org/listinfo/openbmc

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: openpower.xyz updates

[Joel Stanley]

On Tue, Oct 11, 2016 at 10:44 AM, Andrew Jeffery <andrew@aj.id.au> wrote:
> On Mon, 2016-10-10 at 11:09 -0500, Andrew Geissler wrote:
>> I noticed our gerrit trigger job to verify userid's was broken again
>> this morning.  Looks like the trouble plugin (Git Client Plugin) was
>> upgraded again to the latest version.  Anyone out there know why/who
>> did it?  I'm going to put us back to 1.19.7 per my earlier comments.
>
> Joel and Chris Smart were looking at this yesterday. A job broke:
>
> https://openpower.xyz/job/openpower-op-build/184/console
>
> According to Chris it broke because of:
>
> https://bugs.eclipse.org/bugs/show_bug.cgi?id=428039
>
>     15:58 < csmart> joel, basically release() was replaced with close()
>     and so any plugin that uses git needs to be updated

Yep.

Andrew G, do you have a bug report for the issue you're having?

We need to find a solution that lets us stay up to date with fixes from Jenkins.

Cheers,

Joel

>
> Looks like we're stuck between a rock and a hard place.
>
> Cheers,
>
> Andrew
>
>>
>> There's no fix per-se for this, jenkins is always a bit hit or miss
>> with it's latest LTS version working with the latest plugins.  Best
>> thing we can do is when we schedule our next jenkins LTS upgrade, to
>> try all the latest plugins again.
>>
>> Andrew
>>
>> On Fri, Sep 30, 2016 at 8:12 AM, Patrick Williams <patrick@stwcx.xyz> wrote:
>> >
>> > On Fri, Sep 30, 2016 at 11:16:08AM +0930, Joel Stanley wrote:
>> > >
>> > > Okay. Do we plan to test the individual projects as part of an
>> > > openbmc/openbmc build in the future?
>> > >
>> > Yes.  https://github.com/openbmc/openbmc/issues/382
>> >
>> > --
>> > Patrick Williams
>> _______________________________________________
>> openbmc mailing list
>> openbmc@lists.ozlabs.org
>> https://lists.ozlabs.org/listinfo/openbmc

Re: openpower.xyz updates

[Chris Smart]

[-- Attachment #1: Type: text/plain, Size: 2031 bytes --]

On Tue, Oct 11, 2016 at 10:14:18AM +1030, Andrew Jeffery wrote:
>On Mon, 2016-10-10 at 11:09 -0500, Andrew Geissler wrote:
>> I noticed our gerrit trigger job to verify userid's was broken again
>> this morning.  Looks like the trouble plugin (Git Client Plugin) was
>> upgraded again to the latest version.  Anyone out there know why/who
>> did it?  I'm going to put us back to 1.19.7 per my earlier comments.
>
>Joel and Chris Smart were looking at this yesterday. A job broke:
>
>https://openpower.xyz/job/openpower-op-build/184/console
>
>According to Chris it broke because of:
>
>https://bugs.eclipse.org/bugs/show_bug.cgi?id=428039
>
>    15:58 < csmart> joel, basically release() was replaced with close()
>    and so any plugin that uses git needs to be updated
>
>Looks like we're stuck between a rock and a hard place.
>

Yep, I updated the git client plugin because openpower build couldn't
perform any git operations.

I didn't expect that to affect any comms with other Jenkins instances
though, that is strange. Why does the gerrit userid plugin rely on on
git client? Do you have a log or backtrace? Might help to try and track
it down.

-c

>Cheers,
>
>Andrew
>
>>
>> There's no fix per-se for this, jenkins is always a bit hit or miss
>> with it's latest LTS version working with the latest plugins.  Best
>> thing we can do is when we schedule our next jenkins LTS upgrade, to
>> try all the latest plugins again.
>>
>> Andrew
>>
>> On Fri, Sep 30, 2016 at 8:12 AM, Patrick Williams <patrick@stwcx.xyz> wrote:
>> >
>> > On Fri, Sep 30, 2016 at 11:16:08AM +0930, Joel Stanley wrote:
>> > >
>> > > Okay. Do we plan to test the individual projects as part of an
>> > > openbmc/openbmc build in the future?
>> > >
>> > Yes.  https://github.com/openbmc/openbmc/issues/382
>> >
>> > --
>> > Patrick Williams
>> _______________________________________________
>> openbmc mailing list
>> openbmc@lists.ozlabs.org
>> https://lists.ozlabs.org/listinfo/openbmc



[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: openpower.xyz updates

[Andrew Geissler]

On Mon, Oct 10, 2016 at 8:36 PM, Chris Smart <chris@distroguy.com> wrote:
> On Tue, Oct 11, 2016 at 10:14:18AM +1030, Andrew Jeffery wrote:
>>
>> On Mon, 2016-10-10 at 11:09 -0500, Andrew Geissler wrote:
>>>
>>> I noticed our gerrit trigger job to verify userid's was broken again
>>> this morning.  Looks like the trouble plugin (Git Client Plugin) was
>>> upgraded again to the latest version.  Anyone out there know why/who
>>> did it?  I'm going to put us back to 1.19.7 per my earlier comments.
>>
>>
>> Joel and Chris Smart were looking at this yesterday. A job broke:
>>
>> https://openpower.xyz/job/openpower-op-build/184/console
>>
>> According to Chris it broke because of:
>>
>> https://bugs.eclipse.org/bugs/show_bug.cgi?id=428039
>>
>>    15:58 < csmart> joel, basically release() was replaced with close()
>>    and so any plugin that uses git needs to be updated
>>
>> Looks like we're stuck between a rock and a hard place.
>>
>
> Yep, I updated the git client plugin because openpower build couldn't
> perform any git operations.
>
> I didn't expect that to affect any comms with other Jenkins instances
> though, that is strange. Why does the gerrit userid plugin rely on on
> git client? Do you have a log or backtrace? Might help to try and track
> it down.

Here's the fail, it's a weird one:
https://openpower.xyz/job/openbmc-userid-verify/795/console

07:05:18  > git rev-parse FETCH_HEAD^{commit} # timeout=10
07:05:18 FATAL: org.eclipse.jgit.revwalk.RevWalk.release()V
07:05:18 java.lang.NoSuchMethodError:
org.eclipse.jgit.revwalk.RevWalk.release()V
07:05:18 at com.sonyericsson.hudson.plugins.gerrit.trigger.hudsontrigger.GerritTriggerBuildChooser$1.invoke(GerritTriggerBuildChooser.java:185)
07:05:18 at com.sonyericsson.hudson.plugins.gerrit.trigger.hudsontrigger.GerritTriggerBuildChooser$1.invoke(GerritTriggerBuildChooser.java:166)
07:05:18 at org.jenkinsci.plugins.gitclient.AbstractGitAPIImpl.withRepository(AbstractGitAPIImpl.java:27)
07:05:18 at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.withRepository(CliGitAPIImpl.java:64)
07:05:18 at com.sonyericsson.hudson.plugins.gerrit.trigger.hudsontrigger.GerritTriggerBuildChooser.getFirstParent(GerritTriggerBuildChooser.java:166)
...

No bug report on my part since I didn't think it was that big an
issue....I have been proven wrong.  I thing we have an imminent call,
lets touch base there and I'll get one open if needed.

>
> -c
>
>
>> Cheers,
>>
>> Andrew
>>
>>>
>>> There's no fix per-se for this, jenkins is always a bit hit or miss
>>> with it's latest LTS version working with the latest plugins.  Best
>>> thing we can do is when we schedule our next jenkins LTS upgrade, to
>>> try all the latest plugins again.
>>>
>>> Andrew
>>>
>>> On Fri, Sep 30, 2016 at 8:12 AM, Patrick Williams <patrick@stwcx.xyz>
>>> wrote:
>>> >
>>> > On Fri, Sep 30, 2016 at 11:16:08AM +0930, Joel Stanley wrote:
>>> > >
>>> > > Okay. Do we plan to test the individual projects as part of an
>>> > > openbmc/openbmc build in the future?
>>> > >
>>> > Yes.  https://github.com/openbmc/openbmc/issues/382
>>> >
>>> > --
>>> > Patrick Williams
>>> _______________________________________________
>>> openbmc mailing list
>>> openbmc@lists.ozlabs.org
>>> https://lists.ozlabs.org/listinfo/openbmc
>
>
>

Re: openpower.xyz updates

[Chris Smart]

On Mon, Oct 10, 2016 at 08:43:43PM -0500, Andrew Geissler wrote:
>On Mon, Oct 10, 2016 at 8:36 PM, Chris Smart <chris@distroguy.com> wrote:
>> On Tue, Oct 11, 2016 at 10:14:18AM +1030, Andrew Jeffery wrote:
>>>
>>> On Mon, 2016-10-10 at 11:09 -0500, Andrew Geissler wrote:
>>>>
>>>> I noticed our gerrit trigger job to verify userid's was broken again
>>>> this morning.  Looks like the trouble plugin (Git Client Plugin) was
>>>> upgraded again to the latest version.  Anyone out there know why/who
>>>> did it?  I'm going to put us back to 1.19.7 per my earlier comments.
>>>
>>>
>>> Joel and Chris Smart were looking at this yesterday. A job broke:
>>>
>>> https://openpower.xyz/job/openpower-op-build/184/console
>>>
>>> According to Chris it broke because of:
>>>
>>> https://bugs.eclipse.org/bugs/show_bug.cgi?id=428039
>>>
>>>    15:58 < csmart> joel, basically release() was replaced with close()
>>>    and so any plugin that uses git needs to be updated
>>>
>>> Looks like we're stuck between a rock and a hard place.
>>>
>>
>> Yep, I updated the git client plugin because openpower build couldn't
>> perform any git operations.
>>
>> I didn't expect that to affect any comms with other Jenkins instances
>> though, that is strange. Why does the gerrit userid plugin rely on on
>> git client? Do you have a log or backtrace? Might help to try and track
>> it down.
>
>Here's the fail, it's a weird one:
>https://openpower.xyz/job/openbmc-userid-verify/795/console
>
>07:05:18  > git rev-parse FETCH_HEAD^{commit} # timeout=10
>07:05:18 FATAL: org.eclipse.jgit.revwalk.RevWalk.release()V
>07:05:18 java.lang.NoSuchMethodError:
>org.eclipse.jgit.revwalk.RevWalk.release()V
>07:05:18 at com.sonyericsson.hudson.plugins.gerrit.trigger.hudsontrigger.GerritTriggerBuildChooser$1.invoke(GerritTriggerBuildChooser.java:185)
>07:05:18 at com.sonyericsson.hudson.plugins.gerrit.trigger.hudsontrigger.GerritTriggerBuildChooser$1.invoke(GerritTriggerBuildChooser.java:166)
>07:05:18 at org.jenkinsci.plugins.gitclient.AbstractGitAPIImpl.withRepository(AbstractGitAPIImpl.java:27)
>07:05:18 at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.withRepository(CliGitAPIImpl.java:64)
>07:05:18 at com.sonyericsson.hudson.plugins.gerrit.trigger.hudsontrigger.GerritTriggerBuildChooser.getFirstParent(GerritTriggerBuildChooser.java:166)
>...
>
>No bug report on my part since I didn't think it was that big an
>issue....I have been proven wrong.  I thing we have an imminent call,
>lets touch base there and I'll get one open if needed.
>

Yeah, so I think that's the same issue; release() was replaced with
close() in jgit and so any plugin that uses that needs to be updated.

This is what was breaking the openpower build and was fixed by updating
the git client plugin.

The log snippet you provided suggests to me that the gerrit trigger
plugin needs to be updated, you can see it's trying to call that
release() here:

07:05:18 FATAL: org.eclipse.jgit.revwalk.RevWalk.release()V
07:05:18 java.lang.NoSuchMethodError: org.eclipse.jgit.revwalk.RevWalk.release()V
07:05:18 at com.sonyericsson.hudson.plugins.gerrit.trigger.hudsontrigger.GerritTriggerBuildChooser$1.invoke(GerritTriggerBuildChooser.java:185)

-c

>>
>> -c
>>
>>
>>> Cheers,
>>>
>>> Andrew
>>>
>>>>
>>>> There's no fix per-se for this, jenkins is always a bit hit or miss
>>>> with it's latest LTS version working with the latest plugins.  Best
>>>> thing we can do is when we schedule our next jenkins LTS upgrade, to
>>>> try all the latest plugins again.
>>>>
>>>> Andrew
>>>>
>>>> On Fri, Sep 30, 2016 at 8:12 AM, Patrick Williams <patrick@stwcx.xyz>
>>>> wrote:
>>>> >
>>>> > On Fri, Sep 30, 2016 at 11:16:08AM +0930, Joel Stanley wrote:
>>>> > >
>>>> > > Okay. Do we plan to test the individual projects as part of an
>>>> > > openbmc/openbmc build in the future?
>>>> > >
>>>> > Yes.  https://github.com/openbmc/openbmc/issues/382
>>>> >
>>>> > --
>>>> > Patrick Williams
>>>> _______________________________________________
>>>> openbmc mailing list
>>>> openbmc@lists.ozlabs.org
>>>> https://lists.ozlabs.org/listinfo/openbmc
>>
>>
>>

Re: openpower.xyz updates

[Chris Smart]

On Tue, Oct 11, 2016 at 01:00:29PM +1100, Chris Smart wrote:
>On Mon, Oct 10, 2016 at 08:43:43PM -0500, Andrew Geissler wrote:
>>On Mon, Oct 10, 2016 at 8:36 PM, Chris Smart <chris@distroguy.com> wrote:
>>>On Tue, Oct 11, 2016 at 10:14:18AM +1030, Andrew Jeffery wrote:
>>>>
>>>>On Mon, 2016-10-10 at 11:09 -0500, Andrew Geissler wrote:
>>>>>
>>>>>I noticed our gerrit trigger job to verify userid's was broken again
>>>>>this morning.  Looks like the trouble plugin (Git Client Plugin) was
>>>>>upgraded again to the latest version.  Anyone out there know why/who
>>>>>did it?  I'm going to put us back to 1.19.7 per my earlier comments.
>>>>
>>>>
>>>>Joel and Chris Smart were looking at this yesterday. A job broke:
>>>>
>>>>https://openpower.xyz/job/openpower-op-build/184/console
>>>>
>>>>According to Chris it broke because of:
>>>>
>>>>https://bugs.eclipse.org/bugs/show_bug.cgi?id=428039
>>>>
>>>>   15:58 < csmart> joel, basically release() was replaced with close()
>>>>   and so any plugin that uses git needs to be updated
>>>>
>>>>Looks like we're stuck between a rock and a hard place.
>>>>
>>>
>>>Yep, I updated the git client plugin because openpower build couldn't
>>>perform any git operations.
>>>
>>>I didn't expect that to affect any comms with other Jenkins instances
>>>though, that is strange. Why does the gerrit userid plugin rely on on
>>>git client? Do you have a log or backtrace? Might help to try and track
>>>it down.
>>
>>Here's the fail, it's a weird one:
>>https://openpower.xyz/job/openbmc-userid-verify/795/console
>>
>>07:05:18  > git rev-parse FETCH_HEAD^{commit} # timeout=10
>>07:05:18 FATAL: org.eclipse.jgit.revwalk.RevWalk.release()V
>>07:05:18 java.lang.NoSuchMethodError:
>>org.eclipse.jgit.revwalk.RevWalk.release()V
>>07:05:18 at com.sonyericsson.hudson.plugins.gerrit.trigger.hudsontrigger.GerritTriggerBuildChooser$1.invoke(GerritTriggerBuildChooser.java:185)
>>07:05:18 at com.sonyericsson.hudson.plugins.gerrit.trigger.hudsontrigger.GerritTriggerBuildChooser$1.invoke(GerritTriggerBuildChooser.java:166)
>>07:05:18 at org.jenkinsci.plugins.gitclient.AbstractGitAPIImpl.withRepository(AbstractGitAPIImpl.java:27)
>>07:05:18 at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.withRepository(CliGitAPIImpl.java:64)
>>07:05:18 at com.sonyericsson.hudson.plugins.gerrit.trigger.hudsontrigger.GerritTriggerBuildChooser.getFirstParent(GerritTriggerBuildChooser.java:166)
>>...
>>
>>No bug report on my part since I didn't think it was that big an
>>issue....I have been proven wrong.  I thing we have an imminent call,
>>lets touch base there and I'll get one open if needed.
>>
>
>Yeah, so I think that's the same issue; release() was replaced with
>close() in jgit and so any plugin that uses that needs to be updated.
>
>This is what was breaking the openpower build and was fixed by updating
>the git client plugin.
>
>The log snippet you provided suggests to me that the gerrit trigger
>plugin needs to be updated, you can see it's trying to call that
>release() here:
>
>07:05:18 FATAL: org.eclipse.jgit.revwalk.RevWalk.release()V
>07:05:18 java.lang.NoSuchMethodError: org.eclipse.jgit.revwalk.RevWalk.release()V
>07:05:18 at com.sonyericsson.hudson.plugins.gerrit.trigger.hudsontrigger.GerritTriggerBuildChooser$1.invoke(GerritTriggerBuildChooser.java:185)
>

And to that effect:

Release 2.22.0 of GerritTrigger fixes this:

https://wiki.jenkins-ci.org/display/JENKINS/Gerrit+Trigger#GerritTrigger-Version2.22.0%28releasedAug172016%29


Version 2.22.0 (released Aug 17 2016)
 * Prepare for git client plugin 2.0.0 coexistence with 1.x (pull #296)

pull #296:
https://github.com/jenkinsci/gerrit-trigger-plugin/pull/296

So I would expect that updating gitclient and also gerrit-trigger would
solve these issues.

I don't know about the userid thing though - if you have a log for that,
feel free to send it through and I'll take a look.

-c

>-c
>
>>>
>>>-c
>>>
>>>
>>>>Cheers,
>>>>
>>>>Andrew
>>>>
>>>>>
>>>>>There's no fix per-se for this, jenkins is always a bit hit or miss
>>>>>with it's latest LTS version working with the latest plugins.  Best
>>>>>thing we can do is when we schedule our next jenkins LTS upgrade, to
>>>>>try all the latest plugins again.
>>>>>
>>>>>Andrew
>>>>>
>>>>>On Fri, Sep 30, 2016 at 8:12 AM, Patrick Williams <patrick@stwcx.xyz>
>>>>>wrote:
>>>>>>
>>>>>> On Fri, Sep 30, 2016 at 11:16:08AM +0930, Joel Stanley wrote:
>>>>>> >
>>>>>> > Okay. Do we plan to test the individual projects as part of an
>>>>>> > openbmc/openbmc build in the future?
>>>>>> >
>>>>>> Yes.  https://github.com/openbmc/openbmc/issues/382
>>>>>>
>>>>>> --
>>>>>> Patrick Williams
>>>>>_______________________________________________
>>>>>openbmc mailing list
>>>>>openbmc@lists.ozlabs.org
>>>>>https://lists.ozlabs.org/listinfo/openbmc
>>>
>>>
>>>

Re: openpower.xyz updates

[Andrew Geissler]

On Mon, Oct 10, 2016 at 9:00 PM, Chris Smart <chris@distroguy.com> wrote:
> On Mon, Oct 10, 2016 at 08:43:43PM -0500, Andrew Geissler wrote:
>>
>> On Mon, Oct 10, 2016 at 8:36 PM, Chris Smart <chris@distroguy.com> wrote:
>>>
>>> On Tue, Oct 11, 2016 at 10:14:18AM +1030, Andrew Jeffery wrote:
>>>>
>>>>
>>>> On Mon, 2016-10-10 at 11:09 -0500, Andrew Geissler wrote:
>>>>>
>>>>>
>>>>> I noticed our gerrit trigger job to verify userid's was broken again
>>>>> this morning.  Looks like the trouble plugin (Git Client Plugin) was
>>>>> upgraded again to the latest version.  Anyone out there know why/who
>>>>> did it?  I'm going to put us back to 1.19.7 per my earlier comments.
>>>>
>>>>
>>>>
>>>> Joel and Chris Smart were looking at this yesterday. A job broke:
>>>>
>>>> https://openpower.xyz/job/openpower-op-build/184/console
>>>>
>>>> According to Chris it broke because of:
>>>>
>>>> https://bugs.eclipse.org/bugs/show_bug.cgi?id=428039
>>>>
>>>>    15:58 < csmart> joel, basically release() was replaced with close()
>>>>    and so any plugin that uses git needs to be updated
>>>>
>>>> Looks like we're stuck between a rock and a hard place.
>>>>
>>>
>>> Yep, I updated the git client plugin because openpower build couldn't
>>> perform any git operations.
>>>
>>> I didn't expect that to affect any comms with other Jenkins instances
>>> though, that is strange. Why does the gerrit userid plugin rely on on
>>> git client? Do you have a log or backtrace? Might help to try and track
>>> it down.
>>
>>
>> Here's the fail, it's a weird one:
>> https://openpower.xyz/job/openbmc-userid-verify/795/console
>>
>> 07:05:18  > git rev-parse FETCH_HEAD^{commit} # timeout=10
>> 07:05:18 FATAL: org.eclipse.jgit.revwalk.RevWalk.release()V
>> 07:05:18 java.lang.NoSuchMethodError:
>> org.eclipse.jgit.revwalk.RevWalk.release()V
>> 07:05:18 at
>> com.sonyericsson.hudson.plugins.gerrit.trigger.hudsontrigger.GerritTriggerBuildChooser$1.invoke(GerritTriggerBuildChooser.java:185)
>> 07:05:18 at
>> com.sonyericsson.hudson.plugins.gerrit.trigger.hudsontrigger.GerritTriggerBuildChooser$1.invoke(GerritTriggerBuildChooser.java:166)
>> 07:05:18 at
>> org.jenkinsci.plugins.gitclient.AbstractGitAPIImpl.withRepository(AbstractGitAPIImpl.java:27)
>> 07:05:18 at
>> org.jenkinsci.plugins.gitclient.CliGitAPIImpl.withRepository(CliGitAPIImpl.java:64)
>> 07:05:18 at
>> com.sonyericsson.hudson.plugins.gerrit.trigger.hudsontrigger.GerritTriggerBuildChooser.getFirstParent(GerritTriggerBuildChooser.java:166)
>> ...
>>
>> No bug report on my part since I didn't think it was that big an
>> issue....I have been proven wrong.  I thing we have an imminent call,
>> lets touch base there and I'll get one open if needed.
>>
>
> Yeah, so I think that's the same issue; release() was replaced with
> close() in jgit and so any plugin that uses that needs to be updated.
>
> This is what was breaking the openpower build and was fixed by updating
> the git client plugin.
>
> The log snippet you provided suggests to me that the gerrit trigger
> plugin needs to be updated, you can see it's trying to call that
> release() here:
>
> 07:05:18 FATAL: org.eclipse.jgit.revwalk.RevWalk.release()V
> 07:05:18 java.lang.NoSuchMethodError:
> org.eclipse.jgit.revwalk.RevWalk.release()V
> 07:05:18 at
> com.sonyericsson.hudson.plugins.gerrit.trigger.hudsontrigger.GerritTriggerBuildChooser$1.invoke(GerritTriggerBuildChooser.java:185)
>

Yeah, sigh... for some reason, when we had the latest gerrit plugin, I
couldn't trigger off of "Comment Added" anymore which is what we use
for the CI build trigger.  I didn't find any obvious bug reports for
it when I looked.

Andrew

> -c
>
>
>>>
>>> -c
>>>
>>>
>>>> Cheers,
>>>>
>>>> Andrew
>>>>
>>>>>
>>>>> There's no fix per-se for this, jenkins is always a bit hit or miss
>>>>> with it's latest LTS version working with the latest plugins.  Best
>>>>> thing we can do is when we schedule our next jenkins LTS upgrade, to
>>>>> try all the latest plugins again.
>>>>>
>>>>> Andrew
>>>>>
>>>>> On Fri, Sep 30, 2016 at 8:12 AM, Patrick Williams <patrick@stwcx.xyz>
>>>>> wrote:
>>>>> >
>>>>> > On Fri, Sep 30, 2016 at 11:16:08AM +0930, Joel Stanley wrote:
>>>>> > >
>>>>> > > Okay. Do we plan to test the individual projects as part of an
>>>>> > > openbmc/openbmc build in the future?
>>>>> > >
>>>>> > Yes.  https://github.com/openbmc/openbmc/issues/382
>>>>> >
>>>>> > --
>>>>> > Patrick Williams
>>>>> _______________________________________________
>>>>> openbmc mailing list
>>>>> openbmc@lists.ozlabs.org
>>>>> https://lists.ozlabs.org/listinfo/openbmc
>>>
>>>
>>>
>>>
>

Re: openpower.xyz updates

[Joel Stanley]

On Tue, Oct 11, 2016 at 1:06 PM, Andrew Geissler <geissonator@gmail.com> wrote:
>> The log snippet you provided suggests to me that the gerrit trigger
>> plugin needs to be updated, you can see it's trying to call that
>> release() here:
>>
>> 07:05:18 FATAL: org.eclipse.jgit.revwalk.RevWalk.release()V
>> 07:05:18 java.lang.NoSuchMethodError:
>> org.eclipse.jgit.revwalk.RevWalk.release()V
>> 07:05:18 at
>> com.sonyericsson.hudson.plugins.gerrit.trigger.hudsontrigger.GerritTriggerBuildChooser$1.invoke(GerritTriggerBuildChooser.java:185)
>>
>
> Yeah, sigh... for some reason, when we had the latest gerrit plugin, I
> couldn't trigger off of "Comment Added" anymore which is what we use
> for the CI build trigger.  I didn't find any obvious bug reports for
> it when I looked.

Do you have the backtrace and log for that failure?

Cheers,

Joel

Re: openpower.xyz updates

[Joel Stanley]

On Tue, Oct 11, 2016 at 1:12 PM, Joel Stanley <joel@jms.id.au> wrote:
> On Tue, Oct 11, 2016 at 1:06 PM, Andrew Geissler <geissonator@gmail.com> wrote:
>>> The log snippet you provided suggests to me that the gerrit trigger
>>> plugin needs to be updated, you can see it's trying to call that
>>> release() here:
>>>
>>> 07:05:18 FATAL: org.eclipse.jgit.revwalk.RevWalk.release()V
>>> 07:05:18 java.lang.NoSuchMethodError:
>>> org.eclipse.jgit.revwalk.RevWalk.release()V
>>> 07:05:18 at
>>> com.sonyericsson.hudson.plugins.gerrit.trigger.hudsontrigger.GerritTriggerBuildChooser$1.invoke(GerritTriggerBuildChooser.java:185)
>>>
>>
>> Yeah, sigh... for some reason, when we had the latest gerrit plugin, I
>> couldn't trigger off of "Comment Added" anymore which is what we use
>> for the CI build trigger.  I didn't find any obvious bug reports for
>> it when I looked.
>
> Do you have the backtrace and log for that failure?

Patrick, can you look at the logs from the gerrit side? Is the
connection succeeding?

Andrew G, I took a look at the Gerrit plugin between 2.21.1 and 2.22.

https://github.com/jenkinsci/gerrit-trigger-plugin/compare/gerrit-trigger-2.21.1...gerrit-trigger-2.22.0

https://issues.jenkins-ci.org/browse/JENKINS-35379

Perhaps we could try bisecting the changes made to the plugin to work
out what went wrong?

Cheers,

Joel

Re: openpower.xyz updates

[Joel Stanley]

On Tue, Oct 11, 2016 at 5:23 PM, Joel Stanley <joel@jms.id.au> wrote:
> On Tue, Oct 11, 2016 at 1:12 PM, Joel Stanley <joel@jms.id.au> wrote:
> Andrew G, I took a look at the Gerrit plugin between 2.21.1 and 2.22.
>
> https://github.com/jenkinsci/gerrit-trigger-plugin/compare/gerrit-trigger-2.21.1...gerrit-trigger-2.22.0
>
> https://issues.jenkins-ci.org/browse/JENKINS-35379
>
> Perhaps we could try bisecting the changes made to the plugin to work
> out what went wrong?

Overnight we upgraded to the latest LTS release of Jenkins to see if
that fixed our issue. It did not.

I then tried to downgrade the plugins, but this caused our
github-oauth plugin to downgrade (I think the new Jenkins release
tries to downgrade dependencies when downgrading a plugin). This broke
login, which relies on oauth. I tried to re-upgrade the plugin, but it
was still broken - an exception was thrown as the plugin was unable to
parse the oauth config file. For reference, the signatures looked like
this:

Caused by: java.lang.ClassCastException:
org.acegisecurity.providers.anonymous.AnonymousAuthenticationToken
cannot be cast to org.jenkinsci.plugins.GithubAuthenticationToken

com.google.common.util.concurrent.UncheckedExecutionException:
java.lang.ClassCastException:
org.acegisecurity.providers.anonymous.AnonymousAuthenticationToken
cannot be cast to org.jenkinsci.plugins.GithubAuthenticationToken

We tried removing the config file, modifying it, restoring from
backup, to no avail.

To our luck, the github-oauth plugin was updated over night. The new
release Just Worked so we ran with that.

Back to the core issue of the Gerrit - Jenkins failure to kick of
build jobs. I found out more about the design of the Gerrit - Jenkins
system today.

We have a Gerrit Trigger plugin that listens to the event stream from
Gerrit over ssh. When it sees a new patch it runs a job that checks
the submitter against a list of whitelisted users (those in the
OpenBMC team on Github). If the user is approved, it updates the
commit with the ok-to-test label.

Another job uses the Trigger plugin to look for the ok-to-test label.
When it sees this label in the stream, it kicks off a build job. The
reason Andrew separates out the two is so an admin can manually tick
the ok-to-test box allowing drive by contributions to be tested.

The bug is due to the newer version of the plugin not seeing these
events. We're not sure why that happens, but put a workaround in place
that causes the jobs to still trigger while we sort it out.

Cheers,

Joel