Re: openpower.xyz updates

From: Joel Stanley <joel@jms.id.au>
To: Andrew Geissler <geissonator@gmail.com>
Cc: Chris Smart <chris@distroguy.com>,
	Andrew Jeffery <andrew@aj.id.au>,
	 Patrick Williams <patrick@stwcx.xyz>,
	OpenBMC Maillist <openbmc@lists.ozlabs.org>
Subject: Re: openpower.xyz updates
Date: Wed, 12 Oct 2016 16:36:00 +1100	[thread overview]
Message-ID: <CACPK8XeNz44e4yMCvKxAGEO+z51FkgDdgvzzVpBKZusCt27epw@mail.gmail.com> (raw)
In-Reply-To: <CACPK8XeB1cFLJQesU2ELhX=F7Nuqx8vxXsVuL+jxvOLJdvridg@mail.gmail.com>

On Tue, Oct 11, 2016 at 5:23 PM, Joel Stanley <joel@jms.id.au> wrote:
> On Tue, Oct 11, 2016 at 1:12 PM, Joel Stanley <joel@jms.id.au> wrote:
> Andrew G, I took a look at the Gerrit plugin between 2.21.1 and 2.22.
>
> https://github.com/jenkinsci/gerrit-trigger-plugin/compare/gerrit-trigger-2.21.1...gerrit-trigger-2.22.0
>
> https://issues.jenkins-ci.org/browse/JENKINS-35379
>
> Perhaps we could try bisecting the changes made to the plugin to work
> out what went wrong?

Overnight we upgraded to the latest LTS release of Jenkins to see if
that fixed our issue. It did not.

I then tried to downgrade the plugins, but this caused our
github-oauth plugin to downgrade (I think the new Jenkins release
tries to downgrade dependencies when downgrading a plugin). This broke
login, which relies on oauth. I tried to re-upgrade the plugin, but it
was still broken - an exception was thrown as the plugin was unable to
parse the oauth config file. For reference, the signatures looked like
this:

Caused by: java.lang.ClassCastException:
org.acegisecurity.providers.anonymous.AnonymousAuthenticationToken
cannot be cast to org.jenkinsci.plugins.GithubAuthenticationToken

com.google.common.util.concurrent.UncheckedExecutionException:
java.lang.ClassCastException:
org.acegisecurity.providers.anonymous.AnonymousAuthenticationToken
cannot be cast to org.jenkinsci.plugins.GithubAuthenticationToken

We tried removing the config file, modifying it, restoring from
backup, to no avail.

To our luck, the github-oauth plugin was updated over night. The new
release Just Worked so we ran with that.

Back to the core issue of the Gerrit - Jenkins failure to kick of
build jobs. I found out more about the design of the Gerrit - Jenkins
system today.

We have a Gerrit Trigger plugin that listens to the event stream from
Gerrit over ssh. When it sees a new patch it runs a job that checks
the submitter against a list of whitelisted users (those in the
OpenBMC team on Github). If the user is approved, it updates the
commit with the ok-to-test label.

Another job uses the Trigger plugin to look for the ok-to-test label.
When it sees this label in the stream, it kicks off a build job. The
reason Andrew separates out the two is so an admin can manually tick
the ok-to-test box allowing drive by contributions to be tested.

The bug is due to the newer version of the plugin not seeing these
events. We're not sure why that happens, but put a workaround in place
that causes the jobs to still trigger while we sort it out.

Cheers,

Joel