Hourly, daily, weekly monitoring

Hourly, daily, weekly monitoring

[Guillaume Tucker]

Hello,

All the branches currently build on kernelci.org are monitored
for new changes every hour.  While this has shown over time to be
a good frequency for mainline branches, some subsystem ones may
be updated a lot more often.  Monitoring them every hour would
cause a lot of unnecessary builds and tests.  This topic was
recently brought up again[1] around the addition of the GPIO
branches, which Linus Walleij said could be monitored once a day
instead.

So it seems like it would be beneficial to add some parameters to
the YAML build configuration to be able to tell how often each
branch needs to be checked for new commits.  There should also be
some command line option in kci_build to help implementing that
in an automated build system such as Jenkins Pipeline (say, to
know the time when a branch needs to be checked again...).

I'm not entirely sure how much flexibility Jenkins can offer in
that respect, but at least having 3 versions of the monitor job
that runs every hour, every day and every week should cover all
the cases.  If possible, we may be able to implement something
that dynamically schedules the next check for each branch.

How does that sound?

Best wishes,
Guillaume

[1] https://groups.io/g/kernelci/topic/adding_tree_to_kernelci/29605396

Re: Hourly, daily, weekly monitoring

[Kevin Hilman]

"Guillaume Tucker" <guillaume.tucker@gmail.com> writes:

> Hello,
>
> All the branches currently build on kernelci.org are monitored
> for new changes every hour.  While this has shown over time to be
> a good frequency for mainline branches, some subsystem ones may
> be updated a lot more often.  Monitoring them every hour would
> cause a lot of unnecessary builds and tests.  This topic was
> recently brought up again[1] around the addition of the GPIO
> branches, which Linus Walleij said could be monitored once a day
> instead.
>
> So it seems like it would be beneficial to add some parameters to
> the YAML build configuration to be able to tell how often each
> branch needs to be checked for new commits.  There should also be
> some command line option in kci_build to help implementing that
> in an automated build system such as Jenkins Pipeline (say, to
> know the time when a branch needs to be checked again...).
>
> I'm not entirely sure how much flexibility Jenkins can offer in
> that respect, but at least having 3 versions of the monitor job
> that runs every hour, every day and every week should cover all
> the cases.  If possible, we may be able to implement something
> that dynamically schedules the next check for each branch.
>
> How does that sound?

Sounds great.  This is something we've been needing for awhile.

Kevin

Re: Hourly, daily, weekly monitoring

[Mark Brown]

[-- Attachment #1: Type: text/plain, Size: 1188 bytes --]

On Mon, Mar 04, 2019 at 01:20:25PM +0000, Guillaume Tucker wrote:

> I'm not entirely sure how much flexibility Jenkins can offer in
> that respect, but at least having 3 versions of the monitor job
> that runs every hour, every day and every week should cover all
> the cases.  If possible, we may be able to implement something
> that dynamically schedules the next check for each branch.

One big concern I have with this is latency.  One of the common cases
where people don't need builds all the time is when they're mainly
looking for checks before they submit pull requests.  For those if you
might have to wait almost a day before we even queue the build it might
be a bit of an issue.  If it was just a "skip this tree if we built it
in the past X time" check that wouldn't be such an issue, it's just if
the daily check runs at some fixed time each day or whatever that it
might add huge extra latency.

Another idea I just thought of but I'm not sure is practical would be to
only check some trees if the Jenkins queue is less than some number of
builds - that way if we're busy we won't add extra load, but it feels
like it's more trouble than it's worth to implement fairly.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: Hourly, daily, weekly monitoring

[Dan Rue]

On Tue, Mar 05, 2019 at 12:14:12PM +0000, Mark Brown wrote:
> On Mon, Mar 04, 2019 at 01:20:25PM +0000, Guillaume Tucker wrote:
> 
> > I'm not entirely sure how much flexibility Jenkins can offer in
> > that respect, but at least having 3 versions of the monitor job
> > that runs every hour, every day and every week should cover all
> > the cases.  If possible, we may be able to implement something
> > that dynamically schedules the next check for each branch.
> 
> One big concern I have with this is latency.  One of the common cases
> where people don't need builds all the time is when they're mainly
> looking for checks before they submit pull requests.  For those if you
> might have to wait almost a day before we even queue the build it might
> be a bit of an issue.  If it was just a "skip this tree if we built it
> in the past X time" check that wouldn't be such an issue, it's just if
> the daily check runs at some fixed time each day or whatever that it
> might add huge extra latency.
> 
> Another idea I just thought of but I'm not sure is practical would be to
> only check some trees if the Jenkins queue is less than some number of
> builds - that way if we're busy we won't add extra load, but it feels
> like it's more trouble than it's worth to implement fairly.

That's a good point.

I wonder what happens when something doesn't fit in a
hourly/daily/weekly box. It could also cause a daily/weekly bottleneck
if they're all scheduled at the same time.

Perhaps each tree gets a cooling off period defined in e.g. seconds, and
it could be defaulted to current behavior of 1 hour. If a tree is
triggered but its cooling off period hasn't passed, the trigger is
either ignored or deferred. This would also let us increase the
frequency of the build trigger to something like every 5 minutes.

This way, load is evened out a bit (no spikes when 'weekly' or even
'hourly' runs), kernelci is more realtime (to Mark's point), and
configuration is granular and per-tree (we can still offer standard
cool-off periods, of course).

The only state that has to be tracked is the time of the last build per
tree, though I have no idea about the ease of implementation.

Dan

Re: Hourly, daily, weekly monitoring

[Guillaume Tucker]

[-- Attachment #1: Type: text/plain, Size: 5194 bytes --]

On Tue, Mar 5, 2019 at 10:04 PM Dan Rue <dan.rue@linaro.org> wrote:

> On Tue, Mar 05, 2019 at 12:14:12PM +0000, Mark Brown wrote:
> > On Mon, Mar 04, 2019 at 01:20:25PM +0000, Guillaume Tucker wrote:
> >
> > > I'm not entirely sure how much flexibility Jenkins can offer in
> > > that respect, but at least having 3 versions of the monitor job
> > > that runs every hour, every day and every week should cover all
> > > the cases.  If possible, we may be able to implement something
> > > that dynamically schedules the next check for each branch.
> >
> > One big concern I have with this is latency.  One of the common cases
> > where people don't need builds all the time is when they're mainly
> > looking for checks before they submit pull requests.  For those if you
> > might have to wait almost a day before we even queue the build it might
> > be a bit of an issue.  If it was just a "skip this tree if we built it
> > in the past X time" check that wouldn't be such an issue, it's just if
> > the daily check runs at some fixed time each day or whatever that it
> > might add huge extra latency.
> >
> > Another idea I just thought of but I'm not sure is practical would be to
> > only check some trees if the Jenkins queue is less than some number of
> > builds - that way if we're busy we won't add extra load, but it feels
> > like it's more trouble than it's worth to implement fairly.
>

These are different use-cases, some people have said that they
wanted their branches checked every morning (linusw) or every
Monday (media).  For real-time feedback, we need to do something
quite different indeed.  With the ability to whitelist some
defconfigs and arches, we could have quick builds for some
maintainers' trees to optimise the turnaround time / test
coverage ratio.

And yes, a mechanism to skip builds on arbitrary criteria would
seem to be quite hard to design with fair rules.  And at the end
of the day, if we don't have enough build power to cope with the
load, even the best rules would just be moving the problem
somewhere else.

What I think we should have is some kind of OOM system, to track
that if the queue keeps increasing from one day to the next then
some builds need to be killed, because there really isn't any
choice in that situation.  We're still running at about 75% of
our capacity and builds can be cancelled manually if anything
goes very wrong, so it's not a practical concern right now.


> I wonder what happens when something doesn't fit in a
> hourly/daily/weekly box. It could also cause a daily/weekly bottleneck
> if they're all scheduled at the same time.
>

Everything kind of fits in an hourly box, because that's a small
enough interval to start building things shortly after a new
revision was pushed.  Having longer periods is useful for people
who don't want intermediate versions to go through the test
system, or don't want to set up a branch just for KernelCI.

Perhaps each tree gets a cooling off period defined in e.g. seconds, and
> it could be defaulted to current behavior of 1 hour. If a tree is
> triggered but its cooling off period hasn't passed, the trigger is
> either ignored or deferred. This would also let us increase the
> frequency of the build trigger to something like every 5 minutes.
>

If we want that kind of speed then we should be using git hooks
to trigger builds directly rather than polling continuously.


> This way, load is evened out a bit (no spikes when 'weekly' or even
> 'hourly' runs), kernelci is more realtime (to Mark's point), and
> configuration is granular and per-tree (we can still offer standard
> cool-off periods, of course).
>

The issue here is that it would be doing the opposite of what
some people want.  For example, when a branch gets updated
several times during a day, some developers don't want the first
ones to be tested but rather let the system wait until the
evening before doing a build so they get the results the next
morning (see Linus' comment on the GPIO thread).

Also, periodic checks don't have to all be at the same time, they
could be evened out within their polling period.

The only state that has to be tracked is the time of the last build per
> tree, though I have no idea about the ease of implementation.
>

The implementation will depend on a lot of things, i.e. the build
automation tool (Jenkins or other) and the backend / storage
server where previous builds are stored.  At the moment we're
keeping a file containing the commit sha from last time a branch
was sampled, it wouldn't be hard to add date and time information
to that for example (or get the last modified date of that file).

The first thing to do imo is to have the ability to specify how
often a tree should be checked in the YAML config file.  We could
reuse cron's syntax for that, or have simple but rough values
like "daily" to let the implentation decide at what to actually
do the checks.

As mentioned before, in some cases we could be using a hook to
trigger a build directly from the git server every time a commit
is pushed (like a real CI system).  We should also have a way to
define that in the YAML config, probably by not setting any
polling interval.

Guillaume

[-- Attachment #2: Type: text/html, Size: 7443 bytes --]

Re: Hourly, daily, weekly monitoring

[Mark Brown]

[-- Attachment #1: Type: text/plain, Size: 4211 bytes --]

On Wed, Mar 06, 2019 at 09:16:18AM +0000, Guillaume Tucker wrote:
> On Tue, Mar 5, 2019 at 10:04 PM Dan Rue <dan.rue@linaro.org> wrote:
> > On Tue, Mar 05, 2019 at 12:14:12PM +0000, Mark Brown wrote:
> > > On Mon, Mar 04, 2019 at 01:20:25PM +0000, Guillaume Tucker wrote:

> > > One big concern I have with this is latency.  One of the common cases
> > > where people don't need builds all the time is when they're mainly
> > > looking for checks before they submit pull requests.  For those if you
> > > might have to wait almost a day before we even queue the build it might
> > > be a bit of an issue.  If it was just a "skip this tree if we built it
> > > in the past X time" check that wouldn't be such an issue, it's just if
> > > the daily check runs at some fixed time each day or whatever that it
> > > might add huge extra latency.

> These are different use-cases, some people have said that they
> wanted their branches checked every morning (linusw) or every
> Monday (media).  For real-time feedback, we need to do something
> quite different indeed.  With the ability to whitelist some
> defconfigs and arches, we could have quick builds for some
> maintainers' trees to optimise the turnaround time / test
> coverage ratio.

I don't know that we need super real time for most things (though I'm
sure some people would find it useful), more just that we don't want to
be adding huge delays in - it was the scenario where you miss your
single check for the day and add on a whole day of latency on that
worried me.  I think both Linus' case and the media case would be
handled fine by the skip built in last X time suggestion.

> What I think we should have is some kind of OOM system, to track
> that if the queue keeps increasing from one day to the next then
> some builds need to be killed, because there really isn't any
> choice in that situation.  We're still running at about 75% of
> our capacity and builds can be cancelled manually if anything
> goes very wrong, so it's not a practical concern right now.

Yeah, though we have been building up some very big backlogs which users
have been noticing (Google and rmk have both reported things recently).

> > This way, load is evened out a bit (no spikes when 'weekly' or even
> > 'hourly' runs), kernelci is more realtime (to Mark's point), and
> > configuration is granular and per-tree (we can still offer standard
> > cool-off periods, of course).

> The issue here is that it would be doing the opposite of what
> some people want.  For example, when a branch gets updated
> several times during a day, some developers don't want the first
> ones to be tested but rather let the system wait until the
> evening before doing a build so they get the results the next
> morning (see Linus' comment on the GPIO thread).

I'm not sure how many would mind intermediate results being generated,
some will though.

> > The only state that has to be tracked is the time of the last build per
> > tree, though I have no idea about the ease of implementation.

> The implementation will depend on a lot of things, i.e. the build
> automation tool (Jenkins or other) and the backend / storage
> server where previous builds are stored.  At the moment we're
> keeping a file containing the commit sha from last time a branch
> was sampled, it wouldn't be hard to add date and time information
> to that for example (or get the last modified date of that file).

Yes, that's what I was thinking - just add a check for the modification
time on (I'm doing that with some of my scripting for my upstream
handling, works pretty effectively).  Jenkins does feel like a bit of a
limitation sometimes with the simple FIFO queue it has.

> As mentioned before, in some cases we could be using a hook to
> trigger a build directly from the git server every time a commit
> is pushed (like a real CI system).  We should also have a way to
> define that in the YAML config, probably by not setting any
> polling interval.

My faith in the robustness of computers and the internet is such that
I'd suggest making it a separate config so that we can also have a
(fairly long) poll time and fall back to polling if the callback stops
working for some reason :/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]