* Hourly, daily, weekly monitoring @ 2019-03-04 13:20 Guillaume Tucker 2019-03-05 1:53 ` Kevin Hilman 2019-03-05 12:14 ` Mark Brown 0 siblings, 2 replies; 6+ messages in thread From: Guillaume Tucker @ 2019-03-04 13:20 UTC (permalink / raw) To: kernelci; +Cc: Linus Walleij Hello, All the branches currently build on kernelci.org are monitored for new changes every hour. While this has shown over time to be a good frequency for mainline branches, some subsystem ones may be updated a lot more often. Monitoring them every hour would cause a lot of unnecessary builds and tests. This topic was recently brought up again[1] around the addition of the GPIO branches, which Linus Walleij said could be monitored once a day instead. So it seems like it would be beneficial to add some parameters to the YAML build configuration to be able to tell how often each branch needs to be checked for new commits. There should also be some command line option in kci_build to help implementing that in an automated build system such as Jenkins Pipeline (say, to know the time when a branch needs to be checked again...). I'm not entirely sure how much flexibility Jenkins can offer in that respect, but at least having 3 versions of the monitor job that runs every hour, every day and every week should cover all the cases. If possible, we may be able to implement something that dynamically schedules the next check for each branch. How does that sound? Best wishes, Guillaume [1] https://groups.io/g/kernelci/topic/adding_tree_to_kernelci/29605396 ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Hourly, daily, weekly monitoring 2019-03-04 13:20 Hourly, daily, weekly monitoring Guillaume Tucker @ 2019-03-05 1:53 ` Kevin Hilman 2019-03-05 12:14 ` Mark Brown 1 sibling, 0 replies; 6+ messages in thread From: Kevin Hilman @ 2019-03-05 1:53 UTC (permalink / raw) To: kernelci, guillaume.tucker, kernelci; +Cc: Linus Walleij "Guillaume Tucker" <guillaume.tucker@gmail.com> writes: > Hello, > > All the branches currently build on kernelci.org are monitored > for new changes every hour. While this has shown over time to be > a good frequency for mainline branches, some subsystem ones may > be updated a lot more often. Monitoring them every hour would > cause a lot of unnecessary builds and tests. This topic was > recently brought up again[1] around the addition of the GPIO > branches, which Linus Walleij said could be monitored once a day > instead. > > So it seems like it would be beneficial to add some parameters to > the YAML build configuration to be able to tell how often each > branch needs to be checked for new commits. There should also be > some command line option in kci_build to help implementing that > in an automated build system such as Jenkins Pipeline (say, to > know the time when a branch needs to be checked again...). > > I'm not entirely sure how much flexibility Jenkins can offer in > that respect, but at least having 3 versions of the monitor job > that runs every hour, every day and every week should cover all > the cases. If possible, we may be able to implement something > that dynamically schedules the next check for each branch. > > How does that sound? Sounds great. This is something we've been needing for awhile. Kevin ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Hourly, daily, weekly monitoring 2019-03-04 13:20 Hourly, daily, weekly monitoring Guillaume Tucker 2019-03-05 1:53 ` Kevin Hilman @ 2019-03-05 12:14 ` Mark Brown 2019-03-05 22:04 ` Dan Rue 1 sibling, 1 reply; 6+ messages in thread From: Mark Brown @ 2019-03-05 12:14 UTC (permalink / raw) To: kernelci, guillaume.tucker; +Cc: Linus Walleij [-- Attachment #1: Type: text/plain, Size: 1188 bytes --] On Mon, Mar 04, 2019 at 01:20:25PM +0000, Guillaume Tucker wrote: > I'm not entirely sure how much flexibility Jenkins can offer in > that respect, but at least having 3 versions of the monitor job > that runs every hour, every day and every week should cover all > the cases. If possible, we may be able to implement something > that dynamically schedules the next check for each branch. One big concern I have with this is latency. One of the common cases where people don't need builds all the time is when they're mainly looking for checks before they submit pull requests. For those if you might have to wait almost a day before we even queue the build it might be a bit of an issue. If it was just a "skip this tree if we built it in the past X time" check that wouldn't be such an issue, it's just if the daily check runs at some fixed time each day or whatever that it might add huge extra latency. Another idea I just thought of but I'm not sure is practical would be to only check some trees if the Jenkins queue is less than some number of builds - that way if we're busy we won't add extra load, but it feels like it's more trouble than it's worth to implement fairly. [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 488 bytes --] ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Hourly, daily, weekly monitoring 2019-03-05 12:14 ` Mark Brown @ 2019-03-05 22:04 ` Dan Rue 2019-03-06 9:16 ` Guillaume Tucker 0 siblings, 1 reply; 6+ messages in thread From: Dan Rue @ 2019-03-05 22:04 UTC (permalink / raw) To: kernelci, broonie; +Cc: guillaume.tucker, Linus Walleij On Tue, Mar 05, 2019 at 12:14:12PM +0000, Mark Brown wrote: > On Mon, Mar 04, 2019 at 01:20:25PM +0000, Guillaume Tucker wrote: > > > I'm not entirely sure how much flexibility Jenkins can offer in > > that respect, but at least having 3 versions of the monitor job > > that runs every hour, every day and every week should cover all > > the cases. If possible, we may be able to implement something > > that dynamically schedules the next check for each branch. > > One big concern I have with this is latency. One of the common cases > where people don't need builds all the time is when they're mainly > looking for checks before they submit pull requests. For those if you > might have to wait almost a day before we even queue the build it might > be a bit of an issue. If it was just a "skip this tree if we built it > in the past X time" check that wouldn't be such an issue, it's just if > the daily check runs at some fixed time each day or whatever that it > might add huge extra latency. > > Another idea I just thought of but I'm not sure is practical would be to > only check some trees if the Jenkins queue is less than some number of > builds - that way if we're busy we won't add extra load, but it feels > like it's more trouble than it's worth to implement fairly. That's a good point. I wonder what happens when something doesn't fit in a hourly/daily/weekly box. It could also cause a daily/weekly bottleneck if they're all scheduled at the same time. Perhaps each tree gets a cooling off period defined in e.g. seconds, and it could be defaulted to current behavior of 1 hour. If a tree is triggered but its cooling off period hasn't passed, the trigger is either ignored or deferred. This would also let us increase the frequency of the build trigger to something like every 5 minutes. This way, load is evened out a bit (no spikes when 'weekly' or even 'hourly' runs), kernelci is more realtime (to Mark's point), and configuration is granular and per-tree (we can still offer standard cool-off periods, of course). The only state that has to be tracked is the time of the last build per tree, though I have no idea about the ease of implementation. Dan ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Hourly, daily, weekly monitoring 2019-03-05 22:04 ` Dan Rue @ 2019-03-06 9:16 ` Guillaume Tucker 2019-03-06 17:00 ` Mark Brown 0 siblings, 1 reply; 6+ messages in thread From: Guillaume Tucker @ 2019-03-06 9:16 UTC (permalink / raw) To: Dan Rue; +Cc: kernelci, broonie, Linus Walleij [-- Attachment #1: Type: text/plain, Size: 5194 bytes --] On Tue, Mar 5, 2019 at 10:04 PM Dan Rue <dan.rue@linaro.org> wrote: > On Tue, Mar 05, 2019 at 12:14:12PM +0000, Mark Brown wrote: > > On Mon, Mar 04, 2019 at 01:20:25PM +0000, Guillaume Tucker wrote: > > > > > I'm not entirely sure how much flexibility Jenkins can offer in > > > that respect, but at least having 3 versions of the monitor job > > > that runs every hour, every day and every week should cover all > > > the cases. If possible, we may be able to implement something > > > that dynamically schedules the next check for each branch. > > > > One big concern I have with this is latency. One of the common cases > > where people don't need builds all the time is when they're mainly > > looking for checks before they submit pull requests. For those if you > > might have to wait almost a day before we even queue the build it might > > be a bit of an issue. If it was just a "skip this tree if we built it > > in the past X time" check that wouldn't be such an issue, it's just if > > the daily check runs at some fixed time each day or whatever that it > > might add huge extra latency. > > > > Another idea I just thought of but I'm not sure is practical would be to > > only check some trees if the Jenkins queue is less than some number of > > builds - that way if we're busy we won't add extra load, but it feels > > like it's more trouble than it's worth to implement fairly. > These are different use-cases, some people have said that they wanted their branches checked every morning (linusw) or every Monday (media). For real-time feedback, we need to do something quite different indeed. With the ability to whitelist some defconfigs and arches, we could have quick builds for some maintainers' trees to optimise the turnaround time / test coverage ratio. And yes, a mechanism to skip builds on arbitrary criteria would seem to be quite hard to design with fair rules. And at the end of the day, if we don't have enough build power to cope with the load, even the best rules would just be moving the problem somewhere else. What I think we should have is some kind of OOM system, to track that if the queue keeps increasing from one day to the next then some builds need to be killed, because there really isn't any choice in that situation. We're still running at about 75% of our capacity and builds can be cancelled manually if anything goes very wrong, so it's not a practical concern right now. > I wonder what happens when something doesn't fit in a > hourly/daily/weekly box. It could also cause a daily/weekly bottleneck > if they're all scheduled at the same time. > Everything kind of fits in an hourly box, because that's a small enough interval to start building things shortly after a new revision was pushed. Having longer periods is useful for people who don't want intermediate versions to go through the test system, or don't want to set up a branch just for KernelCI. Perhaps each tree gets a cooling off period defined in e.g. seconds, and > it could be defaulted to current behavior of 1 hour. If a tree is > triggered but its cooling off period hasn't passed, the trigger is > either ignored or deferred. This would also let us increase the > frequency of the build trigger to something like every 5 minutes. > If we want that kind of speed then we should be using git hooks to trigger builds directly rather than polling continuously. > This way, load is evened out a bit (no spikes when 'weekly' or even > 'hourly' runs), kernelci is more realtime (to Mark's point), and > configuration is granular and per-tree (we can still offer standard > cool-off periods, of course). > The issue here is that it would be doing the opposite of what some people want. For example, when a branch gets updated several times during a day, some developers don't want the first ones to be tested but rather let the system wait until the evening before doing a build so they get the results the next morning (see Linus' comment on the GPIO thread). Also, periodic checks don't have to all be at the same time, they could be evened out within their polling period. The only state that has to be tracked is the time of the last build per > tree, though I have no idea about the ease of implementation. > The implementation will depend on a lot of things, i.e. the build automation tool (Jenkins or other) and the backend / storage server where previous builds are stored. At the moment we're keeping a file containing the commit sha from last time a branch was sampled, it wouldn't be hard to add date and time information to that for example (or get the last modified date of that file). The first thing to do imo is to have the ability to specify how often a tree should be checked in the YAML config file. We could reuse cron's syntax for that, or have simple but rough values like "daily" to let the implentation decide at what to actually do the checks. As mentioned before, in some cases we could be using a hook to trigger a build directly from the git server every time a commit is pushed (like a real CI system). We should also have a way to define that in the YAML config, probably by not setting any polling interval. Guillaume [-- Attachment #2: Type: text/html, Size: 7443 bytes --] ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Hourly, daily, weekly monitoring 2019-03-06 9:16 ` Guillaume Tucker @ 2019-03-06 17:00 ` Mark Brown 0 siblings, 0 replies; 6+ messages in thread From: Mark Brown @ 2019-03-06 17:00 UTC (permalink / raw) To: Guillaume Tucker; +Cc: Dan Rue, kernelci, Linus Walleij [-- Attachment #1: Type: text/plain, Size: 4211 bytes --] On Wed, Mar 06, 2019 at 09:16:18AM +0000, Guillaume Tucker wrote: > On Tue, Mar 5, 2019 at 10:04 PM Dan Rue <dan.rue@linaro.org> wrote: > > On Tue, Mar 05, 2019 at 12:14:12PM +0000, Mark Brown wrote: > > > On Mon, Mar 04, 2019 at 01:20:25PM +0000, Guillaume Tucker wrote: > > > One big concern I have with this is latency. One of the common cases > > > where people don't need builds all the time is when they're mainly > > > looking for checks before they submit pull requests. For those if you > > > might have to wait almost a day before we even queue the build it might > > > be a bit of an issue. If it was just a "skip this tree if we built it > > > in the past X time" check that wouldn't be such an issue, it's just if > > > the daily check runs at some fixed time each day or whatever that it > > > might add huge extra latency. > These are different use-cases, some people have said that they > wanted their branches checked every morning (linusw) or every > Monday (media). For real-time feedback, we need to do something > quite different indeed. With the ability to whitelist some > defconfigs and arches, we could have quick builds for some > maintainers' trees to optimise the turnaround time / test > coverage ratio. I don't know that we need super real time for most things (though I'm sure some people would find it useful), more just that we don't want to be adding huge delays in - it was the scenario where you miss your single check for the day and add on a whole day of latency on that worried me. I think both Linus' case and the media case would be handled fine by the skip built in last X time suggestion. > What I think we should have is some kind of OOM system, to track > that if the queue keeps increasing from one day to the next then > some builds need to be killed, because there really isn't any > choice in that situation. We're still running at about 75% of > our capacity and builds can be cancelled manually if anything > goes very wrong, so it's not a practical concern right now. Yeah, though we have been building up some very big backlogs which users have been noticing (Google and rmk have both reported things recently). > > This way, load is evened out a bit (no spikes when 'weekly' or even > > 'hourly' runs), kernelci is more realtime (to Mark's point), and > > configuration is granular and per-tree (we can still offer standard > > cool-off periods, of course). > The issue here is that it would be doing the opposite of what > some people want. For example, when a branch gets updated > several times during a day, some developers don't want the first > ones to be tested but rather let the system wait until the > evening before doing a build so they get the results the next > morning (see Linus' comment on the GPIO thread). I'm not sure how many would mind intermediate results being generated, some will though. > > The only state that has to be tracked is the time of the last build per > > tree, though I have no idea about the ease of implementation. > The implementation will depend on a lot of things, i.e. the build > automation tool (Jenkins or other) and the backend / storage > server where previous builds are stored. At the moment we're > keeping a file containing the commit sha from last time a branch > was sampled, it wouldn't be hard to add date and time information > to that for example (or get the last modified date of that file). Yes, that's what I was thinking - just add a check for the modification time on (I'm doing that with some of my scripting for my upstream handling, works pretty effectively). Jenkins does feel like a bit of a limitation sometimes with the simple FIFO queue it has. > As mentioned before, in some cases we could be using a hook to > trigger a build directly from the git server every time a commit > is pushed (like a real CI system). We should also have a way to > define that in the YAML config, probably by not setting any > polling interval. My faith in the robustness of computers and the internet is such that I'd suggest making it a separate config so that we can also have a (fairly long) poll time and fall back to polling if the callback stops working for some reason :/ [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 488 bytes --] ^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2019-03-06 17:00 UTC | newest] Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2019-03-04 13:20 Hourly, daily, weekly monitoring Guillaume Tucker 2019-03-05 1:53 ` Kevin Hilman 2019-03-05 12:14 ` Mark Brown 2019-03-05 22:04 ` Dan Rue 2019-03-06 9:16 ` Guillaume Tucker 2019-03-06 17:00 ` Mark Brown
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.