Hello, On Mon, May 20, 2019 at 4:38 PM Veronika Kabatova wrote: > > > ----- Original Message ----- > > From: "Tom Gall" > > To: kernelci@groups.io, "Dan Rue" > > Cc: "Tim Bird" , vkabatov@redhat.com, > automated-testing@yoctoproject.org, info@kernelci.org > > Sent: Wednesday, May 15, 2019 11:06:33 PM > > Subject: Re: A common place for CI results? > > > > > > > > > On May 15, 2019, at 3:33 PM, Dan Rue wrote: > > > > > > On Tue, May 14, 2019 at 11:01:35PM +0000, Tim.Bird@sony.com wrote: > > >> > > >> > > >>> -----Original Message----- > > >>> From: Veronika Kabatova > > >>> > > >>> Hi, > > >>> > > >>> as we know from this list, there's plenty CI systems doing some > testing > > >>> on > > >>> the > > >>> upstream kernels (and maybe some others we don't know about). > > >>> > > >>> It would be great if there was a single common place where all the CI > > >>> systems > > >>> can put their results. This would make it much easier for the kernel > > >>> maintainers and developers to see testing status since they only > need to > > >>> check one place instead of having a list of sites/mailing lists where > > >>> each CI > > >>> posts their contributions. > > >>> > > >>> > > >>> A few weeks ago, with some people we've been talking about > kernelci.org > > >>> being > > >>> in a good place to act as the central upstream kernel CI piece that > most > > >>> maintainers already know about. So I'm wondering if it would be > possible > > >>> for > > >>> kernelci to also act as an aggregator of all results? There's > already an > > >>> API > > >>> for publishing a report [0] so it shouldn't be too hard to adjust it > to > > >>> handle and show more information. I also found the beta version for > test > > >>> results [1] so actually, most of the needed functionality seems to be > > >>> already > > >>> there. Since there will be multiple CI systems, the source and > contact > > >>> point > > >>> for the contributor (so maintainers know whom to ask about results if > > >>> needed) > > >>> would likely be the only missing essential data point. > > >>> > > >>> > > >>> The common place for results would also make it easier for new CI > systems > > >>> to > > >>> get involved with upstream. There are likely other companies out > there > > >>> running > > >>> some tests on kernel internally but don't publish the results > anywhere. > > >>> Only > > >>> adding some API calls into their code (with the data they are > allowed to > > >>> publish) would make it very simple for them to start contributing. > If we > > >>> want > > >>> to make them interested, the starting point needs to be trivial. > > >>> Different > > >>> companies have different setups and policies and they might not be > able > > >>> to > > >>> fulfill arbitrary requirements so they opt to not get involved at > all, > > >>> which > > >>> is a shame because their results can be useful. After the initial > > >>> "onboarding" > > >>> step they might be willing to contribute more and more too. > > >>> > > >>> > > >>> Please let me know if the idea makes sense or if something similar is > > >>> already > > >>> in plans. I'd be happy to contribute to the effort because I believe > it > > >>> would > > >>> make everyone's life easier and we'd all benefit from it (and maybe > > >>> someone > > >>> else from my team would be willing to help out too if needed). > > >> > > >> I never responded to this, > > > > > > yea, you did. ;) > > > > > >> but this sounds like a really good idea to me. I don't care much which > > >> backend we aggregate to, but it would be good as a community to start > > >> using one service to start with. It would help to find issues with > > >> the API, or the results schema, if multiple people started using it. > > >> > > >> I know that people using Fuego are sending data to their own instances > > >> of KernelCI. But I don't know what the issues are for sending this > > >> data to a shared KernelCI service. > > >> > > >> I would be interested in hooking up my lab to send Fuego results to > > >> KernelCI. This would be a good exercise. I'm not sure what the next > > >> steps would be, but maybe we could discuss this on the next automated > > >> testing conference call. > > > > > > OK here's my idea. > > > > > > I don't personally think kernelci (or LKFT) are set up to aggregate > > > results currently. We have too many assumptions about where tests are > > > coming from, how things are built, etc. In other words, dealing with > > > noisy data is going to be non-trivial in any existing project. > > > > I completely agree. > > > > This is a good point. I'm totally fine with having a separate independent > place for aggregation. > > > > I would propose aggregating data into something like google's BigQuery. > > > This has a few benefits: > > > - Non-opinionated place to hold structured data > > > - Allows many downstream use-cases > > > - Managed hosting, and data is publicly available > > > - Storage is sponsored by google as a part of > > > https://cloud.google.com/bigquery/public-data/ > > > - First 1TB of query per 'project' is free, and users pay for more > > > queries than that > > > > I very much like this idea. I do lots of android kernel testing > > and being able to work with / compare / contribute to what > > is essentially a pile of data in BQ would be great. As an > > end user working with the data I’d also have lots of dash > > board options to customize and share queries with others. > > > > > With storage taken care of, how do we get the data in? > > > > > First, we'll need some canonical data structure defined. I would > > > approach defining the canonical structure in conjunction with the first > > > few projects that are interested in contributing their results. Each > > > project will have an ETL pipeline which will extract the test results > > > from a given project (such as kernelci, lkft, etc), translate it into > > > the canonical data structure, and load it into the google bigquery > > > dataset at a regular interval or in real-time. The translation layer is > > > where things like test names are handled. > > > > +1, exactly how I imagined this part. > > > Exactly. I would hope that the various projects that are producing > > data would be motived to plug in. After all, it makes the data > > they are producing more useful and available to a larger group > > of people. > > > > > The things this leaves me wanting are: > > > - raw data storage. It would be nice if raw data were stored somewhere > > > permanent in some intermediary place so that later implementations > > > could happen, and for data that doesn't fit into whatever structure we > > > end up with. > > > > I agree. > > +1 > > > > > > - time, to actually try it and find the gaps. This is just an idea I've > > > been thinking about. Anyone with experience here that can help flesh > > > this out? > > > > I’m willing to lend a hand. > > > > Thanks for starting up a specific proposal! I agree with everything that > was > brought up. I'll try to find time to participate in the implementation part > too (although my experience with data storage is.. limited, I should be > able > to help out with the structure prototyping and maybe other parts too). > This all sounds great: having a common location to store the results that is scalable and definitions of test case names. However, there is a whole layer of logic above and around this which KernelCI does, and I'm sure other CI systems also do with some degree of overlap. So it seems to me that solving how to deal with the results is only one piece in the puzzle to get a common CI architecture for upstream kernel testing. Sorry I'm a bit late to the party so I'll add my 2¢ here... Around the end of last year I made this document and mentioned it on this list, about making KernelCI more modular: https://docs.google.com/document/d/15F42HdHTO6NbSL53_iLl77lfe1XQKdWaHAf7XCNkKD8/edit?usp=sharing https://groups.io/g/kernelci/topic/kernelci_modular_pipeline/29692355 The idea is to make it possible to have alternative components in the KernelCI "pipeline". Right now, KernelCI has these components: * Jenkins job to monitor git branches and build kernels * LAVA to run tests * Custom backend and storage server to keep binaries and data * Custom web frontend to show the results They could all be replaced or used in conjunction with alternative build systems, database engines, test lab schedulers and dashboards. The key thing is the code orchestrating all this, which is kept in the kernelci-core repository. For example, when a change has been detected in a tree, rather than triggering kernel builds on Jenkins there could be a request sent to another build system to do that elsewhere. Likewise, when some builds are ready to be tested, jobs could be scheduled in non-LAVA labs simply by sending another kind of HTTP request than the ones we're currently sending to the LAVA APIs. This could be easily described in some config files, in fact we already have one with the list of labs where to submit jobs to. Builds and tests are configured in YAML files in KernelCI, which could easily be extended too with new attributes. The big advantage of having a central way to orchestrate all this is that results are going to be consistent and higher-level features can be enabled: each tree will be sampled at the same commit, so we don't end up with one CI lab running a version of mainline a few patches than another one etc... It means we can expand some KernelCI features to a larger ecosystem of CI labs, such as: * redundancy checks when the same test / hardware is tested in multiple places (say, if a RPi fails to boot in a LAVA lab but not in CKI's lab...) * regression tracking across the whole spectrum of CI labs * common reports for each single kernel revision being tested * bisections extended to non-LAVA labs It feels like a diagram would be needed to really give an idea of how this would work. APIs and callback mechanisms would need to be well defined to have clear entry points for the various components in a modular system like this. I think we would be able to reuse some of the things currently used by KernelCI and improve them, taking into account what other CI labs have been doing (LKFT, CKI...). I'm only scratching the surface here, but I wanted to raise this point to see if others shared the same vision. It would be unfortunate if we came up with a great solution focused on results, but then realised that it had big design limitations when trying to add more abstract functionality across all the contributing CI labs. Well, I tried to keep it short - hope this makes any sense. Cheers, Guillaume