CKI hackfest @Plumbers invite

CKI hackfest @Plumbers invite

[Veronika Kabatova]

Hi,

as some of you have heard, CKI Project is planning hackfest CI meetings after
Plumbers conference this year (Sept. 12-13). We would like to invite everyone
who has interest in CI for kernel to come and join us.

The early agenda with summary is at the end of the email. If you think there's
something important missing let us know! Also let us know in case you'd want to
lead any of the sessions, we'd be happy to delegate out some work :)


Please send us an email as soon as you decide to come and feel free to invite
other people who should be present. We are not planning to cap the attendance
right now but need to solve the logistics based on the interest. The event is
free to attend, no additional registration except letting us know is needed.

Feel free to contact us if you have any questions,
Veronika
CKI Project


-----------------------------------------------------------
Here is an early agenda we put together:
- Introductions
- Common place for upstream results, result publishing in general
  - The discussion on the mailing list is going strong so we might be able to
    substitute this session for a different one in case everything is solved by
    September.
- Test result interpretation and bug detection
  - How to autodetect infrastructure failures, regressions/new bugs and test
    bugs? How to handle continuous failures due to known bugs in both tests and
    kernel? What's your solution? Can people always trust the results they
    receive?
- Getting results to developers/maintainers
  - Aimed at kernel developers and maintainers, share your feedback and
    expectations.
  - How much data should be sent in the initial communication vs. a click away
    in a dashboard? Do you want incremental emails with new results as they come
    in?
  - What about adding checks to tested patches in Patchwork when patch series
    are being tested?
  - Providing enough data/script to reproduce the failure. What if special HW
    is needed?
- Onboarding new kernel trees to test
  - Aimed at kernel developers and maintainers.
  - Which trees are most prone to bring in new problems? Which are the most
    critical ones? Do you want them to be tested? Which tests do you feel are
    most beneficial for specific trees or in general?
- Security when testing untrusted patches
  - How do we merge, compile, and test patches that have untrusted code in them
    and have not yet been reviewed? How do we avoid abuse of systems,
    information theft, or other damage?
  - Check out the original patch that sparked the discussion at
    https://patchwork.ozlabs.org/patch/862123/
- Avoiding effort duplication
  - Food for thought by GregKH
  - X different CI systems running ${TEST} on latest stable kernel on x86_64
    might look useless on the first look but is it? AMD/Intel CPUs, different
    network cards, different graphic drivers, compilers, kernel configuration...
    How do we distribute the workload to avoid doing the same thing all over
    again while still running in enough different environments to get the most
    coverage?
- Common hardware pools
  - Is this something people are interested in? Would be helpful especially for
    HW that's hard to access, eg. ppc64le or s390x systems. Companies could also
    sing up to share their HW for testing to ensure kernel works with their
    products.

Re: CKI hackfest @Plumbers invite

[Greg KH]

On Tue, May 21, 2019 at 10:54:12AM -0400, Veronika Kabatova wrote:
> Hi,
> 
> as some of you have heard, CKI Project is planning hackfest CI meetings after
> Plumbers conference this year (Sept. 12-13). We would like to invite everyone
> who has interest in CI for kernel to come and join us.
> 
> The early agenda with summary is at the end of the email. If you think there's
> something important missing let us know! Also let us know in case you'd want to
> lead any of the sessions, we'd be happy to delegate out some work :)
> 
> 
> Please send us an email as soon as you decide to come and feel free to invite
> other people who should be present. We are not planning to cap the attendance
> right now but need to solve the logistics based on the interest. The event is
> free to attend, no additional registration except letting us know is needed.
> 
> Feel free to contact us if you have any questions,
> Veronika
> CKI Project
> 
> 
> -----------------------------------------------------------
> Here is an early agenda we put together:
> - Introductions
> - Common place for upstream results, result publishing in general
>   - The discussion on the mailing list is going strong so we might be able to
>     substitute this session for a different one in case everything is solved by
>     September.
> - Test result interpretation and bug detection
>   - How to autodetect infrastructure failures, regressions/new bugs and test
>     bugs? How to handle continuous failures due to known bugs in both tests and
>     kernel? What's your solution? Can people always trust the results they
>     receive?
> - Getting results to developers/maintainers
>   - Aimed at kernel developers and maintainers, share your feedback and
>     expectations.
>   - How much data should be sent in the initial communication vs. a click away
>     in a dashboard? Do you want incremental emails with new results as they come
>     in?
>   - What about adding checks to tested patches in Patchwork when patch series
>     are being tested?
>   - Providing enough data/script to reproduce the failure. What if special HW
>     is needed?
> - Onboarding new kernel trees to test
>   - Aimed at kernel developers and maintainers.
>   - Which trees are most prone to bring in new problems? Which are the most
>     critical ones? Do you want them to be tested? Which tests do you feel are
>     most beneficial for specific trees or in general?
> - Security when testing untrusted patches
>   - How do we merge, compile, and test patches that have untrusted code in them
>     and have not yet been reviewed? How do we avoid abuse of systems,
>     information theft, or other damage?
>   - Check out the original patch that sparked the discussion at
>     https://patchwork.ozlabs.org/patch/862123/
> - Avoiding effort duplication
>   - Food for thought by GregKH

So I guess I'm going to be there?

Ok, fair enough, I'll be present, looks good :)

thanks,

greg k-h

Re: CKI hackfest @Plumbers invite

[Veronika Kabatova]

----- Original Message -----
> From: "Greg KH" <gregkh@linuxfoundation.org>
> To: "Veronika Kabatova" <vkabatov@redhat.com>
> Cc: automated-testing@yoctoproject.org, info@kernelci.org, "Tim Bird" <Tim.Bird@sony.com>, khilamn@baylibre.org,
> syzkaller@googlegroups.com, lkp@lists.01.org, stable@vger.kernel.org, "Laura Abbott" <labbott@redhat.com>, "Eliska
> Slobodova" <eslobodo@redhat.com>, "CKI Project" <cki-project@redhat.com>
> Sent: Tuesday, May 21, 2019 6:47:04 PM
> Subject: Re: CKI hackfest @Plumbers invite
> 
> On Tue, May 21, 2019 at 10:54:12AM -0400, Veronika Kabatova wrote:
> > Hi,
> > 
> > as some of you have heard, CKI Project is planning hackfest CI meetings
> > after
> > Plumbers conference this year (Sept. 12-13). We would like to invite
> > everyone
> > who has interest in CI for kernel to come and join us.
> > 
> > The early agenda with summary is at the end of the email. If you think
> > there's
> > something important missing let us know! Also let us know in case you'd
> > want to
> > lead any of the sessions, we'd be happy to delegate out some work :)
> > 
> > 
> > Please send us an email as soon as you decide to come and feel free to
> > invite
> > other people who should be present. We are not planning to cap the
> > attendance
> > right now but need to solve the logistics based on the interest. The event
> > is
> > free to attend, no additional registration except letting us know is
> > needed.
> > 
> > Feel free to contact us if you have any questions,
> > Veronika
> > CKI Project
> > 
> > 
> > -----------------------------------------------------------
> > Here is an early agenda we put together:
> > - Introductions
> > - Common place for upstream results, result publishing in general
> >   - The discussion on the mailing list is going strong so we might be able
> >   to
> >     substitute this session for a different one in case everything is
> >     solved by
> >     September.
> > - Test result interpretation and bug detection
> >   - How to autodetect infrastructure failures, regressions/new bugs and
> >   test
> >     bugs? How to handle continuous failures due to known bugs in both tests
> >     and
> >     kernel? What's your solution? Can people always trust the results they
> >     receive?
> > - Getting results to developers/maintainers
> >   - Aimed at kernel developers and maintainers, share your feedback and
> >     expectations.
> >   - How much data should be sent in the initial communication vs. a click
> >   away
> >     in a dashboard? Do you want incremental emails with new results as they
> >     come
> >     in?
> >   - What about adding checks to tested patches in Patchwork when patch
> >   series
> >     are being tested?
> >   - Providing enough data/script to reproduce the failure. What if special
> >   HW
> >     is needed?
> > - Onboarding new kernel trees to test
> >   - Aimed at kernel developers and maintainers.
> >   - Which trees are most prone to bring in new problems? Which are the most
> >     critical ones? Do you want them to be tested? Which tests do you feel
> >     are
> >     most beneficial for specific trees or in general?
> > - Security when testing untrusted patches
> >   - How do we merge, compile, and test patches that have untrusted code in
> >   them
> >     and have not yet been reviewed? How do we avoid abuse of systems,
> >     information theft, or other damage?
> >   - Check out the original patch that sparked the discussion at
> >     https://patchwork.ozlabs.org/patch/862123/
> > - Avoiding effort duplication
> >   - Food for thought by GregKH
> 
> So I guess I'm going to be there?
> 
> Ok, fair enough, I'll be present, looks good :)
> 

Glad to hear that!
You always have valuable feedback and ideas to offer so we are definitely
looking forward to having you there :)


Veronika

> thanks,
> 
> greg k-h
> 

RE: CKI hackfest @Plumbers invite

[Tim.Bird]

> -----Original Message-----
> From: Veronika Kabatova 
> 
> Hi,
> 
> as some of you have heard, CKI Project is planning hackfest CI meetings after
> Plumbers conference this year (Sept. 12-13). We would like to invite
> everyone
> who has interest in CI for kernel to come and join us.
> 
> The early agenda with summary is at the end of the email. If you think there's
> something important missing let us know! Also let us know in case you'd
> want to
> lead any of the sessions, we'd be happy to delegate out some work :)
> 
> 
> Please send us an email as soon as you decide to come and feel free to invite
> other people who should be present. We are not planning to cap the
> attendance
> right now but need to solve the logistics based on the interest. The event is
> free to attend, no additional registration except letting us know is needed.
> 
> Feel free to contact us if you have any questions,

I plan to come to the event.

> -----------------------------------------------------------
> Here is an early agenda we put together:
> - Introductions
> - Common place for upstream results, result publishing in general
>   - The discussion on the mailing list is going strong so we might be able to
>     substitute this session for a different one in case everything is solved by
>     September.
> - Test result interpretation and bug detection
>   - How to autodetect infrastructure failures, regressions/new bugs and test
>     bugs? How to handle continuous failures due to known bugs in both tests
> and
>     kernel? What's your solution? Can people always trust the results they
>     receive?
> - Getting results to developers/maintainers
>   - Aimed at kernel developers and maintainers, share your feedback and
>     expectations.
>   - How much data should be sent in the initial communication vs. a click away
>     in a dashboard? Do you want incremental emails with new results as they
> come
>     in?
>   - What about adding checks to tested patches in Patchwork when patch
> series
>     are being tested?
>   - Providing enough data/script to reproduce the failure. What if special HW
>     is needed?
> - Onboarding new kernel trees to test
>   - Aimed at kernel developers and maintainers.
>   - Which trees are most prone to bring in new problems? Which are the most
>     critical ones? Do you want them to be tested? Which tests do you feel are
>     most beneficial for specific trees or in general?
> - Security when testing untrusted patches
>   - How do we merge, compile, and test patches that have untrusted code in
> them
>     and have not yet been reviewed? How do we avoid abuse of systems,
>     information theft, or other damage?
>   - Check out the original patch that sparked the discussion at
>     https://patchwork.ozlabs.org/patch/862123/
> - Avoiding effort duplication
>   - Food for thought by GregKH
>   - X different CI systems running ${TEST} on latest stable kernel on x86_64
>     might look useless on the first look but is it? AMD/Intel CPUs, different
>     network cards, different graphic drivers, compilers, kernel configuration...
>     How do we distribute the workload to avoid doing the same thing all over
>     again while still running in enough different environments to get the most
>     coverage?
> - Common hardware pools
>   - Is this something people are interested in? Would be helpful especially for
>     HW that's hard to access, eg. ppc64le or s390x systems. Companies could
> also
>     sing up to share their HW for testing to ensure kernel works with their
>     products.

I have strong opinions on some of these, but maybe only useful experience
in a few areas.  Fuego has 2 separate notions, which we call "skiplists"
and "pass criteria", which have to do with this bullet:

- How to autodetect infrastructure failures, regressions/new bugs and test
     bugs? How to handle continuous failures due to known bugs in both
     tests and kernel? What's your solution? Can people always trust the results they
     receive?

I'd be happy to discuss this, if it's desired.

Otherwise, I've recently been working on standards for "test definition",
which defines the data and meta-data associated with a test.   I could talk
about where I'm at with that, if people are interested.

Let me know what you think.
 -- Tim

Re: CKI hackfest @Plumbers invite

[Veronika Kabatova]

----- Original Message -----
> From: "Tim Bird" <Tim.Bird@sony.com>
> To: vkabatov@redhat.com, automated-testing@yoctoproject.org, info@kernelci.org, khilamn@baylibre.org,
> syzkaller@googlegroups.com, lkp@lists.01.org, stable@vger.kernel.org, labbott@redhat.com
> Cc: eslobodo@redhat.com, cki-project@redhat.com
> Sent: Friday, May 24, 2019 10:17:04 PM
> Subject: RE: CKI hackfest @Plumbers invite
> 
> 
> 
> > -----Original Message-----
> > From: Veronika Kabatova
> > 
> > Hi,
> > 
> > as some of you have heard, CKI Project is planning hackfest CI meetings
> > after
> > Plumbers conference this year (Sept. 12-13). We would like to invite
> > everyone
> > who has interest in CI for kernel to come and join us.
> > 
> > The early agenda with summary is at the end of the email. If you think
> > there's
> > something important missing let us know! Also let us know in case you'd
> > want to
> > lead any of the sessions, we'd be happy to delegate out some work :)
> > 
> > 
> > Please send us an email as soon as you decide to come and feel free to
> > invite
> > other people who should be present. We are not planning to cap the
> > attendance
> > right now but need to solve the logistics based on the interest. The event
> > is
> > free to attend, no additional registration except letting us know is
> > needed.
> > 
> > Feel free to contact us if you have any questions,
> 
> I plan to come to the event.
> 
> > -----------------------------------------------------------
> > Here is an early agenda we put together:
> > - Introductions
> > - Common place for upstream results, result publishing in general
> >   - The discussion on the mailing list is going strong so we might be able
> >   to
> >     substitute this session for a different one in case everything is
> >     solved by
> >     September.
> > - Test result interpretation and bug detection
> >   - How to autodetect infrastructure failures, regressions/new bugs and
> >   test
> >     bugs? How to handle continuous failures due to known bugs in both tests
> > and
> >     kernel? What's your solution? Can people always trust the results they
> >     receive?
> > - Getting results to developers/maintainers
> >   - Aimed at kernel developers and maintainers, share your feedback and
> >     expectations.
> >   - How much data should be sent in the initial communication vs. a click
> >   away
> >     in a dashboard? Do you want incremental emails with new results as they
> > come
> >     in?
> >   - What about adding checks to tested patches in Patchwork when patch
> > series
> >     are being tested?
> >   - Providing enough data/script to reproduce the failure. What if special
> >   HW
> >     is needed?
> > - Onboarding new kernel trees to test
> >   - Aimed at kernel developers and maintainers.
> >   - Which trees are most prone to bring in new problems? Which are the most
> >     critical ones? Do you want them to be tested? Which tests do you feel
> >     are
> >     most beneficial for specific trees or in general?
> > - Security when testing untrusted patches
> >   - How do we merge, compile, and test patches that have untrusted code in
> > them
> >     and have not yet been reviewed? How do we avoid abuse of systems,
> >     information theft, or other damage?
> >   - Check out the original patch that sparked the discussion at
> >     https://patchwork.ozlabs.org/patch/862123/
> > - Avoiding effort duplication
> >   - Food for thought by GregKH
> >   - X different CI systems running ${TEST} on latest stable kernel on
> >   x86_64
> >     might look useless on the first look but is it? AMD/Intel CPUs,
> >     different
> >     network cards, different graphic drivers, compilers, kernel
> >     configuration...
> >     How do we distribute the workload to avoid doing the same thing all
> >     over
> >     again while still running in enough different environments to get the
> >     most
> >     coverage?
> > - Common hardware pools
> >   - Is this something people are interested in? Would be helpful especially
> >   for
> >     HW that's hard to access, eg. ppc64le or s390x systems. Companies could
> > also
> >     sing up to share their HW for testing to ensure kernel works with their
> >     products.
> 
> I have strong opinions on some of these, but maybe only useful experience
> in a few areas.  Fuego has 2 separate notions, which we call "skiplists"
> and "pass criteria", which have to do with this bullet:
> 
> - How to autodetect infrastructure failures, regressions/new bugs and test
>      bugs? How to handle continuous failures due to known bugs in both
>      tests and kernel? What's your solution? Can people always trust the
>      results they
>      receive?
> 
> I'd be happy to discuss this, if it's desired.
> 
> Otherwise, I've recently been working on standards for "test definition",
> which defines the data and meta-data associated with a test.   I could talk
> about where I'm at with that, if people are interested.
> 

Sounds great! I added both your points to the agenda as I do think they have
a place here. The list of items is growing so I hope we can still fit
everything into the two days we planned :)


See you there!
Veronika

> Let me know what you think.
>  -- Tim
> 
> 

Re: CKI hackfest @Plumbers invite

[Dmitry Vyukov]

On Mon, May 27, 2019 at 1:52 PM Veronika Kabatova <vkabatov@redhat.com> wrote:
> ----- Original Message -----
> > From: "Tim Bird" <Tim.Bird@sony.com>
> > To: vkabatov@redhat.com, automated-testing@yoctoproject.org, info@kernelci.org, khilamn@baylibre.org,
> > syzkaller@googlegroups.com, lkp@lists.01.org, stable@vger.kernel.org, labbott@redhat.com
> > Cc: eslobodo@redhat.com, cki-project@redhat.com
> > Sent: Friday, May 24, 2019 10:17:04 PM
> > Subject: RE: CKI hackfest @Plumbers invite
> >
> >
> >
> > > -----Original Message-----
> > > From: Veronika Kabatova
> > >
> > > Hi,
> > >
> > > as some of you have heard, CKI Project is planning hackfest CI meetings
> > > after
> > > Plumbers conference this year (Sept. 12-13). We would like to invite
> > > everyone
> > > who has interest in CI for kernel to come and join us.
> > >
> > > The early agenda with summary is at the end of the email. If you think
> > > there's
> > > something important missing let us know! Also let us know in case you'd
> > > want to
> > > lead any of the sessions, we'd be happy to delegate out some work :)
> > >
> > >
> > > Please send us an email as soon as you decide to come and feel free to
> > > invite
> > > other people who should be present. We are not planning to cap the
> > > attendance
> > > right now but need to solve the logistics based on the interest. The event
> > > is
> > > free to attend, no additional registration except letting us know is
> > > needed.
> > >
> > > Feel free to contact us if you have any questions,
> >
> > I plan to come to the event.
> >
> > > -----------------------------------------------------------
> > > Here is an early agenda we put together:
> > > - Introductions
> > > - Common place for upstream results, result publishing in general
> > >   - The discussion on the mailing list is going strong so we might be able
> > >   to
> > >     substitute this session for a different one in case everything is
> > >     solved by
> > >     September.
> > > - Test result interpretation and bug detection
> > >   - How to autodetect infrastructure failures, regressions/new bugs and
> > >   test
> > >     bugs? How to handle continuous failures due to known bugs in both tests
> > > and
> > >     kernel? What's your solution? Can people always trust the results they
> > >     receive?
> > > - Getting results to developers/maintainers
> > >   - Aimed at kernel developers and maintainers, share your feedback and
> > >     expectations.
> > >   - How much data should be sent in the initial communication vs. a click
> > >   away
> > >     in a dashboard? Do you want incremental emails with new results as they
> > > come
> > >     in?
> > >   - What about adding checks to tested patches in Patchwork when patch
> > > series
> > >     are being tested?
> > >   - Providing enough data/script to reproduce the failure. What if special
> > >   HW
> > >     is needed?
> > > - Onboarding new kernel trees to test
> > >   - Aimed at kernel developers and maintainers.
> > >   - Which trees are most prone to bring in new problems? Which are the most
> > >     critical ones? Do you want them to be tested? Which tests do you feel
> > >     are
> > >     most beneficial for specific trees or in general?
> > > - Security when testing untrusted patches
> > >   - How do we merge, compile, and test patches that have untrusted code in
> > > them
> > >     and have not yet been reviewed? How do we avoid abuse of systems,
> > >     information theft, or other damage?
> > >   - Check out the original patch that sparked the discussion at
> > >     https://patchwork.ozlabs.org/patch/862123/
> > > - Avoiding effort duplication
> > >   - Food for thought by GregKH
> > >   - X different CI systems running ${TEST} on latest stable kernel on
> > >   x86_64
> > >     might look useless on the first look but is it? AMD/Intel CPUs,
> > >     different
> > >     network cards, different graphic drivers, compilers, kernel
> > >     configuration...
> > >     How do we distribute the workload to avoid doing the same thing all
> > >     over
> > >     again while still running in enough different environments to get the
> > >     most
> > >     coverage?


Hi Veronika,

All are great questions that we need to resolve!

I am also very much concerned about duplication in 2 other dimensions
with the current approach to kernel testing:

1. If X different CI systems running ${TEST}, developers receive X
reports about the same breakage from X different directions, in
different formats, of different quality, at slightly different times
and somebody needs to act on all of them in some way. The more CI
systems we have, the more run meaningful number of tests and do
automatic reporting, the more and more duplicates developers get.

2. Effort duplication between implementation of different CI systems.
Doing a proper and really good CI is very hard. This includes all
questions that you mentioned here, and fine tuning of all of that,
refining reporting, bisection, onboarding of different test suites,
onboarding of different dynamic/static analysis tools and much more.
Last but not least is duplication of processes related to these CIs.
Speaking of my experience with syzbot, this is extremely hard and
takes years. And we really can't expose a developer to 27 different
systems and slightly different processes (this would mean they follow
0 of these processes).
This is further complicated by the fact that kernel tests are
fragmented, so it's not possible to, say, simply run all kernel tests.
And kernel processes are fragmented, e.g. you mentioned patchwork, but
not all subsystems use patchwork, so it's not possible to simply
extend a CI to all subsystems. And some aspects of the current kernel
development process notoriously complicate automation of things that
really should be trivial. For example, if you have
github/gitlab/gerrit, you can hook into arrival of each new change and
pull exact code state. Done. For kernel some changes appear on
patchwork, some don't, some are duplicated on multiple patchworks,
some duplicated in a weird way on the same patchwork, some non-patches
appear on patchwork because it's confused, and last but not least you
can't really apply any of them because none of them include base
tree/commit info. Handling just this requires lots of effort, guessing
on coffee grounds and heuristics that need to be refined over time.
The total complexity of doing it just once, with all resources
combined and dev process re-shaped to cooperate is close to off-scale.

Do you see these points as a problem too? Or am I exaggerating matters?






> > > - Common hardware pools
> > >   - Is this something people are interested in? Would be helpful especially
> > >   for
> > >     HW that's hard to access, eg. ppc64le or s390x systems. Companies could
> > > also
> > >     sing up to share their HW for testing to ensure kernel works with their
> > >     products.
> >
> > I have strong opinions on some of these, but maybe only useful experience
> > in a few areas.  Fuego has 2 separate notions, which we call "skiplists"
> > and "pass criteria", which have to do with this bullet:
> >
> > - How to autodetect infrastructure failures, regressions/new bugs and test
> >      bugs? How to handle continuous failures due to known bugs in both
> >      tests and kernel? What's your solution? Can people always trust the
> >      results they
> >      receive?
> >
> > I'd be happy to discuss this, if it's desired.
> >
> > Otherwise, I've recently been working on standards for "test definition",
> > which defines the data and meta-data associated with a test.   I could talk
> > about where I'm at with that, if people are interested.
> >
>
> Sounds great! I added both your points to the agenda as I do think they have
> a place here. The list of items is growing so I hope we can still fit
> everything into the two days we planned :)
>
>
> See you there!
> Veronika
>
> > Let me know what you think.
> >  -- Tim

Re: CKI hackfest @Plumbers invite

[Veronika Kabatova]

----- Original Message -----
> From: "Dmitry Vyukov" <dvyukov@google.com>
> To: "Veronika Kabatova" <vkabatov@redhat.com>
> Cc: "Tim Bird" <Tim.Bird@sony.com>, automated-testing@yoctoproject.org, info@kernelci.org, "syzkaller"
> <syzkaller@googlegroups.com>, lkp@lists.01.org, "stable" <stable@vger.kernel.org>, "Laura Abbott"
> <labbott@redhat.com>, "Eliska Slobodova" <eslobodo@redhat.com>, cki-project@redhat.com, "David Rientjes"
> <rientjes@google.com>
> Sent: Monday, May 27, 2019 4:39:16 PM
> Subject: Re: CKI hackfest @Plumbers invite
> 
> On Mon, May 27, 2019 at 1:52 PM Veronika Kabatova <vkabatov@redhat.com>
> wrote:
> > ----- Original Message -----
> > > From: "Tim Bird" <Tim.Bird@sony.com>
> > > To: vkabatov@redhat.com, automated-testing@yoctoproject.org,
> > > info@kernelci.org, khilamn@baylibre.org,
> > > syzkaller@googlegroups.com, lkp@lists.01.org, stable@vger.kernel.org,
> > > labbott@redhat.com
> > > Cc: eslobodo@redhat.com, cki-project@redhat.com
> > > Sent: Friday, May 24, 2019 10:17:04 PM
> > > Subject: RE: CKI hackfest @Plumbers invite
> > >
> > >
> > >
> > > > -----Original Message-----
> > > > From: Veronika Kabatova
> > > >
> > > > Hi,
> > > >
> > > > as some of you have heard, CKI Project is planning hackfest CI meetings
> > > > after
> > > > Plumbers conference this year (Sept. 12-13). We would like to invite
> > > > everyone
> > > > who has interest in CI for kernel to come and join us.
> > > >
> > > > The early agenda with summary is at the end of the email. If you think
> > > > there's
> > > > something important missing let us know! Also let us know in case you'd
> > > > want to
> > > > lead any of the sessions, we'd be happy to delegate out some work :)
> > > >
> > > >
> > > > Please send us an email as soon as you decide to come and feel free to
> > > > invite
> > > > other people who should be present. We are not planning to cap the
> > > > attendance
> > > > right now but need to solve the logistics based on the interest. The
> > > > event
> > > > is
> > > > free to attend, no additional registration except letting us know is
> > > > needed.
> > > >
> > > > Feel free to contact us if you have any questions,
> > >
> > > I plan to come to the event.
> > >
> > > > -----------------------------------------------------------
> > > > Here is an early agenda we put together:
> > > > - Introductions
> > > > - Common place for upstream results, result publishing in general
> > > >   - The discussion on the mailing list is going strong so we might be
> > > >   able
> > > >   to
> > > >     substitute this session for a different one in case everything is
> > > >     solved by
> > > >     September.
> > > > - Test result interpretation and bug detection
> > > >   - How to autodetect infrastructure failures, regressions/new bugs and
> > > >   test
> > > >     bugs? How to handle continuous failures due to known bugs in both
> > > >     tests
> > > > and
> > > >     kernel? What's your solution? Can people always trust the results
> > > >     they
> > > >     receive?
> > > > - Getting results to developers/maintainers
> > > >   - Aimed at kernel developers and maintainers, share your feedback and
> > > >     expectations.
> > > >   - How much data should be sent in the initial communication vs. a
> > > >   click
> > > >   away
> > > >     in a dashboard? Do you want incremental emails with new results as
> > > >     they
> > > > come
> > > >     in?
> > > >   - What about adding checks to tested patches in Patchwork when patch
> > > > series
> > > >     are being tested?
> > > >   - Providing enough data/script to reproduce the failure. What if
> > > >   special
> > > >   HW
> > > >     is needed?
> > > > - Onboarding new kernel trees to test
> > > >   - Aimed at kernel developers and maintainers.
> > > >   - Which trees are most prone to bring in new problems? Which are the
> > > >   most
> > > >     critical ones? Do you want them to be tested? Which tests do you
> > > >     feel
> > > >     are
> > > >     most beneficial for specific trees or in general?
> > > > - Security when testing untrusted patches
> > > >   - How do we merge, compile, and test patches that have untrusted code
> > > >   in
> > > > them
> > > >     and have not yet been reviewed? How do we avoid abuse of systems,
> > > >     information theft, or other damage?
> > > >   - Check out the original patch that sparked the discussion at
> > > >     https://patchwork.ozlabs.org/patch/862123/
> > > > - Avoiding effort duplication
> > > >   - Food for thought by GregKH
> > > >   - X different CI systems running ${TEST} on latest stable kernel on
> > > >   x86_64
> > > >     might look useless on the first look but is it? AMD/Intel CPUs,
> > > >     different
> > > >     network cards, different graphic drivers, compilers, kernel
> > > >     configuration...
> > > >     How do we distribute the workload to avoid doing the same thing all
> > > >     over
> > > >     again while still running in enough different environments to get
> > > >     the
> > > >     most
> > > >     coverage?
> 
> 
> Hi Veronika,
> 

Hi!

> All are great questions that we need to resolve!
> 
> I am also very much concerned about duplication in 2 other dimensions
> with the current approach to kernel testing:
> 
> 1. If X different CI systems running ${TEST}, developers receive X
> reports about the same breakage from X different directions, in
> different formats, of different quality, at slightly different times
> and somebody needs to act on all of them in some way. The more CI
> systems we have, the more run meaningful number of tests and do
> automatic reporting, the more and more duplicates developers get.
> 

This is a very good point that we had to deal with internally -- people aren't
happy getting multiple reports about a same issue.

And that's what part of my point of finding balance was about too. Maybe the
test only breaks on specific HW so getting X reports from runs on different HW
where X-1 pass and one fails helps to narrow the issue down. Same if it fails
everywhere -- it has to be some central code that caused the failures. If all
the reports are just different version of "tested on x86 VM and it failed"
then I totally agree it's not much help.

The formats and quality of the reports is definitely something worth talking
about too, and potentially unify as much as we can (Tim started some work on
this already).


> 2. Effort duplication between implementation of different CI systems.
> Doing a proper and really good CI is very hard. This includes all
> questions that you mentioned here, and fine tuning of all of that,
> refining reporting, bisection, onboarding of different test suites,
> onboarding of different dynamic/static analysis tools and much more.
> Last but not least is duplication of processes related to these CIs.
> Speaking of my experience with syzbot, this is extremely hard and
> takes years. And we really can't expose a developer to 27 different
> systems and slightly different processes (this would mean they follow
> 0 of these processes).
> This is further complicated by the fact that kernel tests are
> fragmented, so it's not possible to, say, simply run all kernel tests.
> And kernel processes are fragmented, e.g. you mentioned patchwork, but
> not all subsystems use patchwork, so it's not possible to simply
> extend a CI to all subsystems. And some aspects of the current kernel
> development process notoriously complicate automation of things that
> really should be trivial. For example, if you have
> github/gitlab/gerrit, you can hook into arrival of each new change and
> pull exact code state. Done. For kernel some changes appear on
> patchwork, some don't, some are duplicated on multiple patchworks,
> some duplicated in a weird way on the same patchwork, some non-patches
> appear on patchwork because it's confused, and last but not least you
> can't really apply any of them because none of them include base
> tree/commit info. Handling just this requires lots of effort, guessing
> on coffee grounds and heuristics that need to be refined over time.
> The total complexity of doing it just once, with all resources
> combined and dev process re-shaped to cooperate is close to off-scale.
> 

Totally agreed, we've have some fun with some of the points you
mentioned as well and I'm sure others too. What we do in CKI is to unify
as much as we can across different trees we're testing. For example the
trees living in GitHub/GitLab/Gerrit -- not all trees do this but all
trees use the git backend so we use polling and commit checking (ugly,
I admit) as it works everywhere and we don't need to maintain multiple
different APIs for the same thing.


I don't want to derail the invite discussion by replying to everything
here but what you brought up are great points that every CI system around
kernel has to deal with.

> Do you see these points as a problem too? Or am I exaggerating matters?
> 
> 

Oh absolutely! While it's what I call "the nasty implementation details"
they are all valid problems that should be talked about more. Whether it's
with developers (eg. the points about putting kernel tests into a single
place and unifying the development process, if possible) or with other CI
teams.

I'd be happy to talk with you about all of this at the hackfest or
privately/off-thread. The automated-testing list (in cc) is a great place
too as a diverse set of people involved with CI gathers there.


Let me know what works for you,
Veronika

> 
> 
> 
> 
> > > > - Common hardware pools
> > > >   - Is this something people are interested in? Would be helpful
> > > >   especially
> > > >   for
> > > >     HW that's hard to access, eg. ppc64le or s390x systems. Companies
> > > >     could
> > > > also
> > > >     sing up to share their HW for testing to ensure kernel works with
> > > >     their
> > > >     products.
> > >
> > > I have strong opinions on some of these, but maybe only useful experience
> > > in a few areas.  Fuego has 2 separate notions, which we call "skiplists"
> > > and "pass criteria", which have to do with this bullet:
> > >
> > > - How to autodetect infrastructure failures, regressions/new bugs and
> > > test
> > >      bugs? How to handle continuous failures due to known bugs in both
> > >      tests and kernel? What's your solution? Can people always trust the
> > >      results they
> > >      receive?
> > >
> > > I'd be happy to discuss this, if it's desired.
> > >
> > > Otherwise, I've recently been working on standards for "test definition",
> > > which defines the data and meta-data associated with a test.   I could
> > > talk
> > > about where I'm at with that, if people are interested.
> > >
> >
> > Sounds great! I added both your points to the agenda as I do think they
> > have
> > a place here. The list of items is growing so I hope we can still fit
> > everything into the two days we planned :)
> >
> >
> > See you there!
> > Veronika
> >
> > > Let me know what you think.
> > >  -- Tim
> 

Re: CKI hackfest @Plumbers invite

[Dan Rue]

On Tue, May 21, 2019 at 10:54:12AM -0400, Veronika Kabatova wrote:
> Hi,
> 
> as some of you have heard, CKI Project is planning hackfest CI meetings after
> Plumbers conference this year (Sept. 12-13). We would like to invite everyone
> who has interest in CI for kernel to come and join us.
> 
> The early agenda with summary is at the end of the email. If you think there's
> something important missing let us know! Also let us know in case you'd want to
> lead any of the sessions, we'd be happy to delegate out some work :)
> 
> 
> Please send us an email as soon as you decide to come and feel free to invite
> other people who should be present. We are not planning to cap the attendance
> right now but need to solve the logistics based on the interest. The event is
> free to attend, no additional registration except letting us know is needed.
> 
> Feel free to contact us if you have any questions,
> Veronika
> CKI Project

Hi Veronika! Thanks for organizing this. I plan to attend, and I'm happy
to help out.

With regard to the agenda, I've been following the '[Ksummit-discuss]
[MAINTAINERS SUMMIT] Squashing bugs!'[1] thread with interest, as it
relates especially to 'Getting results to developers/maintainers'. This,
along with result aggregation, are important areas to focus.

Dan

[1] https://lists.linuxfoundation.org/pipermail/ksummit-discuss/2019-May/006389.html

> 
> 
> -----------------------------------------------------------
> Here is an early agenda we put together:
> - Introductions
> - Common place for upstream results, result publishing in general
>   - The discussion on the mailing list is going strong so we might be able to
>     substitute this session for a different one in case everything is solved by
>     September.
> - Test result interpretation and bug detection
>   - How to autodetect infrastructure failures, regressions/new bugs and test
>     bugs? How to handle continuous failures due to known bugs in both tests and
>     kernel? What's your solution? Can people always trust the results they
>     receive?
> - Getting results to developers/maintainers
>   - Aimed at kernel developers and maintainers, share your feedback and
>     expectations.
>   - How much data should be sent in the initial communication vs. a click away
>     in a dashboard? Do you want incremental emails with new results as they come
>     in?
>   - What about adding checks to tested patches in Patchwork when patch series
>     are being tested?
>   - Providing enough data/script to reproduce the failure. What if special HW
>     is needed?
> - Onboarding new kernel trees to test
>   - Aimed at kernel developers and maintainers.
>   - Which trees are most prone to bring in new problems? Which are the most
>     critical ones? Do you want them to be tested? Which tests do you feel are
>     most beneficial for specific trees or in general?
> - Security when testing untrusted patches
>   - How do we merge, compile, and test patches that have untrusted code in them
>     and have not yet been reviewed? How do we avoid abuse of systems,
>     information theft, or other damage?
>   - Check out the original patch that sparked the discussion at
>     https://patchwork.ozlabs.org/patch/862123/
> - Avoiding effort duplication
>   - Food for thought by GregKH
>   - X different CI systems running ${TEST} on latest stable kernel on x86_64
>     might look useless on the first look but is it? AMD/Intel CPUs, different
>     network cards, different graphic drivers, compilers, kernel configuration...
>     How do we distribute the workload to avoid doing the same thing all over
>     again while still running in enough different environments to get the most
>     coverage?
> - Common hardware pools
>   - Is this something people are interested in? Would be helpful especially for
>     HW that's hard to access, eg. ppc64le or s390x systems. Companies could also
>     sing up to share their HW for testing to ensure kernel works with their
>     products.

-- 
Linaro - Kernel Validation

Re: CKI hackfest @Plumbers invite

[Shuah Khan]

Hi Veronika,

On Wed, Jun 5, 2019 at 2:47 PM Dan Rue <dan.rue@linaro.org> wrote:
>
> On Tue, May 21, 2019 at 10:54:12AM -0400, Veronika Kabatova wrote:
> > Hi,
> >
> > as some of you have heard, CKI Project is planning hackfest CI meetings after
> > Plumbers conference this year (Sept. 12-13). We would like to invite everyone
> > who has interest in CI for kernel to come and join us.
> >
> > The early agenda with summary is at the end of the email. If you think there's
> > something important missing let us know! Also let us know in case you'd want to
> > lead any of the sessions, we'd be happy to delegate out some work :)
> >
> >
> > Please send us an email as soon as you decide to come and feel free to invite
> > other people who should be present. We are not planning to cap the attendance
> > right now but need to solve the logistics based on the interest. The event is
> > free to attend, no additional registration except letting us know is needed.
> >

I am going be there and plan to attend.

> > Feel free to contact us if you have any questions,
> > Veronika
> > CKI Project
>
> Hi Veronika! Thanks for organizing this. I plan to attend, and I'm happy
> to help out.
>
> With regard to the agenda, I've been following the '[Ksummit-discuss]
> [MAINTAINERS SUMMIT] Squashing bugs!'[1] thread with interest, as it
> relates especially to 'Getting results to developers/maintainers'. This,
> along with result aggregation, are important areas to focus.
>
>
> [1] https://lists.linuxfoundation.org/pipermail/ksummit-discuss/2019-May/006389.html
>

Good to know there is an overlap and it makes sense for me to attend. :)

thanks,
-- Shuah

Re: CKI hackfest @Plumbers invite

[Tomeu Vizoso]

On 5/21/19 4:54 PM, Veronika Kabatova wrote:
> Hi,
> 
> as some of you have heard, CKI Project is planning hackfest CI meetings after
> Plumbers conference this year (Sept. 12-13). We would like to invite everyone
> who has interest in CI for kernel to come and join us.
> 
> The early agenda with summary is at the end of the email. If you think there's
> something important missing let us know! Also let us know in case you'd want to
> lead any of the sessions, we'd be happy to delegate out some work :)
> 
> 
> Please send us an email as soon as you decide to come and feel free to invite
> other people who should be present. We are not planning to cap the attendance
> right now but need to solve the logistics based on the interest. The event is
> free to attend, no additional registration except letting us know is needed.

Hi Veronika, I would like to be there.

Cheers,

Tomeu

> Feel free to contact us if you have any questions,
> Veronika
> CKI Project
> 
> 
> -----------------------------------------------------------
> Here is an early agenda we put together:
> - Introductions
> - Common place for upstream results, result publishing in general
>    - The discussion on the mailing list is going strong so we might be able to
>      substitute this session for a different one in case everything is solved by
>      September.
> - Test result interpretation and bug detection
>    - How to autodetect infrastructure failures, regressions/new bugs and test
>      bugs? How to handle continuous failures due to known bugs in both tests and
>      kernel? What's your solution? Can people always trust the results they
>      receive?
> - Getting results to developers/maintainers
>    - Aimed at kernel developers and maintainers, share your feedback and
>      expectations.
>    - How much data should be sent in the initial communication vs. a click away
>      in a dashboard? Do you want incremental emails with new results as they come
>      in?
>    - What about adding checks to tested patches in Patchwork when patch series
>      are being tested?
>    - Providing enough data/script to reproduce the failure. What if special HW
>      is needed?
> - Onboarding new kernel trees to test
>    - Aimed at kernel developers and maintainers.
>    - Which trees are most prone to bring in new problems? Which are the most
>      critical ones? Do you want them to be tested? Which tests do you feel are
>      most beneficial for specific trees or in general?
> - Security when testing untrusted patches
>    - How do we merge, compile, and test patches that have untrusted code in them
>      and have not yet been reviewed? How do we avoid abuse of systems,
>      information theft, or other damage?
>    - Check out the original patch that sparked the discussion at
>      https://patchwork.ozlabs.org/patch/862123/
> - Avoiding effort duplication
>    - Food for thought by GregKH
>    - X different CI systems running ${TEST} on latest stable kernel on x86_64
>      might look useless on the first look but is it? AMD/Intel CPUs, different
>      network cards, different graphic drivers, compilers, kernel configuration...
>      How do we distribute the workload to avoid doing the same thing all over
>      again while still running in enough different environments to get the most
>      coverage?
> - Common hardware pools
>    - Is this something people are interested in? Would be helpful especially for
>      HW that's hard to access, eg. ppc64le or s390x systems. Companies could also
>      sing up to share their HW for testing to ensure kernel works with their
>      products.
> 
> -=-=-=-=-=-=-=-=-=-=-=-
> Groups.io Links: You receive all messages sent to this group.
> 
> View/Reply Online (#404): https://groups.io/g/kernelci/message/404
> Mute This Topic: https://groups.io/mt/31697554/925689
> Group Owner: kernelci+owner@groups.io
> Unsubscribe: https://groups.io/g/kernelci/unsub  [tomeu.vizoso@collabora.com]
> -=-=-=-=-=-=-=-=-=-=-=-
> 

Re: CKI hackfest @Plumbers invite

[Veronika Kabatova]

Added you both to the list :)

----- Original Message -----
> From: "Shuah Khan" <shuahkhan@gmail.com>
> To: "Veronika Kabatova" <vkabatov@redhat.com>, automated-testing@yoctoproject.org, info@kernelci.org, "Tim Bird"
> <Tim.Bird@sony.com>, khilamn@baylibre.org, syzkaller@googlegroups.com, lkp@lists.01.org, "stable"
> <stable@vger.kernel.org>, "Laura Abbott" <labbott@redhat.com>, "Eliska Slobodova" <eslobodo@redhat.com>, "CKI
> Project" <cki-project@redhat.com>
> Sent: Thursday, June 6, 2019 12:00:13 AM
> Subject: Re: CKI hackfest @Plumbers invite
> 
> Hi Veronika,
> 
> On Wed, Jun 5, 2019 at 2:47 PM Dan Rue <dan.rue@linaro.org> wrote:
> >
> > On Tue, May 21, 2019 at 10:54:12AM -0400, Veronika Kabatova wrote:
> > > Hi,
> > >
> > > as some of you have heard, CKI Project is planning hackfest CI meetings
> > > after
> > > Plumbers conference this year (Sept. 12-13). We would like to invite
> > > everyone
> > > who has interest in CI for kernel to come and join us.
> > >
> > > The early agenda with summary is at the end of the email. If you think
> > > there's
> > > something important missing let us know! Also let us know in case you'd
> > > want to
> > > lead any of the sessions, we'd be happy to delegate out some work :)
> > >
> > >
> > > Please send us an email as soon as you decide to come and feel free to
> > > invite
> > > other people who should be present. We are not planning to cap the
> > > attendance
> > > right now but need to solve the logistics based on the interest. The
> > > event is
> > > free to attend, no additional registration except letting us know is
> > > needed.
> > >
> 
> I am going be there and plan to attend.
> 
> > > Feel free to contact us if you have any questions,
> > > Veronika
> > > CKI Project
> >
> > Hi Veronika! Thanks for organizing this. I plan to attend, and I'm happy
> > to help out.
> >
> > With regard to the agenda, I've been following the '[Ksummit-discuss]
> > [MAINTAINERS SUMMIT] Squashing bugs!'[1] thread with interest, as it
> > relates especially to 'Getting results to developers/maintainers'. This,
> > along with result aggregation, are important areas to focus.
> >
> >
> > [1]
> > https://lists.linuxfoundation.org/pipermail/ksummit-discuss/2019-May/006389.html
> >
> 
> Good to know there is an overlap and it makes sense for me to attend. :)
> 

I've been pointed to this thread just yesterday (thanks Laura!) and I agree
you bring up interesting topics in there. In fact, the "Getting results out"
topic Dan mentioned has the reproducibility of the failures as one of the
agenda items.

There definitely *is* an overlap in some of the topics and we'd be excited
to have you both there to talk more!

Veronika

> thanks,
> -- Shuah
> 

Re: [Automated-testing] CKI hackfest @Plumbers invite

[Michal Simek]

[-- Attachment #1.1: Type: text/plain, Size: 991 bytes --]

On 21. 05. 19 16:54, Veronika Kabatova wrote:
> Hi,
> 
> as some of you have heard, CKI Project is planning hackfest CI meetings after
> Plumbers conference this year (Sept. 12-13). We would like to invite everyone
> who has interest in CI for kernel to come and join us.
> 
> The early agenda with summary is at the end of the email. If you think there's
> something important missing let us know! Also let us know in case you'd want to
> lead any of the sessions, we'd be happy to delegate out some work :)
> 
> 
> Please send us an email as soon as you decide to come and feel free to invite
> other people who should be present. We are not planning to cap the attendance
> right now but need to solve the logistics based on the interest. The event is
> free to attend, no additional registration except letting us know is needed.
> 

I would like to join these meetings too. Please put me on the list. I
was also on the first meeting after ELCE.

Thanks,
Michal


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: CKI hackfest @Plumbers invite

[Mark Brown]

[-- Attachment #1: Type: text/plain, Size: 455 bytes --]

On Tue, May 21, 2019 at 10:54:12AM -0400, Veronika Kabatova wrote:

> Please send us an email as soon as you decide to come and feel free to invite
> other people who should be present. We are not planning to cap the attendance
> right now but need to solve the logistics based on the interest. The event is
> free to attend, no additional registration except letting us know is needed.

Still waiting for final confirmation but I expect to be there too.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: CKI hackfest @Plumbers invite

[Dmitry Vyukov]

On Thu, Jun 6, 2019 at 12:00 AM Shuah Khan <shuahkhan@gmail.com> wrote:
>
> Hi Veronika,
>
> On Wed, Jun 5, 2019 at 2:47 PM Dan Rue <dan.rue@linaro.org> wrote:
> >
> > On Tue, May 21, 2019 at 10:54:12AM -0400, Veronika Kabatova wrote:
> > > Hi,
> > >
> > > as some of you have heard, CKI Project is planning hackfest CI meetings after
> > > Plumbers conference this year (Sept. 12-13). We would like to invite everyone
> > > who has interest in CI for kernel to come and join us.
> > >
> > > The early agenda with summary is at the end of the email. If you think there's
> > > something important missing let us know! Also let us know in case you'd want to
> > > lead any of the sessions, we'd be happy to delegate out some work :)
> > >
> > >
> > > Please send us an email as soon as you decide to come and feel free to invite
> > > other people who should be present. We are not planning to cap the attendance
> > > right now but need to solve the logistics based on the interest. The event is
> > > free to attend, no additional registration except letting us know is needed.
> > >
>
> I am going be there and plan to attend.
>
> > > Feel free to contact us if you have any questions,
> > > Veronika
> > > CKI Project
> >
> > Hi Veronika! Thanks for organizing this. I plan to attend, and I'm happy
> > to help out.
> >
> > With regard to the agenda, I've been following the '[Ksummit-discuss]
> > [MAINTAINERS SUMMIT] Squashing bugs!'[1] thread with interest, as it
> > relates especially to 'Getting results to developers/maintainers'. This,
> > along with result aggregation, are important areas to focus.
> >
> >
> > [1] https://lists.linuxfoundation.org/pipermail/ksummit-discuss/2019-May/006389.html
> >
>
> Good to know there is an overlap and it makes sense for me to attend. :)

Hi Shuah,

Oh, and I did not even know about
https://lists.linuxfoundation.org/pipermail/ksummit-discuss/2019-May/006389.html
How can I be kept in the loop/provide inputs/receive
feedback/discussion summary?

Thanks

Re: CKI hackfest @Plumbers invite

[Veronika Kabatova]

----- Original Message -----
> From: "Guillaume Tucker" <guillaume.tucker@gmail.com>
> To: kernelci@groups.io, vkabatov@redhat.com
> Cc: automated-testing@yoctoproject.org, info@kernelci.org, "Tim Bird" <Tim.Bird@sony.com>, khilamn@baylibre.org,
> syzkaller@googlegroups.com, lkp@lists.01.org, stable@vger.kernel.org, "Laura Abbott" <labbott@redhat.com>, "Eliska
> Slobodova" <eslobodo@redhat.com>, "CKI Project" <cki-project@redhat.com>
> Sent: Thursday, June 20, 2019 5:42:11 PM
> Subject: Re: CKI hackfest @Plumbers invite
> 
> Hi Veronika,
> 
> On Tue, May 21, 2019 at 3:55 PM Veronika Kabatova <vkabatov@redhat.com>
> wrote:
> 
> > Hi,
> >
> > as some of you have heard, CKI Project is planning hackfest CI meetings
> > after
> > Plumbers conference this year (Sept. 12-13). We would like to invite
> > everyone
> > who has interest in CI for kernel to come and join us.
> >
> > The early agenda with summary is at the end of the email. If you think
> > there's
> > something important missing let us know! Also let us know in case you'd
> > want to
> > lead any of the sessions, we'd be happy to delegate out some work :)
> >
> >
> > Please send us an email as soon as you decide to come and feel free to
> > invite
> > other people who should be present. We are not planning to cap the
> > attendance
> > right now but need to solve the logistics based on the interest. The event
> > is
> > free to attend, no additional registration except letting us know is
> > needed.
> >
> 
> Please do count me in as well!
> 

\o/

> One topic I would like to add to the agenda is:
> 
> - Open testing philosophy
>   - Connecting components from different origins: builders, test
>     labs, databases, dashboards...
>   - Interoperability: documented remote APIs to let components
>     talk to each other
>   - kernelci.org already does this with distributed builds and
>     test labs, it would be good to apply the same principles to
>     to other existing systems doing upstream kernel testing for
>     everyone's benefit
>   - Optimal utilisation of available resources
>   - Enable more high-level features by joining
>     forces (bisections, cross-referencing of results, bug
>     tracking...)
> 
> This does have some commonality with "Common hardware pools"
> and "Avoiding effort duplication" but I think it makes sense to
> keep it together as a general approach.
> 

I agree that this topic is important (and I believe some other CKI people
made that clear as well) so I added it to the agenda topics. The list of
those is getting long so we'd definitely need to curate it properly soon
but I'll make sure this stays there.


Thanks for the interest!
Veronika

> Thanks,
> Guillaume
> 
> Feel free to contact us if you have any questions,
> > Veronika
> > CKI Project
> >
> >
> > -----------------------------------------------------------
> > Here is an early agenda we put together:
> > - Introductions
> > - Common place for upstream results, result publishing in general
> >   - The discussion on the mailing list is going strong so we might be able
> > to
> >     substitute this session for a different one in case everything is
> > solved by
> >     September.
> > - Test result interpretation and bug detection
> >   - How to autodetect infrastructure failures, regressions/new bugs and
> > test
> >     bugs? How to handle continuous failures due to known bugs in both
> > tests and
> >     kernel? What's your solution? Can people always trust the results they
> >     receive?
> > - Getting results to developers/maintainers
> >   - Aimed at kernel developers and maintainers, share your feedback and
> >     expectations.
> >   - How much data should be sent in the initial communication vs. a click
> > away
> >     in a dashboard? Do you want incremental emails with new results as
> > they come
> >     in?
> >   - What about adding checks to tested patches in Patchwork when patch
> > series
> >     are being tested?
> >   - Providing enough data/script to reproduce the failure. What if special
> > HW
> >     is needed?
> > - Onboarding new kernel trees to test
> >   - Aimed at kernel developers and maintainers.
> >   - Which trees are most prone to bring in new problems? Which are the most
> >     critical ones? Do you want them to be tested? Which tests do you feel
> > are
> >     most beneficial for specific trees or in general?
> > - Security when testing untrusted patches
> >   - How do we merge, compile, and test patches that have untrusted code in
> > them
> >     and have not yet been reviewed? How do we avoid abuse of systems,
> >     information theft, or other damage?
> >   - Check out the original patch that sparked the discussion at
> >     https://patchwork.ozlabs.org/patch/862123/
> > - Avoiding effort duplication
> >   - Food for thought by GregKH
> >   - X different CI systems running ${TEST} on latest stable kernel on
> > x86_64
> >     might look useless on the first look but is it? AMD/Intel CPUs,
> > different
> >     network cards, different graphic drivers, compilers, kernel
> > configuration...
> >     How do we distribute the workload to avoid doing the same thing all
> > over
> >     again while still running in enough different environments to get the
> > most
> >     coverage?
> > - Common hardware pools
> >   - Is this something people are interested in? Would be helpful
> > especially for
> >     HW that's hard to access, eg. ppc64le or s390x systems. Companies
> > could also
> >     sing up to share their HW for testing to ensure kernel works with their
> >     products.
> >
> > -=-=-=-=-=-=-=-=-=-=-=-
> > Groups.io Links: You receive all messages sent to this group.
> >
> > View/Reply Online (#404): https://groups.io/g/kernelci/message/404
> > Mute This Topic: https://groups.io/mt/31697554/924702
> > Group Owner: kernelci+owner@groups.io
> > Unsubscribe: https://groups.io/g/kernelci/unsub  [
> > guillaume.tucker@gmail.com]
> > -=-=-=-=-=-=-=-=-=-=-=-
> >
> >
> 

Re: CKI hackfest @Plumbers invite

[Shuah Khan]

On Fri, Jun 7, 2019 at 10:28 AM Dmitry Vyukov <dvyukov@google.com> wrote:
>
> On Thu, Jun 6, 2019 at 12:00 AM Shuah Khan <shuahkhan@gmail.com> wrote:
> >
> > Hi Veronika,
> >
> > On Wed, Jun 5, 2019 at 2:47 PM Dan Rue <dan.rue@linaro.org> wrote:
> > >
> > > On Tue, May 21, 2019 at 10:54:12AM -0400, Veronika Kabatova wrote:
> > > > Hi,
> > > >
> > > > as some of you have heard, CKI Project is planning hackfest CI meetings after
> > > > Plumbers conference this year (Sept. 12-13). We would like to invite everyone
> > > > who has interest in CI for kernel to come and join us.
> > > >
> > > > The early agenda with summary is at the end of the email. If you think there's
> > > > something important missing let us know! Also let us know in case you'd want to
> > > > lead any of the sessions, we'd be happy to delegate out some work :)
> > > >
> > > >
> > > > Please send us an email as soon as you decide to come and feel free to invite
> > > > other people who should be present. We are not planning to cap the attendance
> > > > right now but need to solve the logistics based on the interest. The event is
> > > > free to attend, no additional registration except letting us know is needed.
> > > >
> >
> > I am going be there and plan to attend.
> >
> > > > Feel free to contact us if you have any questions,
> > > > Veronika
> > > > CKI Project
> > >
> > > Hi Veronika! Thanks for organizing this. I plan to attend, and I'm happy
> > > to help out.
> > >
> > > With regard to the agenda, I've been following the '[Ksummit-discuss]
> > > [MAINTAINERS SUMMIT] Squashing bugs!'[1] thread with interest, as it
> > > relates especially to 'Getting results to developers/maintainers'. This,
> > > along with result aggregation, are important areas to focus.
> > >
> > >
> > > [1] https://lists.linuxfoundation.org/pipermail/ksummit-discuss/2019-May/006389.html
> > >
> >
> > Good to know there is an overlap and it makes sense for me to attend. :)
>
> Hi Shuah,
>
> Oh, and I did not even know about
> https://lists.linuxfoundation.org/pipermail/ksummit-discuss/2019-May/006389.html
> How can I be kept in the loop/provide inputs/receive
> feedback/discussion summary?
>

Sorry didn't see this until now. You can subscribe ksummit-discuss to
follow the thread.

thanks,
-- Shuah

RE: CKI hackfest @Plumbers invite

[Tim.Bird]

> -----Original Message-----
> From: Veronika Kabatova 
> 
> ----- Original Message -----
> > From: "Guillaume Tucker" 
> >
> > Hi Veronika,
> >
> > On Tue, May 21, 2019 at 3:55 PM Veronika Kabatova
> <vkabatov@redhat.com>
> > wrote:
> >
> 
> I agree that this topic is important (and I believe some other CKI people
> made that clear as well) so I added it to the agenda topics. The list of
> those is getting long so we'd definitely need to curate it properly soon
> but I'll make sure this stays there.

I agree that Guillaume's topic would be good to discuss.

Is the draft agenda online anywhere?
 -- Tim

Re: CKI hackfest @Plumbers invite

[Veronika Kabatova]

----- Original Message -----
> From: "Tim Bird" <Tim.Bird@sony.com>
> To: vkabatov@redhat.com, "guillaume tucker" <guillaume.tucker@gmail.com>
> Cc: kernelci@groups.io, automated-testing@yoctoproject.org, info@kernelci.org, syzkaller@googlegroups.com,
> lkp@lists.01.org, stable@vger.kernel.org, labbott@redhat.com, eslobodo@redhat.com, cki-project@redhat.com
> Sent: Monday, June 24, 2019 8:55:12 PM
> Subject: RE: CKI hackfest @Plumbers invite
> 
> > -----Original Message-----
> > From: Veronika Kabatova
> > 
> > ----- Original Message -----
> > > From: "Guillaume Tucker"
> > >
> > > Hi Veronika,
> > >
> > > On Tue, May 21, 2019 at 3:55 PM Veronika Kabatova
> > <vkabatov@redhat.com>
> > > wrote:
> > >
> > 
> > I agree that this topic is important (and I believe some other CKI people
> > made that clear as well) so I added it to the agenda topics. The list of
> > those is getting long so we'd definitely need to curate it properly soon
> > but I'll make sure this stays there.
> 
> I agree that Guillaume's topic would be good to discuss.
> 
> Is the draft agenda online anywhere?

I sent the draft with the original invite so you can check that out. Since
then, we added a few new topics to the list:

- Test definition standardization (from you)
- Onboarding new tests to run (versioning, unification of test locations
  etc.)
- Open testing philosophy (from Guillaume)

We'll try our best to cover all mentioned topics during the hackfest but
haven't started working on the schedule yet. We'll send it out before the
hackfest and likely will publish it on cki-project.org as well.


Veronika


>  -- Tim
> 
>