Bisection email recipients

Bisection email recipients

[Guillaume Tucker]

As you know, we're now running automated bisections for all boot
regressions found in mainline and stable trees.  The email reports
are currently only sent to a fixed list of recipients who manually
check them and report the result as needed.  As it turns out, there
haven't been any real false positives[1] thanks to the checks
(verify breaking commit, then revert in-place).  Some lab issues
can mislead bisections but the odds for all the checks to pass with
a false positive are really low, and I've got a task to better
handle infrastructure errors in the bisection job to further reduce
that.

So unless anyone disagrees, I think it's now mature enough to start
sending the report automatically to some relevant recipients.  I
would first add a paragraph to give some context regarding the
automated bisection so when people receive one out of the blue they
get a clue as to where this is coming from.  I'll make a PR to
update the template for this.

Here's a list of potential recipients that could be automatically
added, using the "breaking" commit found and other metadata:

* author of the commit
* committer
* Signed-off-by
* Acked-by
* Reported-by
* mailing list associated with the tree (like for test reports)
* maintainers of the parts of the code touched (using
  scripts/get_maintainer.pl and diff stat)
* kernel-build-reports@lists.linaro.org
* as well as a fixed list of recipients configured in Jenkins like
  we already do now

It would also be good to send the report as a reply to either the
boot report showing the regression, or the original patch on a
mailing list if that can be found in the archives.  I'll see if it
can be done easily to directly do this as well as enabling more
recipients.

I also think it would be worth sending an email to the stable
mailing list, and probably also lkml to explain what we're doing
and warn people that we're turning this on.

Please let me know your thoughts; I'm planning to start working on
it this week and have a first version to try out and review for
next week.

Best wishes,
Guillaume


[1] except for one single case in lab-baylibre :)
http://lava.baylibre.com:10080/scheduler/alljobs?length=25&search=lava-bisect-9339#table

Re: [kernelci] Bisection email recipients

[Tomeu Vizoso]

On 10/30/18 6:08 PM, Guillaume Tucker wrote:
> As you know, we're now running automated bisections for all boot
> regressions found in mainline and stable trees.  The email reports
> are currently only sent to a fixed list of recipients who manually
> check them and report the result as needed.  As it turns out, there
> haven't been any real false positives[1] thanks to the checks
> (verify breaking commit, then revert in-place).  Some lab issues
> can mislead bisections but the odds for all the checks to pass with
> a false positive are really low, and I've got a task to better
> handle infrastructure errors in the bisection job to further reduce
> that.
> 
> So unless anyone disagrees, I think it's now mature enough to start
> sending the report automatically to some relevant recipients.  I
> would first add a paragraph to give some context regarding the
> automated bisection so when people receive one out of the blue they
> get a clue as to where this is coming from.  I'll make a PR to
> update the template for this.
> 
> Here's a list of potential recipients that could be automatically
> added, using the "breaking" commit found and other metadata:
> 
> * author of the commit
> * committer
> * Signed-off-by
> * Acked-by
> * Reported-by
> * mailing list associated with the tree (like for test reports)
> * maintainers of the parts of the code touched (using
>    scripts/get_maintainer.pl and diff stat)
> * kernel-build-reports@lists.linaro.org
> * as well as a fixed list of recipients configured in Jenkins like
>    we already do now
> 
> It would also be good to send the report as a reply to either the
> boot report showing the regression, or the original patch on a
> mailing list if that can be found in the archives.  I'll see if it
> can be done easily to directly do this as well as enabling more
> recipients.
> 
> I also think it would be worth sending an email to the stable
> mailing list, and probably also lkml to explain what we're doing
> and warn people that we're turning this on.
> 
> Please let me know your thoughts; I'm planning to start working on
> it this week and have a first version to try out and review for
> next week.

Hi Guillaume,

I think that regardless of how well it's working already, it could make 
sense to increase the list of recipients incrementally, starting with 
those most likely to appreciate the email, and growing towards those less 
likely. Maybe I'm being overly cautious, but from past experiences I 
don't think we want to risk too much antagonizing people.

Once we have a feeling that the recipients in a stage are happy with 
getting the emails, we can expand to other recipients in the next stage.

So, to illustrate the point, we could start with the author and 
kernel-build-reports@lists.linaro.org, then include ackers, reviewers, 
reporter and sobers, then mailing lists.

Hope that makes sense.

Cheers,

Tomeu

Re: [kernelci] Bisection email recipients

[Mark Brown]

[-- Attachment #1: Type: text/plain, Size: 2667 bytes --]

On Wed, Oct 31, 2018 at 08:37:13AM +0100, Tomeu Vizoso wrote:
> On 10/30/18 6:08 PM, Guillaume Tucker wrote:

> > * author of the commit
> > * committer
> > * Signed-off-by
> > * Acked-by
> > * Reported-by

I'd just include any e-mail address in the tags for the mail - tested-by
is definitely fair game (there's a decent chance they'll want to be
involved in testing further fixes), and it's a fairly common practice
for other kinds of reporting to just include everyone rather than
explicitly learning all the random tags different maintainers use.

> > * mailing list associated with the tree (like for test reports)
> > * maintainers of the parts of the code touched (using
> >    scripts/get_maintainer.pl and diff stat)

If you're using get_maintainer I'd make sure to use --nogit so that you
don't get false positives for people doing cleanup patches.  You can
probably just format the patch with git format-patch and then use
get_maintainer normally.

> > I also think it would be worth sending an email to the stable
> > mailing list, and probably also lkml to explain what we're doing
> > and warn people that we're turning this on.

Seems sensible.  It's a shame that we can't turn this on for -next yet,
but obviously the way that's built isn't super helpful.

> I think that regardless of how well it's working already, it could make
> sense to increase the list of recipients incrementally, starting with those
> most likely to appreciate the email, and growing towards those less likely.
> Maybe I'm being overly cautious, but from past experiences I don't think we
> want to risk too much antagonizing people.

> Once we have a feeling that the recipients in a stage are happy with getting
> the emails, we can expand to other recipients in the next stage.

> So, to illustrate the point, we could start with the author and
> kernel-build-reports@lists.linaro.org, then include ackers, reviewers,
> reporter and sobers, then mailing lists.

Not copying enough people can also cause issues as people will expect
other relevant people to be copied in and be annoyed about having to do
it themselves or possibly even just miss handling the report if they
expect someone has been CCed who hasn't - for example if we only send to
the patch author then maintainers might be annoyed if they later find
out there was a problem we didn't tell them about.  People will
definitely get annoyed about reports being off list because then the
discussion of the problem doesn't get archived for other people who
might benefit from it in future.

It is already common practice for people doing testing to copy fairly
widely so it shouldn't be that much of a surprise.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [kernelci] Bisection email recipients

[Guillaume Tucker]

[-- Attachment #1: Type: text/plain, Size: 3771 bytes --]

On Wed, Oct 31, 2018 at 10:26 AM Mark Brown <broonie@kernel.org> wrote:

> On Wed, Oct 31, 2018 at 08:37:13AM +0100, Tomeu Vizoso wrote:
> > On 10/30/18 6:08 PM, Guillaume Tucker wrote:
>
> > > * author of the commit
> > > * committer
> > > * Signed-off-by
> > > * Acked-by
> > > * Reported-by
>
> I'd just include any e-mail address in the tags for the mail - tested-by
>

Do you mean, to get the same as what "git send-email" would give?
I had a look and it doesn't seem easy to extract this
information, maybe you have a tip for that.  Otherwise, I've
added a simple Python function with a regex to the script I have
that triggers the bisection email (PR coming soon with details).


> is definitely fair game (there's a decent chance they'll want to be
> involved in testing further fixes), and it's a fairly common practice
> for other kinds of reporting to just include everyone rather than
> explicitly learning all the random tags different maintainers use.
>
> > > * mailing list associated with the tree (like for test reports)
> > > * maintainers of the parts of the code touched (using
> > >    scripts/get_maintainer.pl and diff stat)
>
> If you're using get_maintainer I'd make sure to use --nogit so that you
> don't get false positives for people doing cleanup patches.  You can
> probably just format the patch with git format-patch and then use
> get_maintainer normally.
>

Great tip, I'm using it with "--nogit" on the output of "git
show" and that seems to be doing a good job.

> > I also think it would be worth sending an email to the stable
> > > mailing list, and probably also lkml to explain what we're doing
> > > and warn people that we're turning this on.
>
> Seems sensible.  It's a shame that we can't turn this on for -next yet,
> but obviously the way that's built isn't super helpful.
>

Bisections on linux-next are coming soon, we just have to get the
basics all sorted out for mainline and stable first.

> I think that regardless of how well it's working already, it could make
> > sense to increase the list of recipients incrementally, starting with
> those
> > most likely to appreciate the email, and growing towards those less
> likely.
> > Maybe I'm being overly cautious, but from past experiences I don't think
> we
> > want to risk too much antagonizing people.
>
> > Once we have a feeling that the recipients in a stage are happy with
> getting
> > the emails, we can expand to other recipients in the next stage.
>
> > So, to illustrate the point, we could start with the author and
> > kernel-build-reports@lists.linaro.org, then include ackers, reviewers,
> > reporter and sobers, then mailing lists.
>
> Not copying enough people can also cause issues as people will expect
> other relevant people to be copied in and be annoyed about having to do
> it themselves or possibly even just miss handling the report if they
> expect someone has been CCed who hasn't - for example if we only send to
> the patch author then maintainers might be annoyed if they later find
> out there was a problem we didn't tell them about.  People will
> definitely get annoyed about reports being off list because then the
> discussion of the problem doesn't get archived for other people who
> might benefit from it in future.
>
> It is already common practice for people doing testing to copy fairly
> widely so it shouldn't be that much of a surprise.
>

As we're only bisecting boot on mainline and stable, the number
of emails is actually pretty low at the moment.  There's a valid
result only about once a month.  That will definitely change as
we start bisecting -next and other tests than plain boot, which I
think should give us a natural progressive ramp-up.  We'll also
have a chance to tune things along the way.

Guillaume

[-- Attachment #2: Type: text/html, Size: 5368 bytes --]

Re: [kernelci] Bisection email recipients

[Mark Brown]

[-- Attachment #1: Type: text/plain, Size: 4177 bytes --]



> On 9 Nov 2018, at 16:59, Guillaume Tucker <guillaume.tucker@gmail.com> wrote:
> 
> 
>> On Wed, Oct 31, 2018 at 10:26 AM Mark Brown <broonie@kernel.org> wrote:
>> On Wed, Oct 31, 2018 at 08:37:13AM +0100, Tomeu Vizoso wrote:
>> > On 10/30/18 6:08 PM, Guillaume Tucker wrote:
>> 
>> > > * author of the commit
>> > > * committer
>> > > * Signed-off-by
>> > > * Acked-by
>> > > * Reported-by
>> 
>> I'd just include any e-mail address in the tags for the mail - tested-by
> 
> Do you mean, to get the same as what "git send-email" would give?
> I had a look and it doesn't seem easy to extract this
> information, maybe you have a tip for that.  Otherwise, I've
> added a simple Python function with a regex to the script I have
> that triggers the bisection email (PR coming soon with details).

Yes, roughly - a good first step would be something like ^.*: .*@ (ie, anything with a colon/space sequence followed by an @).

>
>> is definitely fair game (there's a decent chance they'll want to be
>> involved in testing further fixes), and it's a fairly common practice
>> for other kinds of reporting to just include everyone rather than
>> explicitly learning all the random tags different maintainers use.
>> 
>> > > * mailing list associated with the tree (like for test reports)
>> > > * maintainers of the parts of the code touched (using
>> > >    scripts/get_maintainer.pl and diff stat)
>> 
>> If you're using get_maintainer I'd make sure to use --nogit so that you
>> don't get false positives for people doing cleanup patches.  You can
>> probably just format the patch with git format-patch and then use
>> get_maintainer normally.
>
> Great tip, I'm using it with "--nogit" on the output of "git
> show" and that seems to be doing a good job.
> 
>> > > I also think it would be worth sending an email to the stable
>> > > mailing list, and probably also lkml to explain what we're doing
>> > > and warn people that we're turning this on.
>> 
>> Seems sensible.  It's a shame that we can't turn this on for -next yet,
>> but obviously the way that's built isn't super helpful.
> 
> Bisections on linux-next are coming soon, we just have to get the
> basics all sorted out for mainline and stable first. 
> 
>> > I think that regardless of how well it's working already, it could make
>> > sense to increase the list of recipients incrementally, starting with those
>> > most likely to appreciate the email, and growing towards those less likely.
>> > Maybe I'm being overly cautious, but from past experiences I don't think we
>> > want to risk too much antagonizing people.
>> 
>> > Once we have a feeling that the recipients in a stage are happy with getting
>> > the emails, we can expand to other recipients in the next stage.
>> 
>> > So, to illustrate the point, we could start with the author and
>> > kernel-build-reports@lists.linaro.org, then include ackers, reviewers,
>> > reporter and sobers, then mailing lists.
>> 
>> Not copying enough people can also cause issues as people will expect
>> other relevant people to be copied in and be annoyed about having to do
>> it themselves or possibly even just miss handling the report if they
>> expect someone has been CCed who hasn't - for example if we only send to
>> the patch author then maintainers might be annoyed if they later find
>> out there was a problem we didn't tell them about.  People will
>> definitely get annoyed about reports being off list because then the
>> discussion of the problem doesn't get archived for other people who
>> might benefit from it in future.
>> 
>> It is already common practice for people doing testing to copy fairly
>> widely so it shouldn't be that much of a surprise.
>
> As we're only bisecting boot on mainline and stable, the number
> of emails is actually pretty low at the moment.  There's a valid
> result only about once a month.  That will definitely change as
> we start bisecting -next and other tests than plain boot, which I
> think should give us a natural progressive ramp-up.  We'll also
> have a chance to tune things along the way.
> 
> Guillaume
> 

[-- Attachment #2: Type: text/html, Size: 5971 bytes --]

Re: [kernelci] Bisection email recipients

[Guillaume Tucker]

[-- Attachment #1: Type: text/plain, Size: 2104 bytes --]

On Fri, Nov 9, 2018 at 8:38 PM Mark Brown <broonie@gmail.com> wrote:

>
>
> On 9 Nov 2018, at 16:59, Guillaume Tucker <guillaume.tucker@gmail.com>
> wrote:
>
>
> On Wed, Oct 31, 2018 at 10:26 AM Mark Brown <broonie@kernel.org> wrote:
>
>> On Wed, Oct 31, 2018 at 08:37:13AM +0100, Tomeu Vizoso wrote:
>> > On 10/30/18 6:08 PM, Guillaume Tucker wrote:
>>
>> > > * author of the commit
>> > > * committer
>> > > * Signed-off-by
>> > > * Acked-by
>> > > * Reported-by
>>
>> I'd just include any e-mail address in the tags for the mail - tested-by
>>
>
> Do you mean, to get the same as what "git send-email" would give?
> I had a look and it doesn't seem easy to extract this
> information, maybe you have a tip for that.  Otherwise, I've
> added a simple Python function with a regex to the script I have
> that triggers the bisection email (PR coming soon with details).
>
>
> Yes, roughly - a good first step would be something like ^.*: .*@ (ie,
> anything with a colon/space sequence followed by an @).
>

Got delayed while chasing up other issues, but here's the PR:

    https://github.com/kernelci/kernelci-core-staging/pull/47

[...]

> If you're using get_maintainer I'd make sure to use --nogit so that you
>> don't get false positives for people doing cleanup patches.  You can
>> probably just format the patch with git format-patch and then use
>> get_maintainer normally.
>>
>
> Great tip, I'm using it with "--nogit" on the output of "git
> show" and that seems to be doing a good job.
>
> > > I also think it would be worth sending an email to the stable
>> > > mailing list, and probably also lkml to explain what we're doing
>> > > and warn people that we're turning this on.
>>
>> Seems sensible.  It's a shame that we can't turn this on for -next yet,
>> but obviously the way that's built isn't super helpful.
>>
>
> Bisections on linux-next are coming soon, we just have to get the
> basics all sorted out for mainline and stable first.
>
> There's also a PR to add a "disclaimer" paragraph in the email
report:

    https://github.com/kernelci/kernelci-backend/pull/89

Guillaume

>

[-- Attachment #2: Type: text/html, Size: 4391 bytes --]