* Re: [Ksummit-discuss] Topics for the Maintainer's Summit
2019-09-03 17:27 ` Konstantin Ryabitsev
@ 2019-09-03 17:40 ` Bjorn Helgaas
2019-09-06 10:21 ` Rob Herring
2019-09-03 17:57 ` Mark Brown
` (4 subsequent siblings)
5 siblings, 1 reply; 37+ messages in thread
From: Bjorn Helgaas @ 2019-09-03 17:40 UTC (permalink / raw)
To: Linus Torvalds, Laura Abbott, Bjorn Helgaas, ksummit-discuss
On Tue, Sep 3, 2019 at 12:27 PM Konstantin Ryabitsev
<konstantin@linuxfoundation.org> wrote:
> For quite some time now I've been trying to fund some client-side
> tooling development around public-inbox (the software that drives
> lore.kernel.org). Eric Wong (the principal author of public-inbox), and
> I have had lengthy chats about potential functionality of such tool, and
> what we envision can be described as "local patchwork with a mutt-like
> interface":
>
> - It would use public-inbox repositories to track new patches and
> conversations, so it would no longer be necessary to subscribe to the
> actual mailing list(s). Getting new mail would be equivalent to a "git
> pull".
> - It would have an equivalent of notmuch search, so instead of needing
> to go to lore.kernel.org, you could search the entire mailing list
> locally and perform actions on the results found.
> - Just like Patchwork, it would keep track of new patches and series of
> patches, recognize when new patch/series revisions are posted, track
> reviewed-by's, tested-by's, etc, and provide useful maintainer
> functionality, such as showing interdiffs between revisions.
> - Patches and series can be pre-filtered by keywords or file paths (e.g.
> if you're only interested in arch/arm64/mm/.*, the tool would ignore
> any patches/revisions not touching files in that dir).
> - It would support creating workflows and conditional response actions,
> e.g. "create new branch, apply this series, run these test suites; if
> tests succeed, merge into branch `for-linus`; if merge successful,
> reply to submitter with 'thanks, applied!'; if all went well, archive
> the series; if any steps failed, flag the series for my review".
> - The workflows would run in the background, including using external
> systems if preferred. Maintainers can contribute their workflows to a
> shared repository so others can easily copy and adapt them.
>
> That's obviously not a complete list, but it seems to me that something
> like this would be quite welcome by a lot of maintainers (at least,
> everyone I've talked to about this got really excited). Eric Wong is
> quite willing to work on something like this, but he is not in a
> position to donate so much of his time and effort (especially on top of
> maintaining public-inbox) -- so if we want to see this happen, we need
> to come up with some funds.
>
> I've inquired internally at the Foundation, and while there's general
> willingness to fund such initiatives, the People In Charge Of Money want
> to see a buy-in from maintainers. The natural instinct is to talk to
> Greg, but I believe he's quite happy with his workflow, so while I'm
> sure he'd be happy to feign excitement, he's unlikely to be interested
> in the tool. Linus is not the right person to talk to either, because he
> doesn't deal with patches and tests, so wouldn't benefit from such tool.
>
> So, my plan was to track down Shuah (who's also at the Foundation) and
> Laura (who is on the TAB) at the upcoming summit to float this idea with
> them to see what they think. However, since we're talking about
> lore.kernel.org, tooling and workflows quite a bit already, I figured
> I'll bring this up here as well.
>
> It just seems that every maintainer I spoke with is generally making
> things "sort-of work well enough" by applying a lot of baling wire
> around mail clients, patchwork.kernel.org, gitlab, or all of the above,
> and I'm wondering if everyone is happy to do that, or only doing that
> because a good tool written to fit with the "kernel development model"
> doesn't exist.
>
> So:
>
> - would a tool with such functionality be useful, or would every
> maintainer prefer to continue doing their own thing (in slightly
> different ways)?
I would find something like this incredibly useful. I currently use
patchwork, but I am really sick of the only-when-online, mouse-around,
clickety-click, wait-for-the-web model.
^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: [Ksummit-discuss] Topics for the Maintainer's Summit
2019-09-03 17:40 ` Bjorn Helgaas
@ 2019-09-06 10:21 ` Rob Herring
2019-09-19 1:47 ` Bjorn Helgaas
0 siblings, 1 reply; 37+ messages in thread
From: Rob Herring @ 2019-09-06 10:21 UTC (permalink / raw)
To: bjorn; +Cc: Bjorn Helgaas, ksummit-discuss
On Tue, Sep 3, 2019 at 6:40 PM Bjorn Helgaas <bjorn.helgaas@gmail.com> wrote:
>
> On Tue, Sep 3, 2019 at 12:27 PM Konstantin Ryabitsev
> <konstantin@linuxfoundation.org> wrote:
>
> > For quite some time now I've been trying to fund some client-side
> > tooling development around public-inbox (the software that drives
> > lore.kernel.org). Eric Wong (the principal author of public-inbox), and
> > I have had lengthy chats about potential functionality of such tool, and
> > what we envision can be described as "local patchwork with a mutt-like
> > interface":
> >
> > - It would use public-inbox repositories to track new patches and
> > conversations, so it would no longer be necessary to subscribe to the
> > actual mailing list(s). Getting new mail would be equivalent to a "git
> > pull".
> > - It would have an equivalent of notmuch search, so instead of needing
> > to go to lore.kernel.org, you could search the entire mailing list
> > locally and perform actions on the results found.
> > - Just like Patchwork, it would keep track of new patches and series of
> > patches, recognize when new patch/series revisions are posted, track
> > reviewed-by's, tested-by's, etc, and provide useful maintainer
> > functionality, such as showing interdiffs between revisions.
> > - Patches and series can be pre-filtered by keywords or file paths (e.g.
> > if you're only interested in arch/arm64/mm/.*, the tool would ignore
> > any patches/revisions not touching files in that dir).
> > - It would support creating workflows and conditional response actions,
> > e.g. "create new branch, apply this series, run these test suites; if
> > tests succeed, merge into branch `for-linus`; if merge successful,
> > reply to submitter with 'thanks, applied!'; if all went well, archive
> > the series; if any steps failed, flag the series for my review".
> > - The workflows would run in the background, including using external
> > systems if preferred. Maintainers can contribute their workflows to a
> > shared repository so others can easily copy and adapt them.
> >
> > That's obviously not a complete list, but it seems to me that something
> > like this would be quite welcome by a lot of maintainers (at least,
> > everyone I've talked to about this got really excited). Eric Wong is
> > quite willing to work on something like this, but he is not in a
> > position to donate so much of his time and effort (especially on top of
> > maintaining public-inbox) -- so if we want to see this happen, we need
> > to come up with some funds.
> >
> > I've inquired internally at the Foundation, and while there's general
> > willingness to fund such initiatives, the People In Charge Of Money want
> > to see a buy-in from maintainers. The natural instinct is to talk to
> > Greg, but I believe he's quite happy with his workflow, so while I'm
> > sure he'd be happy to feign excitement, he's unlikely to be interested
> > in the tool. Linus is not the right person to talk to either, because he
> > doesn't deal with patches and tests, so wouldn't benefit from such tool.
> >
> > So, my plan was to track down Shuah (who's also at the Foundation) and
> > Laura (who is on the TAB) at the upcoming summit to float this idea with
> > them to see what they think. However, since we're talking about
> > lore.kernel.org, tooling and workflows quite a bit already, I figured
> > I'll bring this up here as well.
> >
> > It just seems that every maintainer I spoke with is generally making
> > things "sort-of work well enough" by applying a lot of baling wire
> > around mail clients, patchwork.kernel.org, gitlab, or all of the above,
> > and I'm wondering if everyone is happy to do that, or only doing that
> > because a good tool written to fit with the "kernel development model"
> > doesn't exist.
> >
> > So:
> >
> > - would a tool with such functionality be useful, or would every
> > maintainer prefer to continue doing their own thing (in slightly
> > different ways)?
>
> I would find something like this incredibly useful. I currently use
> patchwork, but I am really sick of the only-when-online, mouse-around,
> clickety-click, wait-for-the-web model.
You might like my set of bailing wire using patchwork and mutt. It
works offline if you download the patchwork state beforehand and
queues up state changes. The basic flow is:
Load the "New" list from PW (my PW instance is pre-filtered on paths,
so I don't have to sort thru everything on the DT list)
Check for multiple versions of patches, auto email on failure to add
my review tag, check for already applied (to next).
Iterate thru the patch list:
- Run checkpatch.pl
- open mutt for each patch. Mutt has the full DT list, so I can look
at the rest of the series if I want.
- After exiting mutt, prompt for PW state change
- Possibly apply it
- Generate replies for applied, reviewed-by or acked-by
Happy to demo it at LPC if you are interested. You can find it
here[1]. The main script is pw-review.
Of course I would happily switch to something else like this proposal
if it shrinks the scripts I have to maintain. Especially for
generating quoted email replies as dealing with mime, utf-8, base64,
quoted printable is "fun".
Rob
[1] https://gitlab.com/robherring/pw-utils
^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: [Ksummit-discuss] Topics for the Maintainer's Summit
2019-09-06 10:21 ` Rob Herring
@ 2019-09-19 1:47 ` Bjorn Helgaas
2019-09-19 20:52 ` Rob Herring
0 siblings, 1 reply; 37+ messages in thread
From: Bjorn Helgaas @ 2019-09-19 1:47 UTC (permalink / raw)
To: Rob Herring; +Cc: Bjorn Helgaas, ksummit-discuss
On Fri, Sep 6, 2019 at 5:21 AM Rob Herring <robherring2@gmail.com> wrote:
> You might like my set of bailing wire using patchwork and mutt. It
> works offline if you download the patchwork state beforehand and
> queues up state changes. The basic flow is:
>
> Load the "New" list from PW (my PW instance is pre-filtered on paths,
> so I don't have to sort thru everything on the DT list)
> Check for multiple versions of patches, auto email on failure to add
> my review tag, check for already applied (to next).
> Iterate thru the patch list:
> - Run checkpatch.pl
> - open mutt for each patch. Mutt has the full DT list, so I can look
> at the rest of the series if I want.
> - After exiting mutt, prompt for PW state change
> - Possibly apply it
> - Generate replies for applied, reviewed-by or acked-by
>
> Happy to demo it at LPC if you are interested. You can find it
> here[1]. The main script is pw-review.
Thanks for the demo at LPC! I'm trying to understand how all the
pieces fit together.
How do you download the patchwork state beforehand for working
offline? For me, actually working offline is nice but rare; my
complaint is that I have to wait for every little interaction
(delegating, superseding, changing state, etc) to talk to the server.
The waits aren't long, but they make the whole process feel sluggish.
You mentioned some CI bits (to run checkpatch, change patchwork state,
etc). Is there a way to look at that? I'm guessing you also have
some mutt keybindings or macros?
Is http://patchwork.ozlabs.org/project/devicetree-bindings/list/ the
patchwork you're using? ISTR one that showed the CI results.
I guess you keep your mbox trimmed somehow? Starting mutt on my
linux-pci folder takes 5-10 seconds. But for this purpose there
wouldn't really be a need to have the *entire* history, I guess.
_______________________________________________
Ksummit-discuss mailing list
Ksummit-discuss@lists.linuxfoundation.org
https://lists.linuxfoundation.org/mailman/listinfo/ksummit-discuss
^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: [Ksummit-discuss] Topics for the Maintainer's Summit
2019-09-19 1:47 ` Bjorn Helgaas
@ 2019-09-19 20:52 ` Rob Herring
2019-09-20 13:37 ` Mark Brown
0 siblings, 1 reply; 37+ messages in thread
From: Rob Herring @ 2019-09-19 20:52 UTC (permalink / raw)
To: bjorn; +Cc: Bjorn Helgaas, ksummit-discuss
On Wed, Sep 18, 2019 at 8:48 PM Bjorn Helgaas <bjorn.helgaas@gmail.com> wrote:
>
> On Fri, Sep 6, 2019 at 5:21 AM Rob Herring <robherring2@gmail.com> wrote:
> > You might like my set of bailing wire using patchwork and mutt. It
> > works offline if you download the patchwork state beforehand and
> > queues up state changes. The basic flow is:
> >
> > Load the "New" list from PW (my PW instance is pre-filtered on paths,
> > so I don't have to sort thru everything on the DT list)
> > Check for multiple versions of patches, auto email on failure to add
> > my review tag, check for already applied (to next).
> > Iterate thru the patch list:
> > - Run checkpatch.pl
> > - open mutt for each patch. Mutt has the full DT list, so I can look
> > at the rest of the series if I want.
> > - After exiting mutt, prompt for PW state change
> > - Possibly apply it
> > - Generate replies for applied, reviewed-by or acked-by
> >
> > Happy to demo it at LPC if you are interested. You can find it
> > here[1]. The main script is pw-review.
>
> Thanks for the demo at LPC! I'm trying to understand how all the
> pieces fit together.
>
> How do you download the patchwork state beforehand for working
> offline? For me, actually working offline is nice but rare; my
> complaint is that I have to wait for every little interaction
> (delegating, superseding, changing state, etc) to talk to the server.
> The waits aren't long, but they make the whole process feel sluggish.
I just run 'pwclient list' formatted so I can parse it and dump into a
file. After that, the server interaction is mainly just doing
'pwclient update' commands in the review loop. In the offline case,
instead of running the commands, I just save them to another file to
run later.
> You mentioned some CI bits (to run checkpatch, change patchwork state,
> etc). Is there a way to look at that? I'm guessing you also have
> some mutt keybindings or macros?
Basically, I run this script which can run either locally on your
system or as a CI job:
https://gitlab.com/robherring/pw-utils/blob/master/pw-checks
This is the CI job:
https://gitlab.com/robherring/linux-dt-review/-/jobs/299584584
Either way, checks get added to the patch state. For example:
https://patchwork.ozlabs.org/patch/1164550/
A somewhat design goal I had was to not tie this into mutt too much.
About all I have is a git am key binding, but now I usually apply
using 'pwclient git-am' so a I get the tags. That's one thing that
doesn't work offline. Not a big deal for me as most things go thru
other maintainers. I just leave anything I'm applying pending and go
thru them again when online. It wouldn't be too hard to just download
all the patches from patchwork up front and then use that to apply
patches.
> Is http://patchwork.ozlabs.org/project/devicetree-bindings/list/ the
> patchwork you're using? ISTR one that showed the CI results.
Yes.
> I guess you keep your mbox trimmed somehow? Starting mutt on my
> linux-pci folder takes 5-10 seconds. But for this purpose there
> wouldn't really be a need to have the *entire* history, I guess.
Yeah, gmail limits it for me.
BTW, I'm using maildir currently. I switched from mbox at some point
as I had some issues with searching the mbox.
Rob
_______________________________________________
Ksummit-discuss mailing list
Ksummit-discuss@lists.linuxfoundation.org
https://lists.linuxfoundation.org/mailman/listinfo/ksummit-discuss
^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: [Ksummit-discuss] Topics for the Maintainer's Summit
2019-09-03 17:27 ` Konstantin Ryabitsev
2019-09-03 17:40 ` Bjorn Helgaas
@ 2019-09-03 17:57 ` Mark Brown
2019-09-03 18:14 ` Dan Williams
` (3 subsequent siblings)
5 siblings, 0 replies; 37+ messages in thread
From: Mark Brown @ 2019-09-03 17:57 UTC (permalink / raw)
To: Linus Torvalds, Laura Abbott, Bjorn Helgaas, ksummit-discuss
[-- Attachment #1: Type: text/plain, Size: 576 bytes --]
On Tue, Sep 03, 2019 at 01:27:08PM -0400, Konstantin Ryabitsev wrote:
> For quite some time now I've been trying to fund some client-side tooling
> development around public-inbox (the software that drives lore.kernel.org).
> Eric Wong (the principal author of public-inbox), and I have had lengthy
> chats about potential functionality of such tool, and what we envision can
> be described as "local patchwork with a mutt-like interface":
This all sounds incredibly useful, I'd be super happy to see it - the
initial proposal does a lot of what my existing scripting does.
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]
^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: [Ksummit-discuss] Topics for the Maintainer's Summit
2019-09-03 17:27 ` Konstantin Ryabitsev
2019-09-03 17:40 ` Bjorn Helgaas
2019-09-03 17:57 ` Mark Brown
@ 2019-09-03 18:14 ` Dan Williams
2019-09-03 21:59 ` Wolfram Sang
` (2 subsequent siblings)
5 siblings, 0 replies; 37+ messages in thread
From: Dan Williams @ 2019-09-03 18:14 UTC (permalink / raw)
To: Linus Torvalds, Laura Abbott, Bjorn Helgaas, ksummit-discuss
On Tue, Sep 3, 2019 at 10:27 AM Konstantin Ryabitsev
<konstantin@linuxfoundation.org>
[..]
> So:
>
> - would a tool with such functionality be useful, or would every
> maintainer prefer to continue doing their own thing (in slightly
> different ways)?
Yes, I would consider switching to this. The kernel.org patchwork-bot
+ the getpatchwork tool [1] does some of this for me, but lossless
patch reception, sharing rules and triggers for patches (not just
git-hooks) is a powerful superset.
[1]: https://github.com/getpatchwork/git-pw
^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: [Ksummit-discuss] Topics for the Maintainer's Summit
2019-09-03 17:27 ` Konstantin Ryabitsev
` (2 preceding siblings ...)
2019-09-03 18:14 ` Dan Williams
@ 2019-09-03 21:59 ` Wolfram Sang
2019-09-04 8:34 ` Jan Kara
2019-09-04 12:08 ` Laurent Pinchart
5 siblings, 0 replies; 37+ messages in thread
From: Wolfram Sang @ 2019-09-03 21:59 UTC (permalink / raw)
To: Konstantin Ryabitsev; +Cc: ksummit-discuss
[-- Attachment #1: Type: text/plain, Size: 504 bytes --]
> For quite some time now I've been trying to fund some client-side tooling
> development around public-inbox (the software that drives lore.kernel.org).
> Eric Wong (the principal author of public-inbox), and I have had lengthy
> chats about potential functionality of such tool, and what we envision can
> be described as "local patchwork with a mutt-like interface":
Thanks, Konstantin, this sounds like something I would like to try and
see if it can replace my current "works-good-enough" setup.
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: [Ksummit-discuss] Topics for the Maintainer's Summit
2019-09-03 17:27 ` Konstantin Ryabitsev
` (3 preceding siblings ...)
2019-09-03 21:59 ` Wolfram Sang
@ 2019-09-04 8:34 ` Jan Kara
2019-09-04 12:08 ` Laurent Pinchart
5 siblings, 0 replies; 37+ messages in thread
From: Jan Kara @ 2019-09-04 8:34 UTC (permalink / raw)
To: Konstantin Ryabitsev; +Cc: Bjorn Helgaas, ksummit-discuss
On Tue 03-09-19 13:27:08, Konstantin Ryabitsev wrote:
> - would a tool with such functionality be useful, or would every maintainer
> prefer to continue doing their own thing (in slightly different ways)?
I'm more often on the 'review stuff' side of things rather than juggling
huge amounts of patches but having ability to easily go to previous
versions of series / seeing interdiffs would be certainly useful for me and
you seem to offer that :).
Honza
--
Jan Kara <jack@suse.com>
SUSE Labs, CR
^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: [Ksummit-discuss] Topics for the Maintainer's Summit
2019-09-03 17:27 ` Konstantin Ryabitsev
` (4 preceding siblings ...)
2019-09-04 8:34 ` Jan Kara
@ 2019-09-04 12:08 ` Laurent Pinchart
2019-09-04 13:47 ` Konstantin Ryabitsev
5 siblings, 1 reply; 37+ messages in thread
From: Laurent Pinchart @ 2019-09-04 12:08 UTC (permalink / raw)
To: Konstantin Ryabitsev; +Cc: Bjorn Helgaas, ksummit-discuss
Hi Konstantin,
On Tue, Sep 03, 2019 at 01:27:08PM -0400, Konstantin Ryabitsev wrote:
> On Tue, Sep 03, 2019 at 09:07:00AM -0700, Linus Torvalds wrote:
> > I think some of the push-back to the GPU people wasn't from them not
> > inventing the group maintainership like Dave said, but from that being
> > presented as some kind of "this is the way to do it". When it's just
> > _one_ way to do it, and other groups have completely different
> > infrastructure and models..
> >
> > So I don't think we can force some workflows.
>
> For quite some time now I've been trying to fund some client-side
> tooling development around public-inbox (the software that drives
> lore.kernel.org). Eric Wong (the principal author of public-inbox), and
> I have had lengthy chats about potential functionality of such tool, and
> what we envision can be described as "local patchwork with a mutt-like
> interface":
>
> - It would use public-inbox repositories to track new patches and
> conversations, so it would no longer be necessary to subscribe to the
> actual mailing list(s). Getting new mail would be equivalent to a "git
> pull".
> - It would have an equivalent of notmuch search, so instead of needing
> to go to lore.kernel.org, you could search the entire mailing list
> locally and perform actions on the results found.
> - Just like Patchwork, it would keep track of new patches and series of
> patches, recognize when new patch/series revisions are posted, track
> reviewed-by's, tested-by's, etc, and provide useful maintainer
> functionality, such as showing interdiffs between revisions.
I've been thinking about this for about a year now. One issue that makes
this difficult is that many M[UTD]A software mangle e-mails in a way
that make extracting information automatically pretty painful. Google's
answer to this was Gerrit, which solved this particular issue, but
disrupted the e-mail-based workflow in a way that is not acceptable to
many developers (myself included). A better, in my opinion, solution
would have been standardisation of a format to exchange review
information. Quite obviously going for a markup language (be it XML,
json, yaml or whatever is hype today) is a no-go, given that we would
destroy everybody's workflow again. My idea was to use a review format
that is as close to e-mail as possible (with > quote markers and
everything that people are already familiar with). Of course M[UTD]A
software would still mangle it, but given reasonable M[TD]As, I think we
could teach some of the MUAs commonly used (mutt comes to mind) to
behave properly (through plugins, scripts, settings files, ...). E-mails
that would not be mangled through the process would be easily parsable
by the tool you would like to develop. That would not give us full
coverage, but if we manage to establish such an end-to-end solution, we
could then push it to more MUAs. This would allow to tackle this complex
problem one step at a time, while not alienating developers by asking
them to leave their MUA. Over time we could the develop more tooling
around that review exchange format, once a large enough portion of
exchanged reviews will follow it.
I know I'm dreaming, but is anyone else sharing this dream ?
> - Patches and series can be pre-filtered by keywords or file paths (e.g.
> if you're only interested in arch/arm64/mm/.*, the tool would ignore
> any patches/revisions not touching files in that dir).
> - It would support creating workflows and conditional response actions,
> e.g. "create new branch, apply this series, run these test suites; if
> tests succeed, merge into branch `for-linus`; if merge successful,
> reply to submitter with 'thanks, applied!'; if all went well, archive
> the series; if any steps failed, flag the series for my review".
> - The workflows would run in the background, including using external
> systems if preferred. Maintainers can contribute their workflows to a
> shared repository so others can easily copy and adapt them.
>
> That's obviously not a complete list, but it seems to me that something
> like this would be quite welcome by a lot of maintainers (at least,
> everyone I've talked to about this got really excited).
I'd say it sounds too good to be true. Being often on the move I would
love a tool that would let me work offline. Even when online, being able
to run queries locally and script actions without the need for a web
browser interface would be amazing.
> Eric Wong is
> quite willing to work on something like this, but he is not in a
> position to donate so much of his time and effort (especially on top of
> maintaining public-inbox) -- so if we want to see this happen, we need
> to come up with some funds.
>
> I've inquired internally at the Foundation, and while there's general
> willingness to fund such initiatives, the People In Charge Of Money want
> to see a buy-in from maintainers. The natural instinct is to talk to
> Greg, but I believe he's quite happy with his workflow, so while I'm
> sure he'd be happy to feign excitement, he's unlikely to be interested
> in the tool. Linus is not the right person to talk to either, because he
> doesn't deal with patches and tests, so wouldn't benefit from such tool.
>
> So, my plan was to track down Shuah (who's also at the Foundation) and
> Laura (who is on the TAB) at the upcoming summit to float this idea with
> them to see what they think. However, since we're talking about
> lore.kernel.org, tooling and workflows quite a bit already, I figured
> I'll bring this up here as well.
>
> It just seems that every maintainer I spoke with is generally making
> things "sort-of work well enough" by applying a lot of baling wire
> around mail clients, patchwork.kernel.org, gitlab, or all of the above,
> and I'm wondering if everyone is happy to do that, or only doing that
> because a good tool written to fit with the "kernel development model"
> doesn't exist.
>
> So:
>
> - would a tool with such functionality be useful, or would every
> maintainer prefer to continue doing their own thing (in slightly
> different ways)?
> - would you (or your employer entity) be willing to participate in a
> fundraiser to help fund the development of such tool (in case we
> cannot get the LF to fully fund it)?
> - would it be okay if the tool is written in NPM/javascript?
>
> Okay, just kidding about the NPM bit. ;)
You won't fool us, everybody knows it will be written in Rust ;-)
--
Regards,
Laurent Pinchart
^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: [Ksummit-discuss] Topics for the Maintainer's Summit
2019-09-04 12:08 ` Laurent Pinchart
@ 2019-09-04 13:47 ` Konstantin Ryabitsev
2019-09-05 8:21 ` Jani Nikula
0 siblings, 1 reply; 37+ messages in thread
From: Konstantin Ryabitsev @ 2019-09-04 13:47 UTC (permalink / raw)
To: Laurent Pinchart; +Cc: Bjorn Helgaas, ksummit-discuss
On Wed, Sep 04, 2019 at 03:08:43PM +0300, Laurent Pinchart wrote:
> > - Just like Patchwork, it would keep track of new patches and series of
> > patches, recognize when new patch/series revisions are posted, track
> > reviewed-by's, tested-by's, etc, and provide useful maintainer
> > functionality, such as showing interdiffs between revisions.
>
> I've been thinking about this for about a year now. One issue that makes
> this difficult is that many M[UTD]A software mangle e-mails in a way
> that make extracting information automatically pretty painful. Google's
> answer to this was Gerrit, which solved this particular issue, but
> disrupted the e-mail-based workflow in a way that is not acceptable to
> many developers (myself included). A better, in my opinion, solution
> would have been standardisation of a format to exchange review
> information. Quite obviously going for a markup language (be it XML,
> json, yaml or whatever is hype today) is a no-go, given that we would
> destroy everybody's workflow again. My idea was to use a review format
> that is as close to e-mail as possible (with > quote markers and
> everything that people are already familiar with). Of course M[UTD]A
> software would still mangle it, but given reasonable M[TD]As, I think we
> could teach some of the MUAs commonly used (mutt comes to mind) to
> behave properly (through plugins, scripts, settings files, ...). E-mails
> that would not be mangled through the process would be easily parsable
> by the tool you would like to develop. That would not give us full
> coverage, but if we manage to establish such an end-to-end solution, we
> could then push it to more MUAs. This would allow to tackle this complex
> problem one step at a time, while not alienating developers by asking
> them to leave their MUA. Over time we could the develop more tooling
> around that review exchange format, once a large enough portion of
> exchanged reviews will follow it.
One way to prevent mail software from mangling message bodies is to send
them as multipart/signed -- at least most MTA/MDAs know not to touch
those. I know git-am handles multipart/signed patches just fine (though
it just ignores signatures), and most gui MUAs just shrug the signatures
off by showing them as useless attachments.
Doesn't help much for cases where people use their own MUAs to send
patches, but at least we can prevent the transmission/display parts of
the equation from messing with structured mail content.
(Of course, in my beautiful vision of the future we aren't using
mail clients at all any more, but let's leave that topic on the
sci-fi/fantasy shelf for now.)
> I'd say it sounds too good to be true. Being often on the move I would
> love a tool that would let me work offline. Even when online, being able
> to run queries locally and script actions without the need for a web
> browser interface would be amazing.
It is quite literally too good to be true, because this tool does not
exist. ;) However, having this feedback will hopefully help me make the
case for coming up with some funds to get things going.
Best regards,
-K
^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: [Ksummit-discuss] Topics for the Maintainer's Summit
2019-09-04 13:47 ` Konstantin Ryabitsev
@ 2019-09-05 8:21 ` Jani Nikula
2019-09-06 10:50 ` Rob Herring
0 siblings, 1 reply; 37+ messages in thread
From: Jani Nikula @ 2019-09-05 8:21 UTC (permalink / raw)
To: Konstantin Ryabitsev, Laurent Pinchart; +Cc: Bjorn Helgaas, ksummit-discuss
On Wed, 04 Sep 2019, Konstantin Ryabitsev <konstantin@linuxfoundation.org> wrote:
> On Wed, Sep 04, 2019 at 03:08:43PM +0300, Laurent Pinchart wrote:
>> > - Just like Patchwork, it would keep track of new patches and series of
>> > patches, recognize when new patch/series revisions are posted, track
>> > reviewed-by's, tested-by's, etc, and provide useful maintainer
>> > functionality, such as showing interdiffs between revisions.
>>
>> I've been thinking about this for about a year now. One issue that makes
>> this difficult is that many M[UTD]A software mangle e-mails in a way
>> that make extracting information automatically pretty painful. Google's
>> answer to this was Gerrit, which solved this particular issue, but
>> disrupted the e-mail-based workflow in a way that is not acceptable to
>> many developers (myself included). A better, in my opinion, solution
>> would have been standardisation of a format to exchange review
>> information. Quite obviously going for a markup language (be it XML,
>> json, yaml or whatever is hype today) is a no-go, given that we would
>> destroy everybody's workflow again. My idea was to use a review format
>> that is as close to e-mail as possible (with > quote markers and
>> everything that people are already familiar with). Of course M[UTD]A
>> software would still mangle it, but given reasonable M[TD]As, I think we
>> could teach some of the MUAs commonly used (mutt comes to mind) to
>> behave properly (through plugins, scripts, settings files, ...). E-mails
>> that would not be mangled through the process would be easily parsable
>> by the tool you would like to develop. That would not give us full
>> coverage, but if we manage to establish such an end-to-end solution, we
>> could then push it to more MUAs. This would allow to tackle this complex
>> problem one step at a time, while not alienating developers by asking
>> them to leave their MUA. Over time we could the develop more tooling
>> around that review exchange format, once a large enough portion of
>> exchanged reviews will follow it.
>
> One way to prevent mail software from mangling message bodies is to send
> them as multipart/signed -- at least most MTA/MDAs know not to touch
> those. I know git-am handles multipart/signed patches just fine (though
> it just ignores signatures), and most gui MUAs just shrug the signatures
> off by showing them as useless attachments.
>
> Doesn't help much for cases where people use their own MUAs to send
> patches, but at least we can prevent the transmission/display parts of
> the equation from messing with structured mail content.
>
> (Of course, in my beautiful vision of the future we aren't using
> mail clients at all any more, but let's leave that topic on the
> sci-fi/fantasy shelf for now.)
The sci-fi doesn't turn to reality in massive disruptive jumps. There
are realistic intermediate steps that could be taken.
I have been, and still am, a proponent of email based review.
I've also contributed significantly to a MUA, and my observation is that
email is a massively distributed fuzzing project for email software that
allows message transmission in the sideband.
What if git push and pull operated on top of an unreliable and lossy
transmission channel, without integrity checks, that you had to
configure and set up yourself? That's pretty much what we're doing with
git send-email and am, isn't it?
Generally I think there are more issues in the submission side; there
are more people contributing than applying patches, more setups, more
configuration that can go wrong. Roughly speaking the masses of
contributors are less experienced than the maintainers. What if we tried
to provide a way to contribute using something based on git push
instead?
I'm sure you'll think of a thousand things that would not work and why
it would be just another broken github like thing, but consider this:
- The system would receive the changes by git push, and would mail out
the patches to the relevant lists itself. It would have SMTP figured
out. It would always mail the patches using the right git send-email
options. It could always send a cover letter with the right diffstat,
and with a git url to the commits.
- The system could decide what the relevant lists as well as maintainers
to Cc are, using up-to-date info from MAINTAINERS. It could provide a
way for maintainers and developers to opt in/out of Cc, in fine
grained ways, instead of leaving that decision to the contributor.
- The system would know what the message-ids of the patches are, because
it would generate them itself. Thus it would know what messages on the
list are patches it sent, and which versions of the patches and/or
series, and which replies are review. (It's incredibly hard for
patchwork to identify patch series and their versions just by reading
the list. It can get confused by review that contains a patch.)
- New versions of patch series could automatically contain a diff
against the previous version of the patches/series. You could easily
tell if the previous review still holds or needs checking.
- You'd still retain the familiar email based review, but it would be
easier to find the various versions of the series, and you'd always
have the changes readily available in a git repo. (This ties to a
previous discussion about how to link patch series versions
together. It could be all taken care of, automatically.)
- The maintainers could keep their email based flow, applying patches
from the list. But there'd be fewer email related technical problems
with them. Additionally, there'd be a way to pull the patches directly
from a git tree, possibly pre-amended with the Reviewed-by, Tested-by,
Link, etc. tags.
- You could add tons of useful git hooks on the contributions, ranging
from pre-merge testing to notifying linked bugs and whatnot.
Note that I'm not suggesting contributors should have git repos from
which they send pull requests. Instead, you'd have a centralized repo
for the project where you can push your submission. Sure, lots of
details to work out there. But the git send-email part is, IMHO, a
broken part of our process, even if the changes keep being distributed
as emailed patches. It just doesn't seem that way to the people on this
list, because we've figured all of this out eons ago for ourselves. We
forget the long tail of contributors that we always brag about.
BR,
Jani.
--
Jani Nikula, Intel Open Source Graphics Center
^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: [Ksummit-discuss] Topics for the Maintainer's Summit
2019-09-05 8:21 ` Jani Nikula
@ 2019-09-06 10:50 ` Rob Herring
2019-09-06 19:21 ` Linus Torvalds
[not found] ` <20190911095305.36104206A1@mail.kernel.org>
0 siblings, 2 replies; 37+ messages in thread
From: Rob Herring @ 2019-09-06 10:50 UTC (permalink / raw)
To: Jani Nikula; +Cc: ksummit, Bjorn Helgaas, Konstantin Ryabitsev
On Thu, Sep 5, 2019 at 9:20 AM Jani Nikula <jani.nikula@intel.com> wrote:
>
> On Wed, 04 Sep 2019, Konstantin Ryabitsev <konstantin@linuxfoundation.org> wrote:
> > On Wed, Sep 04, 2019 at 03:08:43PM +0300, Laurent Pinchart wrote:
> >> > - Just like Patchwork, it would keep track of new patches and series of
> >> > patches, recognize when new patch/series revisions are posted, track
> >> > reviewed-by's, tested-by's, etc, and provide useful maintainer
> >> > functionality, such as showing interdiffs between revisions.
> >>
> >> I've been thinking about this for about a year now. One issue that makes
> >> this difficult is that many M[UTD]A software mangle e-mails in a way
> >> that make extracting information automatically pretty painful. Google's
> >> answer to this was Gerrit, which solved this particular issue, but
> >> disrupted the e-mail-based workflow in a way that is not acceptable to
> >> many developers (myself included). A better, in my opinion, solution
> >> would have been standardisation of a format to exchange review
> >> information. Quite obviously going for a markup language (be it XML,
> >> json, yaml or whatever is hype today) is a no-go, given that we would
> >> destroy everybody's workflow again. My idea was to use a review format
> >> that is as close to e-mail as possible (with > quote markers and
> >> everything that people are already familiar with). Of course M[UTD]A
> >> software would still mangle it, but given reasonable M[TD]As, I think we
> >> could teach some of the MUAs commonly used (mutt comes to mind) to
> >> behave properly (through plugins, scripts, settings files, ...). E-mails
> >> that would not be mangled through the process would be easily parsable
> >> by the tool you would like to develop. That would not give us full
> >> coverage, but if we manage to establish such an end-to-end solution, we
> >> could then push it to more MUAs. This would allow to tackle this complex
> >> problem one step at a time, while not alienating developers by asking
> >> them to leave their MUA. Over time we could the develop more tooling
> >> around that review exchange format, once a large enough portion of
> >> exchanged reviews will follow it.
> >
> > One way to prevent mail software from mangling message bodies is to send
> > them as multipart/signed -- at least most MTA/MDAs know not to touch
> > those. I know git-am handles multipart/signed patches just fine (though
> > it just ignores signatures), and most gui MUAs just shrug the signatures
> > off by showing them as useless attachments.
> >
> > Doesn't help much for cases where people use their own MUAs to send
> > patches, but at least we can prevent the transmission/display parts of
> > the equation from messing with structured mail content.
> >
> > (Of course, in my beautiful vision of the future we aren't using
> > mail clients at all any more, but let's leave that topic on the
> > sci-fi/fantasy shelf for now.)
>
> The sci-fi doesn't turn to reality in massive disruptive jumps. There
> are realistic intermediate steps that could be taken.
>
> I have been, and still am, a proponent of email based review.
>
> I've also contributed significantly to a MUA, and my observation is that
> email is a massively distributed fuzzing project for email software that
> allows message transmission in the sideband.
>
> What if git push and pull operated on top of an unreliable and lossy
> transmission channel, without integrity checks, that you had to
> configure and set up yourself? That's pretty much what we're doing with
> git send-email and am, isn't it?
>
> Generally I think there are more issues in the submission side; there
> are more people contributing than applying patches, more setups, more
> configuration that can go wrong. Roughly speaking the masses of
> contributors are less experienced than the maintainers. What if we tried
> to provide a way to contribute using something based on git push
> instead?
>
> I'm sure you'll think of a thousand things that would not work and why
> it would be just another broken github like thing, but consider this:
>
> - The system would receive the changes by git push, and would mail out
> the patches to the relevant lists itself. It would have SMTP figured
> out. It would always mail the patches using the right git send-email
> options. It could always send a cover letter with the right diffstat,
> and with a git url to the commits.
Independent of the exact process, a git branch for every series would
be great. As a maintainer, I would love to be able to do 'git fetch
some-remote <message-id>'. I don't really care to write and maintain
code to apply series and figure out what branch they apply to. That
code already exists and I'm sure is more robust. If the submitter
provides the branch to begin with in a automatable way, then great,
but that's a much bigger process change.
> - The system could decide what the relevant lists as well as maintainers
> to Cc are, using up-to-date info from MAINTAINERS. It could provide a
> way for maintainers and developers to opt in/out of Cc, in fine
> grained ways, instead of leaving that decision to the contributor.
>
> - The system would know what the message-ids of the patches are, because
> it would generate them itself. Thus it would know what messages on the
> list are patches it sent, and which versions of the patches and/or
> series, and which replies are review. (It's incredibly hard for
> patchwork to identify patch series and their versions just by reading
> the list. It can get confused by review that contains a patch.)
>
> - New versions of patch series could automatically contain a diff
> against the previous version of the patches/series. You could easily
> tell if the previous review still holds or needs checking.
>
> - You'd still retain the familiar email based review, but it would be
> easier to find the various versions of the series, and you'd always
> have the changes readily available in a git repo. (This ties to a
> previous discussion about how to link patch series versions
> together. It could be all taken care of, automatically.)
>
> - The maintainers could keep their email based flow, applying patches
> from the list. But there'd be fewer email related technical problems
> with them. Additionally, there'd be a way to pull the patches directly
> from a git tree, possibly pre-amended with the Reviewed-by, Tested-by,
> Link, etc. tags.
>
> - You could add tons of useful git hooks on the contributions, ranging
> from pre-merge testing to notifying linked bugs and whatnot.
>
> Note that I'm not suggesting contributors should have git repos from
> which they send pull requests. Instead, you'd have a centralized repo
> for the project where you can push your submission. Sure, lots of
> details to work out there. But the git send-email part is, IMHO, a
> broken part of our process, even if the changes keep being distributed
> as emailed patches. It just doesn't seem that way to the people on this
> list, because we've figured all of this out eons ago for ourselves. We
> forget the long tail of contributors that we always brag about.
I certainly agree that the steps between having a git branch ready and
sending the patches could be improved. If we can automate taking a git
branch and sending the emails on the server side, we could do it on
the client side too. The same problems will exist and need to be
solved: get_maintainers.pl is not completely accurate, who to Cc on
individual patches vs. series, writing version history, etc.
Rob
^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: [Ksummit-discuss] Topics for the Maintainer's Summit
2019-09-06 10:50 ` Rob Herring
@ 2019-09-06 19:21 ` Linus Torvalds
2019-09-06 19:53 ` Olof Johansson
[not found] ` <20190911095305.36104206A1@mail.kernel.org>
1 sibling, 1 reply; 37+ messages in thread
From: Linus Torvalds @ 2019-09-06 19:21 UTC (permalink / raw)
To: Rob Herring; +Cc: Bjorn Helgaas, Konstantin Ryabitsev, ksummit
On Fri, Sep 6, 2019 at 3:51 AM Rob Herring <robherring2@gmail.com> wrote:
>
> Independent of the exact process, a git branch for every series would
> be great.
That actually sounds really nice. Especially if the cover-letter is
then done as a tag (probably not signed), so that when you fetch it
you get the overview automatically - and if you actually do "git pull"
it would fill the merge editor with the cover letter thing.
Even if you then don't really merge it as-is, it would be a lovely way
to just get the whole series to work with locally.
Of course, I'm likely biased. Since I do almost everything with git
(occasional random one-off patches and Andrew's patch-bomb being the
exceptions), I end up just doing a lot of "git fetch" and then looking
at the results. Despite still probably being able to edit patches in
my sleep after decades of looking at them, these days I find that
easier and more powerful to look at things in git than working on
patches.
I end up often doing things like just doing "gitk ..FETCH_HEAD" and
then increasing the context window to see more of the code around the
patch etc.
Of course, right now I only do it with people who use git branches
(doing the "git fetch" for the next pull request while the previous is
going through my build tests, or when people post pointers WIP
branches etc). Being able to do it for random 50-patch series sounds
lovely, particularly when you then can limit the gitk to only the
parts you care about, while _having_ the whole series.
Linus
^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: [Ksummit-discuss] Topics for the Maintainer's Summit
2019-09-06 19:21 ` Linus Torvalds
@ 2019-09-06 19:53 ` Olof Johansson
2019-09-09 8:40 ` Jani Nikula
0 siblings, 1 reply; 37+ messages in thread
From: Olof Johansson @ 2019-09-06 19:53 UTC (permalink / raw)
To: Linus Torvalds; +Cc: Bjorn Helgaas, ksummit, Konstantin Ryabitsev
On Fri, Sep 6, 2019 at 12:22 PM Linus Torvalds
<torvalds@linux-foundation.org> wrote:
>
> On Fri, Sep 6, 2019 at 3:51 AM Rob Herring <robherring2@gmail.com> wrote:
> >
> > Independent of the exact process, a git branch for every series would
> > be great.
>
> That actually sounds really nice. Especially if the cover-letter is
> then done as a tag (probably not signed), so that when you fetch it
> you get the overview automatically - and if you actually do "git pull"
> it would fill the merge editor with the cover letter thing.
>
> Even if you then don't really merge it as-is, it would be a lovely way
> to just get the whole series to work with locally.
>
> Of course, I'm likely biased. Since I do almost everything with git
> (occasional random one-off patches and Andrew's patch-bomb being the
> exceptions), I end up just doing a lot of "git fetch" and then looking
> at the results. Despite still probably being able to edit patches in
> my sleep after decades of looking at them, these days I find that
> easier and more powerful to look at things in git than working on
> patches.
>
> I end up often doing things like just doing "gitk ..FETCH_HEAD" and
> then increasing the context window to see more of the code around the
> patch etc.
>
> Of course, right now I only do it with people who use git branches
> (doing the "git fetch" for the next pull request while the previous is
> going through my build tests, or when people post pointers WIP
> branches etc). Being able to do it for random 50-patch series sounds
> lovely, particularly when you then can limit the gitk to only the
> parts you care about, while _having_ the whole series.
Applying patches to branches with automation is something that's been
on my todo list for a while as well -- especially since having a git
branch pre-staged makes some things easier (running checks, linters,
checkpatch, whatnot) with the way most CI tools work. I'd love to see
this happen. Patchwork has hooks so you can have these "checkers"
report back status, but it can be done over email as well, of course.
Random observation: We're slowly migrating closer to the "web" based
model of github/gitlab/bitbucket where changes come in via a merge
request + branch, but we would be reconstructing it out of email with
the cover letter equivalent of the merge request description, etc.
That's obviously not a problem, just an interesting observation. The
final step of merging it in is still manual in our setup, and that's
what most maintainers still prefer; the "hands off" part of the web
model where you don't actively download and look at the code is what
feels less careful and involved at least for me. Plus the fact that
the master contents of the tree would reside up somewhere on the
internet instead of on the maintainers locally controlled machine with
the trust complications involved in that.
-Olof
^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: [Ksummit-discuss] Topics for the Maintainer's Summit
2019-09-06 19:53 ` Olof Johansson
@ 2019-09-09 8:40 ` Jani Nikula
2019-09-09 9:49 ` Geert Uytterhoeven
0 siblings, 1 reply; 37+ messages in thread
From: Jani Nikula @ 2019-09-09 8:40 UTC (permalink / raw)
To: Olof Johansson, Linus Torvalds
Cc: Bjorn Helgaas, Konstantin Ryabitsev, ksummit
On Fri, 06 Sep 2019, Olof Johansson <olof@lixom.net> wrote:
> Random observation: We're slowly migrating closer to the "web" based
> model of github/gitlab/bitbucket where changes come in via a merge
> request + branch, but we would be reconstructing it out of email with
> the cover letter equivalent of the merge request description, etc.
> That's obviously not a problem, just an interesting observation.
Well, as I tried to explain up-thread, I think it *is* a problem we're
building infrastructure on top of git send-email and am, while we have
git push and pull. Trying to reconstruct everything from email is
problematic because it is lossy.
We can still have the review on emailed patches, and we could still use
git am to apply patches from email, with better reliability if the
sending was done by a service in, say, kernel.org control. Though if we
had the series automatically available in a branch, I'd think people
would move over to picking up the commits from git. And email would only
be used for communication, not data transfer.
> The final step of merging it in is still manual in our setup, and
> that's what most maintainers still prefer; the "hands off" part of the
> web model where you don't actively download and look at the code is
> what feels less careful and involved at least for me. Plus the fact
> that the master contents of the tree would reside up somewhere on the
> internet instead of on the maintainers locally controlled machine with
> the trust complications involved in that.
I'm suggesting maintainers would still have their trees wherever they
feel comfortable having them. I find it hard to understand why emailed
patches would somehow be inherently safer and more trustworthy than git
pull.
BR,
Jani.
--
Jani Nikula, Intel Open Source Graphics Center
^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: [Ksummit-discuss] Topics for the Maintainer's Summit
2019-09-09 8:40 ` Jani Nikula
@ 2019-09-09 9:49 ` Geert Uytterhoeven
2019-09-09 10:16 ` Konstantin Ryabitsev
0 siblings, 1 reply; 37+ messages in thread
From: Geert Uytterhoeven @ 2019-09-09 9:49 UTC (permalink / raw)
To: Jani Nikula; +Cc: Bjorn Helgaas, ksummit, Konstantin Ryabitsev
Hi Jani,
On Mon, Sep 9, 2019 at 10:39 AM Jani Nikula <jani.nikula@intel.com> wrote:
> On Fri, 06 Sep 2019, Olof Johansson <olof@lixom.net> wrote:
> > Random observation: We're slowly migrating closer to the "web" based
> > model of github/gitlab/bitbucket where changes come in via a merge
> > request + branch, but we would be reconstructing it out of email with
> > the cover letter equivalent of the merge request description, etc.
> > That's obviously not a problem, just an interesting observation.
>
> Well, as I tried to explain up-thread, I think it *is* a problem we're
> building infrastructure on top of git send-email and am, while we have
> git push and pull. Trying to reconstruct everything from email is
> problematic because it is lossy.
>
> We can still have the review on emailed patches, and we could still use
> git am to apply patches from email, with better reliability if the
> sending was done by a service in, say, kernel.org control. Though if we
> had the series automatically available in a branch, I'd think people
> would move over to picking up the commits from git. And email would only
> be used for communication, not data transfer.
Do we trust the branch to contain the exact same commits as the
patches reviewed on the mailing list?
For an automatic service on kernel.org, we could.
For branches provided manually by the submitter, or elsewhere, we cannot.
Note that we already trust patchwork, assuming it received the exact
same patch as our inboxes. But for patchwork, the human factor is not
involved, so human mistakes are assumed to be absent.
> > The final step of merging it in is still manual in our setup, and
> > that's what most maintainers still prefer; the "hands off" part of the
> > web model where you don't actively download and look at the code is
> > what feels less careful and involved at least for me. Plus the fact
> > that the master contents of the tree would reside up somewhere on the
> > internet instead of on the maintainers locally controlled machine with
> > the trust complications involved in that.
>
> I'm suggesting maintainers would still have their trees wherever they
> feel comfortable having them. I find it hard to understand why emailed
> patches would somehow be inherently safer and more trustworthy than git
> pull.
The emailed patch is what has been reviewed.
For (sub)maintainers, we trust that the branch they provide contains the
right commits. Still, mistakes happens (check how many pull requests
Linus complains about due to wrong/missing branch/tag).
For random contributors, we do not.
Gr{oetje,eeting}s,
Geert
--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org
In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
-- Linus Torvalds
^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: [Ksummit-discuss] Topics for the Maintainer's Summit
2019-09-09 9:49 ` Geert Uytterhoeven
@ 2019-09-09 10:16 ` Konstantin Ryabitsev
2019-09-09 10:59 ` Geert Uytterhoeven
0 siblings, 1 reply; 37+ messages in thread
From: Konstantin Ryabitsev @ 2019-09-09 10:16 UTC (permalink / raw)
To: Geert Uytterhoeven; +Cc: Bjorn Helgaas, ksummit
On Mon, Sep 09, 2019 at 11:49:48AM +0200, Geert Uytterhoeven wrote:
> > We can still have the review on emailed patches, and we could still use
> > git am to apply patches from email, with better reliability if the
> > sending was done by a service in, say, kernel.org control. Though if we
> > had the series automatically available in a branch, I'd think people
> > would move over to picking up the commits from git. And email would only
> > be used for communication, not data transfer.
>
> Do we trust the branch to contain the exact same commits as the
> patches reviewed on the mailing list?
> For an automatic service on kernel.org, we could.
But we really shouldn't, considering kernel.org is the exact kind of
target that attackers would go after if it was implicitly trusted by
developers. Once patches have been reviewed by maintainers and merged
into their tree, we should be using cryptographic attestation for all
git-centric operations after that -- regardless of whether you pulled
from kernel.org or any other location.
Kernel.org is just there to simplify the moving of bits and we shouldn't
make it a source of trust.
-K
^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: [Ksummit-discuss] Topics for the Maintainer's Summit
2019-09-09 10:16 ` Konstantin Ryabitsev
@ 2019-09-09 10:59 ` Geert Uytterhoeven
2019-09-09 12:37 ` Konstantin Ryabitsev
0 siblings, 1 reply; 37+ messages in thread
From: Geert Uytterhoeven @ 2019-09-09 10:59 UTC (permalink / raw)
To: Konstantin Ryabitsev; +Cc: Bjorn Helgaas, ksummit
Hi Konstantin,
On Mon, Sep 9, 2019 at 12:16 PM Konstantin Ryabitsev
<konstantin@linuxfoundation.org> wrote:
> On Mon, Sep 09, 2019 at 11:49:48AM +0200, Geert Uytterhoeven wrote:
> > > We can still have the review on emailed patches, and we could still use
> > > git am to apply patches from email, with better reliability if the
> > > sending was done by a service in, say, kernel.org control. Though if we
> > > had the series automatically available in a branch, I'd think people
> > > would move over to picking up the commits from git. And email would only
> > > be used for communication, not data transfer.
> >
> > Do we trust the branch to contain the exact same commits as the
> > patches reviewed on the mailing list?
> > For an automatic service on kernel.org, we could.
>
> But we really shouldn't, considering kernel.org is the exact kind of
> target that attackers would go after if it was implicitly trusted by
Sure.
> developers. Once patches have been reviewed by maintainers and merged
> into their tree, we should be using cryptographic attestation for all
> git-centric operations after that -- regardless of whether you pulled
> from kernel.org or any other location.
We already use cryptographic attestations for the later operations.
So the weakest link seems to be the step between public review and
import into git by the maintainer. E.g.
- The review chain on multiple submissions: Vn+1 may contain an evil
change that was not in Vn. As this happens in public, it may be
noticed by reviewers.
- The path between patch submission and git-am: if a patchwork
instance is compromised, evil changes may sneak in.
Gr{oetje,eeting}s,
Geert
--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org
In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
-- Linus Torvalds
^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: [Ksummit-discuss] Topics for the Maintainer's Summit
2019-09-09 10:59 ` Geert Uytterhoeven
@ 2019-09-09 12:37 ` Konstantin Ryabitsev
0 siblings, 0 replies; 37+ messages in thread
From: Konstantin Ryabitsev @ 2019-09-09 12:37 UTC (permalink / raw)
To: Geert Uytterhoeven; +Cc: Bjorn Helgaas, ksummit
On Mon, Sep 09, 2019 at 12:59:26PM +0200, Geert Uytterhoeven wrote:
> So the weakest link seems to be the step between public review and
> import into git by the maintainer. E.g.
> - The review chain on multiple submissions: Vn+1 may contain an evil
> change that was not in Vn. As this happens in public, it may be
> noticed by reviewers.
Theoretically, a tool that is able to show interdiffs between
patches/series would be able to help identify those.
> - The path between patch submission and git-am: if a patchwork
> instance is compromised, evil changes may sneak in.
I'm not sure we can fix this problem in any way that would be
meaningful. I did have some thoughts about this -- for example, we could
run an API service on lore.kernel.org that would allow issuing queries
like:
- do you have a record of a patch
- with patch-id c052719edde5b0c648b2e4629a7fbd72fd948652
- sent by shengjiu.wang@nxp.com
- sent to list with id linux-kernel.vger.kernel.org
- sent on 2019-09-09
If the query returns "True" then you know that at least lore.kernel.org
has received the same patch, but I'm not sure if we'd be solving
anything in the grander scheme of things. We still have to trust
kernel.org, plus an attacker could have sent a malicious patch to LKML
with a forged "From" to make this check return true. Nobody really reads
LKML, so sneaking something unnoticed would be fairly easy.
Alternatively, a developer could require that all submitted patches
must include a cryptographic signature. It could even be made relatively
painless, especially if we taught git tools to do it.
For git-format-patch we'd need something like this:
- generate a patch-id (using `git patch-id --stable`)
- generate a signify-compatible signature for the patch-id
- add 'Patch-id-sig' and 'Patch-id-key' into the basement (under "---")
E.g., for the same patch as the example I used above:
------
Subject: [PATCH 1/3] ASoC: fsl_asrc: Use in(out)put_format instead of in(out)put_word_width
snd_pcm_format_t is more formal than enum asrc_word_width, which has
two property, width and physical width, which is more accurate than
enum asrc_word_width. So it is better to use in(out)put_format
instead of in(out)put_word_width.
Signed-off-by: Shengjiu Wang <shengjiu.wang@nxp.com>
---
sound/soc/fsl/fsl_asrc.c | 56 +++++++++++++++++++++++++++-------------
sound/soc/fsl/fsl_asrc.h | 4 +--
2 files changed, 40 insertions(+), 20 deletions(-)
Patch-id-sig: RWT9fcUvSnHPLrCB+miMdp13r39W9az07CWW+b4OLz5zdtPUaj0N9qnfdNB+cbs8T1jYzHPIWfaf+B6z/ZvSVG9rfE1/Xx6+EgU=
Patch-id-key: RWT9fcUvSnHPLqqyfLbkGBMEscBWciFFp2iBj2XnZPzW69OVIoYwZ25q
diff --git a/sound/soc/fsl/fsl_asrc.c b/sound/soc/fsl/fsl_asrc.c
...etc...
------
Then we teach git-am to keep a trust-on-first-use (TOFU) database of
public keys, so that:
- if there are no entries matching the email in From:, the key is
added and automatically considered trusted
- if there are entries matching From:, the public key is compared
with the TOFU database
- if match, the key is used to validate that the patch-id hasn't
changed
- if no match, generate a warning and let the developer decide if they
trust the new key or not
- if there are entries matching From: in the local TOFU db but the patch
does not have a "Patch-id-sig", generate a warning
This would be more workable than using PGP signatures, but this would
still be a significant pain point for everyone, so I'm not sure how many
would consider using this. What seems reasonable from the security
high-horse I'm sitting on is not always reasonable for others.
:)
-K
^ permalink raw reply [flat|nested] 37+ messages in thread
[parent not found: <20190911095305.36104206A1@mail.kernel.org>]
* Re: [Ksummit-discuss] Topics for the Maintainer's Summit
[not found] ` <20190911095305.36104206A1@mail.kernel.org>
@ 2019-09-11 11:03 ` Christoph Hellwig
2019-09-13 8:19 ` Matthias Brugger
1 sibling, 0 replies; 37+ messages in thread
From: Christoph Hellwig @ 2019-09-11 11:03 UTC (permalink / raw)
To: Stephen Boyd; +Cc: Bjorn Helgaas, ksummit, Konstantin Ryabitsev
On Wed, Sep 11, 2019 at 02:53:04AM -0700, Stephen Boyd wrote:
> > Independent of the exact process, a git branch for every series would
> > be great. As a maintainer, I would love to be able to do 'git fetch
> > some-remote <message-id>'. I don't really care to write and maintain
> > code to apply series and figure out what branch they apply to. That
> > code already exists and I'm sure is more robust.
>
> +1. It would be huge if 'git am' could gain the ability to apply an mbox
> (which it can already do) and parse out the tags to add them to all the
> right patches. I have a script that mostly does this but it needs some
> more work because sometimes people reply to the cover letter and say
> their reviewed-by tag applies to patches 1-3, 5 and 6 and parsing that
> isn't necessarily easy.
Yes, that would help a lot. Any ignoring cover letters and allowing
for normal diff fuzz so that it doesn't completely fail with the
slightest movement in lines. And a vaguely git-rebase like way
to resolve conflicts instead of the current mess requring a manual
patch application.
_______________________________________________
Ksummit-discuss mailing list
Ksummit-discuss@lists.linuxfoundation.org
https://lists.linuxfoundation.org/mailman/listinfo/ksummit-discuss
^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: [Ksummit-discuss] Topics for the Maintainer's Summit
[not found] ` <20190911095305.36104206A1@mail.kernel.org>
2019-09-11 11:03 ` Christoph Hellwig
@ 2019-09-13 8:19 ` Matthias Brugger
1 sibling, 0 replies; 37+ messages in thread
From: Matthias Brugger @ 2019-09-13 8:19 UTC (permalink / raw)
To: Stephen Boyd, Rob Herring, Jani Nikula
Cc: Bjorn Helgaas, ksummit, Konstantin Ryabitsev
On 11/09/2019 11:53, Stephen Boyd wrote:
> Quoting Rob Herring (2019-09-06 03:50:47)
>>
>> Independent of the exact process, a git branch for every series would
>> be great. As a maintainer, I would love to be able to do 'git fetch
>> some-remote <message-id>'. I don't really care to write and maintain
>> code to apply series and figure out what branch they apply to. That
>> code already exists and I'm sure is more robust.
>
> +1. It would be huge if 'git am' could gain the ability to apply an mbox
> (which it can already do) and parse out the tags to add them to all the
> right patches. I have a script that mostly does this but it needs some
> more work because sometimes people reply to the cover letter and say
> their reviewed-by tag applies to patches 1-3, 5 and 6 and parsing that
> isn't necessarily easy.
>
>> If the submitter
>> provides the branch to begin with in a automatable way, then great,
>> but that's a much bigger process change.
>
> I've been formatting patches with the --base option for a few months now
> so that the information about the base commit to apply the patches to is
> part of the cover letter or after the patch if it's a single patch. The
> one missing piece is I don't have a database of patches that are in
> -next or being discussed on the list that these patches may depend on
> (indicated by prerequisite-patch-id:). Of course, this changes the
> process a bit so unless we can somehow force this option on for all git
> users in the kernel and require new users to do this then we're not
> really able to do much. My script falls back to the clk/master branch
> (typically -rc1) so that there's a sane base to patch against when this
> information is missing.
>
>>
>>> - The system could decide what the relevant lists as well as maintainers
>>> to Cc are, using up-to-date info from MAINTAINERS. It could provide a
>>> way for maintainers and developers to opt in/out of Cc, in fine
>>> grained ways, instead of leaving that decision to the contributor.
>>>
>>> - The system would know what the message-ids of the patches are, because
>>> it would generate them itself. Thus it would know what messages on the
>>> list are patches it sent, and which versions of the patches and/or
>>> series, and which replies are review. (It's incredibly hard for
>>> patchwork to identify patch series and their versions just by reading
>>> the list. It can get confused by review that contains a patch.)
>>>
>>> - New versions of patch series could automatically contain a diff
>>> against the previous version of the patches/series. You could easily
>>> tell if the previous review still holds or needs checking.
>
> I do this by applying all patch series and grabbing the version tag out
> of the PATCHv<N> part and using 'git range-diff' to see what's changed.
> I don't do this very often though because it's a huge pain. We could ask
> developers to use --interdiff on 'git format-patch' but again this is a
> process roadblock.
>
>>>
>>> - You'd still retain the familiar email based review, but it would be
>>> easier to find the various versions of the series, and you'd always
>>> have the changes readily available in a git repo. (This ties to a
>>> previous discussion about how to link patch series versions
>>> together. It could be all taken care of, automatically.)
>>>
>>> - The maintainers could keep their email based flow, applying patches
>>> from the list. But there'd be fewer email related technical problems
>>> with them. Additionally, there'd be a way to pull the patches directly
>>> from a git tree, possibly pre-amended with the Reviewed-by, Tested-by,
>>> Link, etc. tags.
>>>
>>> - You could add tons of useful git hooks on the contributions, ranging
>>> from pre-merge testing to notifying linked bugs and whatnot.
>>>
>>> Note that I'm not suggesting contributors should have git repos from
>>> which they send pull requests. Instead, you'd have a centralized repo
>>> for the project where you can push your submission. Sure, lots of
>>> details to work out there. But the git send-email part is, IMHO, a
>>> broken part of our process, even if the changes keep being distributed
>>> as emailed patches. It just doesn't seem that way to the people on this
>>> list, because we've figured all of this out eons ago for ourselves. We
>>> forget the long tail of contributors that we always brag about.
>>
>> I certainly agree that the steps between having a git branch ready and
>> sending the patches could be improved. If we can automate taking a git
>> branch and sending the emails on the server side, we could do it on
>> the client side too. The same problems will exist and need to be
>> solved: get_maintainers.pl is not completely accurate, who to Cc on
>> individual patches vs. series, writing version history, etc.
>
> The git project has this "bridge", sort of. It's called GitGitGadget[1].
> It would be awesome if we could have something similar for the kernel so
> that we can get more contributors through the 'github' model and try to
> nudge them to use email at the same time.
>
I think that would help to lower the burden to contribute to the kernel.
Actually my experience with people starting to work with the kernel is not so
much the technical part they are struggling with. But more all the processes we
have around the kernel. If we could get an "intelligent" GitGitGadget type of
service, that could help all the developers that come from a github-centered
background.
> [1] https://gitgitgadget.github.io
> _______________________________________________
> Ksummit-discuss mailing list
> Ksummit-discuss@lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/ksummit-discuss
>
_______________________________________________
Ksummit-discuss mailing list
Ksummit-discuss@lists.linuxfoundation.org
https://lists.linuxfoundation.org/mailman/listinfo/ksummit-discuss
^ permalink raw reply [flat|nested] 37+ messages in thread