[Ksummit-discuss] [MAINTAINERS SUMMIT] Reflections on kernel development processes

[Ksummit-discuss] [MAINTAINERS SUMMIT] Reflections on kernel development processes

[Theodore Y. Ts'o]

Hi all,

Many of you attended Dmitry Vyukov's talk at the Kernel Summit track
today, "Reflections on Kernel Development Process, Quality, and
Testing".  (For those of you who haven't, the slides are available
here[1].)

[1] https://linuxplumbersconf.org/event/4/contributions/554/attachments/353/584/Reflections__Kernel_Summit_2019.pdf

Greg K-H has suggested, and I strongly agree, that it would be
worthwhile to add this to the agenda of the Maintainer's Summit.  In
particular, what next steps should we take and what should be the
criteria and the process for trying to further standardize our tools
and processes in order to make make our development processes more
mature and to improve developer productivity and happiness.

If you didn't attend the talk, I encourage you to take a look at the
slide, so we don't have to spend time trying to bring people up to
speed on the discussion to date.  My plan is to schedule this as our
first topic tomorrow afternoon.

Thanks,

						- Ted
_______________________________________________
Ksummit-discuss mailing list
Ksummit-discuss@lists.linuxfoundation.org
https://lists.linuxfoundation.org/mailman/listinfo/ksummit-discuss

[Ksummit-discuss] Video of Dmitry's Kernel Summit talk (Was: Reflections on kernel development processes)

[Theodore Y. Ts'o]

On Wed, Sep 11, 2019 at 11:08:04AM -0400, Theodore Y. Ts'o wrote:
> Hi all,
> 
> Many of you attended Dmitry Vyukov's talk at the Kernel Summit track
> today, "Reflections on Kernel Development Process, Quality, and
> Testing".  (For those of you who haven't, the slides are available
> here[1].)
> 
> [1] https://linuxplumbersconf.org/event/4/contributions/554/attachments/353/584/Reflections__Kernel_Summit_2019.pdf
> 
> Greg K-H has suggested, and I strongly agree, that it would be
> worthwhile to add this to the agenda of the Maintainer's Summit.  In
> particular, what next steps should we take and what should be the
> criteria and the process for trying to further standardize our tools
> and processes in order to make make our development processes more
> mature and to improve developer productivity and happiness.

The raw video is (temporarily) available at:

	https://youtu.be/a2Nv-KJyqPk?t=5239

This will disappear when we can get something which is properly
cropped and edited --- this is basically the "hot wash" video which
was made available this morning, and after I failed my saving throw
against Linux video editing tools' user interfaces, I just uploaded it
straight to YouTube.

						- Ted
_______________________________________________
Ksummit-discuss mailing list
Ksummit-discuss@lists.linuxfoundation.org
https://lists.linuxfoundation.org/mailman/listinfo/ksummit-discuss

Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] Reflections on kernel development processes

[Konstantin Ryabitsev]

On Wed, Sep 11, 2019 at 11:08:04AM -0400, Theodore Y. Ts'o wrote:
> Hi all,
> 
> Many of you attended Dmitry Vyukov's talk at the Kernel Summit track
> today, "Reflections on Kernel Development Process, Quality, and
> Testing".  (For those of you who haven't, the slides are available
> here[1].)
> 
> [1] https://linuxplumbersconf.org/event/4/contributions/554/attachments/353/584/Reflections__Kernel_Summit_2019.pdf
> 
> Greg K-H has suggested, and I strongly agree, that it would be
> worthwhile to add this to the agenda of the Maintainer's Summit.  In
> particular, what next steps should we take and what should be the
> criteria and the process for trying to further standardize our tools
> and processes in order to make make our development processes more
> mature and to improve developer productivity and happiness.
> 
> If you didn't attend the talk, I encourage you to take a look at the
> slide, so we don't have to spend time trying to bring people up to
> speed on the discussion to date.  My plan is to schedule this as our
> first topic tomorrow afternoon.

To follow-up, this is a very rough outline of a proposal that I am going
to submit to the Foundation in hopes to fund maintainer tool
development. It follows along some of the lines highlighted in Dmitry's
talk.

--------

# Stage 1 (Normal brain): "local patchwork"

- Implement a mutt-like tool ("putt"?) that uses locally cloned
  public-inbox archives to track patches/series submitted to mailing
  lists
    - Pre-filters by keywords and paths in patches
    - Tracks and automatically inserts taglines
      (Reviewed-by, Acked-by, Tested-by)
    - Can ignore a patch/series until it sees certain taglines
      (Tested-by: zeroday bot, Reviewed-by: Trusty Intern)
    - Automatically tracks latest series and offers an interdiff view
      between series revisions ("show me what changed between v1 and v2")
    - Allows responding to patches and conversations a-la mutt
    - Allows applying patches/series to local repos


# Stage 2 (Enlightened brain): "now with CI and workflows"

- Add configurable workflow functionality allowing maintainers to run
  local or remote tasks on patches and series, before maintainer sees
  the patches, e.g.:
    - Create a branch and attempt to apply series
    - If succeeds, run a batch of CI tests
    - If succeeds, mark as "CI passed" and show the maintainer
    - If fails, reject automatically using a "sorry, tests failed"
      template, including relevant error messages

- All of the above runs outside of the UI tool ("putt-cid"?) and defines CI
  routines that can run in cloudy environments or locally using
  containers.
- Putt communicates with putt-cid locally or remotely to identify
  patches/series that the maintainer should review


# Stage 3 (Galaxy brain): "email as a secondary channel"

- Support additional distributed communication mechanisms in conjunction
  with existing mailing lists.
  - SSB is a peer-to-peer replication framework that has built-in
    cryptographic integrity and attestation ("immutable git-like
    chains per participating developer")
    - offers native support for structured data like bug reports, CI
      results, code review comments, etc.
    - can easily support email-to-SSB and web-to-SSB bridges, so
      developers can choose to participate using familiar tools
    - has known limitations in v1 of the protocol, but v2 is being
      actively developed to address them.
    - or we can take it as a base and develop an SSB-like protocol that
      better suits distributed development needs.

  - Radicle is another interesting alternative that creates a mechanism
    for automating some maintainer tasks by defining "state machines,"
    e.g.:
    - automatically merge a revision if all tests pass and at least 2
      Reviewed-by's are seen.
    - May have been sipping the blockchain cool-aid a bit too much
      ("Immutable append-only records").
_______________________________________________
Ksummit-discuss mailing list
Ksummit-discuss@lists.linuxfoundation.org
https://lists.linuxfoundation.org/mailman/listinfo/ksummit-discuss

Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] Reflections on kernel development processes

[Rob Herring]

On Thu, Sep 12, 2019 at 1:06 PM Konstantin Ryabitsev
<konstantin@linuxfoundation.org> wrote:
>
> On Wed, Sep 11, 2019 at 11:08:04AM -0400, Theodore Y. Ts'o wrote:
> > Hi all,
> >
> > Many of you attended Dmitry Vyukov's talk at the Kernel Summit track
> > today, "Reflections on Kernel Development Process, Quality, and
> > Testing".  (For those of you who haven't, the slides are available
> > here[1].)
> >
> > [1] https://linuxplumbersconf.org/event/4/contributions/554/attachments/353/584/Reflections__Kernel_Summit_2019.pdf
> >
> > Greg K-H has suggested, and I strongly agree, that it would be
> > worthwhile to add this to the agenda of the Maintainer's Summit.  In
> > particular, what next steps should we take and what should be the
> > criteria and the process for trying to further standardize our tools
> > and processes in order to make make our development processes more
> > mature and to improve developer productivity and happiness.
> >
> > If you didn't attend the talk, I encourage you to take a look at the
> > slide, so we don't have to spend time trying to bring people up to
> > speed on the discussion to date.  My plan is to schedule this as our
> > first topic tomorrow afternoon.
>
> To follow-up, this is a very rough outline of a proposal that I am going
> to submit to the Foundation in hopes to fund maintainer tool
> development. It follows along some of the lines highlighted in Dmitry's
> talk.

As long as the submitter side is a free-for-all, that's going to make
the maintainer tools more complex to implement and less reliable.
Seems like submitter tools should be included here too. Specifically,
the steps between having a git branch ready for review and
git-send-email are a lot of manual steps. Though maybe a lot of that
is just all of us agreeing on more standardized requirements as is
being discussed in 'Maintainer Entry Profiles'.

> --------
>
> # Stage 1 (Normal brain): "local patchwork"
>
> - Implement a mutt-like tool ("putt"?) that uses locally cloned
>   public-inbox archives to track patches/series submitted to mailing
>   lists
>     - Pre-filters by keywords and paths in patches
>     - Tracks and automatically inserts taglines
>       (Reviewed-by, Acked-by, Tested-by)

A frequent problem I see is submitters failing to add tags on
subsequent versions. Either the tool can check older versions and
apply those tags too or provide an auto reply to the submitter. Either
way, if we have an interdiff, it should be easy to distinguish cases
of the submitter forgetting vs. patch changed and we need to re-review
it.

>     - Can ignore a patch/series until it sees certain taglines
>       (Tested-by: zeroday bot, Reviewed-by: Trusty Intern)

There's also ignore after seeing a tag. IOW, if I've acked something I
want to ignore it.

>     - Automatically tracks latest series and offers an interdiff view
>       between series revisions ("show me what changed between v1 and v2")
>     - Allows responding to patches and conversations a-la mutt
>     - Allows applying patches/series to local repos
_______________________________________________
Ksummit-discuss mailing list
Ksummit-discuss@lists.linuxfoundation.org
https://lists.linuxfoundation.org/mailman/listinfo/ksummit-discuss

Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] Reflections on kernel development processes

[Laurent Pinchart]

Hi Konstantin,

On Thu, Sep 12, 2019 at 08:06:02AM -0400, Konstantin Ryabitsev wrote:
> On Wed, Sep 11, 2019 at 11:08:04AM -0400, Theodore Y. Ts'o wrote:
> > Hi all,
> > 
> > Many of you attended Dmitry Vyukov's talk at the Kernel Summit track
> > today, "Reflections on Kernel Development Process, Quality, and
> > Testing".  (For those of you who haven't, the slides are available
> > here[1].)
> > 
> > [1] https://linuxplumbersconf.org/event/4/contributions/554/attachments/353/584/Reflections__Kernel_Summit_2019.pdf
> > 
> > Greg K-H has suggested, and I strongly agree, that it would be
> > worthwhile to add this to the agenda of the Maintainer's Summit.  In
> > particular, what next steps should we take and what should be the
> > criteria and the process for trying to further standardize our tools
> > and processes in order to make make our development processes more
> > mature and to improve developer productivity and happiness.
> > 
> > If you didn't attend the talk, I encourage you to take a look at the
> > slide, so we don't have to spend time trying to bring people up to
> > speed on the discussion to date.  My plan is to schedule this as our
> > first topic tomorrow afternoon.
> 
> To follow-up, this is a very rough outline of a proposal that I am going
> to submit to the Foundation in hopes to fund maintainer tool
> development. It follows along some of the lines highlighted in Dmitry's
> talk.
> 
> --------
> 
> # Stage 1 (Normal brain): "local patchwork"
> 
> - Implement a mutt-like tool ("putt"?) that uses locally cloned
>   public-inbox archives to track patches/series submitted to mailing
>   lists
>     - Pre-filters by keywords and paths in patches
>     - Tracks and automatically inserts taglines
>       (Reviewed-by, Acked-by, Tested-by)
>     - Can ignore a patch/series until it sees certain taglines
>       (Tested-by: zeroday bot, Reviewed-by: Trusty Intern)
>     - Automatically tracks latest series and offers an interdiff view
>       between series revisions ("show me what changed between v1 and v2")
>     - Allows responding to patches and conversations a-la mutt
>     - Allows applying patches/series to local repos

Do you plan for this tool to support shallow clones ? Some mailing lists
have really high traffic an have been around for years, one may not want
to clone a full public-inbox archives when interested in patches
submitted for the last N months only.

> # Stage 2 (Enlightened brain): "now with CI and workflows"
> 
> - Add configurable workflow functionality allowing maintainers to run
>   local or remote tasks on patches and series, before maintainer sees
>   the patches, e.g.:
>     - Create a branch and attempt to apply series
>     - If succeeds, run a batch of CI tests
>     - If succeeds, mark as "CI passed" and show the maintainer
>     - If fails, reject automatically using a "sorry, tests failed"
>       template, including relevant error messages
> 
> - All of the above runs outside of the UI tool ("putt-cid"?) and defines CI
>   routines that can run in cloudy environments or locally using
>   containers.
> - Putt communicates with putt-cid locally or remotely to identify
>   patches/series that the maintainer should review
> 
> 
> # Stage 3 (Galaxy brain): "email as a secondary channel"
> 
> - Support additional distributed communication mechanisms in conjunction
>   with existing mailing lists.
>   - SSB is a peer-to-peer replication framework that has built-in
>     cryptographic integrity and attestation ("immutable git-like
>     chains per participating developer")
>     - offers native support for structured data like bug reports, CI
>       results, code review comments, etc.
>     - can easily support email-to-SSB and web-to-SSB bridges, so
>       developers can choose to participate using familiar tools
>     - has known limitations in v1 of the protocol, but v2 is being
>       actively developed to address them.
>     - or we can take it as a base and develop an SSB-like protocol that
>       better suits distributed development needs.
> 
>   - Radicle is another interesting alternative that creates a mechanism
>     for automating some maintainer tasks by defining "state machines,"
>     e.g.:
>     - automatically merge a revision if all tests pass and at least 2
>       Reviewed-by's are seen.
>     - May have been sipping the blockchain cool-aid a bit too much
>       ("Immutable append-only records").

-- 
Regards,

Laurent Pinchart
_______________________________________________
Ksummit-discuss mailing list
Ksummit-discuss@lists.linuxfoundation.org
https://lists.linuxfoundation.org/mailman/listinfo/ksummit-discuss

Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] Reflections on kernel development processes

[Dmitry Vyukov]

From: Dmitry Vyukov <dvyukov@google.com>

On Thu, Sep 12, 2019 at 08:06:02AM -0400, Konstantin Ryabitsev wrote:
> 
> To follow-up, this is a very rough outline of a proposal that I am going
> to submit to the Foundation in hopes to fund maintainer tool
> development. It follows along some of the lines highlighted in Dmitry's
> talk.
> 
> --------
> 
> # Stage 1 (Normal brain): "local patchwork"
> 
> - Implement a mutt-like tool ("putt"?) that uses locally cloned
>   public-inbox archives to track patches/series submitted to mailing
>   lists
>     - Pre-filters by keywords and paths in patches
>     - Tracks and automatically inserts taglines
>       (Reviewed-by, Acked-by, Tested-by)
>     - Can ignore a patch/series until it sees certain taglines
>       (Tested-by: zeroday bot, Reviewed-by: Trusty Intern)
>     - Automatically tracks latest series and offers an interdiff view
>       between series revisions ("show me what changed between v1 and v2")
>     - Allows responding to patches and conversations a-la mutt
>     - Allows applying patches/series to local repos
> # Stage 2 (Enlightened brain): "now with CI and workflows"
> 
> - Add configurable workflow functionality allowing maintainers to run
>   local or remote tasks on patches and series, before maintainer sees
>   the patches, e.g.:
>     - Create a branch and attempt to apply series
>     - If succeeds, run a batch of CI tests
>     - If succeeds, mark as "CI passed" and show the maintainer
>     - If fails, reject automatically using a "sorry, tests failed"
>       template, including relevant error messages
> 
> - All of the above runs outside of the UI tool ("putt-cid"?) and defines CI
>   routines that can run in cloudy environments or locally using
>   containers.
> - Putt communicates with putt-cid locally or remotely to identify
>   patches/series that the maintainer should review
> 
> 
> # Stage 3 (Galaxy brain): "email as a secondary channel"
> 
> - Support additional distributed communication mechanisms in conjunction
>   with existing mailing lists.
>   - SSB is a peer-to-peer replication framework that has built-in
>     cryptographic integrity and attestation ("immutable git-like
>     chains per participating developer")
>     - offers native support for structured data like bug reports, CI
>       results, code review comments, etc.
>     - can easily support email-to-SSB and web-to-SSB bridges, so
>       developers can choose to participate using familiar tools
>     - has known limitations in v1 of the protocol, but v2 is being
>       actively developed to address them.
>     - or we can take it as a base and develop an SSB-like protocol that
>       better suits distributed development needs.
> 
>   - Radicle is another interesting alternative that creates a mechanism
>     for automating some maintainer tasks by defining "state machines,"
>     e.g.:
>     - automatically merge a revision if all tests pass and at least 2
>       Reviewed-by's are seen.
>     - May have been sipping the blockchain cool-aid a bit too much
>       ("Immutable append-only records").

Hi Konstantin,

Also adding people from the "Kernel development collaboration platform
wish list" discussion on the workflows list [1].
(Rafael et al, thanks for collecting the requirements, that's very useful!)

I second the idea expressed by several people that addressing the
contributor side is a very important part of this effort.

While I understand the intention to provide something useful as fast
as possible, I also a bit afraid that the Stage 1 ("putt") diverges
us into investing into particular UI, tying capabilities with this UI
and not addressing the fundamental problems.
People are expressing a preference for different types of UIs
(CL, terminal GUI, web, scripting), I think eventually we will have
several. So I would suggest to untie features/capabilities from
any particular UI as much as possible, and start with addressing more
fundamental aspects. Building richer features on top of the current
human-oriented emails is also going to be much harder, and that's the
work that we eventually will (hopefully) throw away.

From UI perspective I think we should start with a CL interface because
(1) it's the simplest to build (we don't invest too much into it,
don't shift focus and will shake down more important things faster),
(2) there are some important actions that are best done with CL
anyway (e.g. mailing a patch). Later it may serve as an
entry point for starting the richer terminal GUI or other types of GUIs.

There are 3 groups of people we should be looking at:
- contributors (important special case: sending first patch)
- maintainers
- reviewers

I would set the first milestone as having the CL utility (let's call
it "kit"*) which can do:

$ kit init
# Does some necessary one-time initialization, executed from the
# kernel git checkout.

$ kit mail
# Sends the top commit in the current branch for review.

So that would be the workflow for sending your first kernel patch.

Later "kit mail" can also run checkpatch, check SOB tag, add some kind
of change ID or anything else we will consider necessary. It may be
necessary to be able to force-override some of the checks, but by default
you are now getting patches that have SOB, checkpatch-clean, etc.

If there is an easy way to make it work with the current email-based
process (i.e. send email on your behalf and you receive incoming emails),
then we could do that first and give it to new developers to relief from
setting up email client. Otherwise, we should continue developing it
based on something like SSB (or whatever protocol we will choose).

Obviously, the intention is that if you do "kit mail" second time
with a changed patch, it sends "V2". Or if you have multiple local
commits it will properly mail the series (or V2 of the series).

Most (all) of the "kit" functionality should be separated from the UI
and be available for scripting/automation/other UIs. Whether it's
done as "libgit" or as "shell out" is discussable.

On the protocol side I don't have strong preference for SSB or
something similar custom. It seems that we will use SSB in somewhat
simplified way, i.e. global connected graph, rather than several large
groups or small isolated groups. We won't need Facebook-like following
nor Pubs announcements. You obviously don't want to be notified of all
messages in the system (LKML), but still it's a global graph in the
sense that you can receive anything if you want or CC anybody.
That limited subset of SSB should be easier to implement.
So as Konstantin said, we could fork SSB to better fit our needs.
The more important part will be the application-level protocol that
we will transfer inside of SSB messages, which is mostly transport
protocol for our needs (at least for the majority, maybe not for
Konstantin's concerns :)).

I would suggest to put bug/issue tracking aside for now (it's an
important thing, but it should be possible to add it later) and also
"bare chatting" (which can be done over emails for now) and
concentrate on patches just to make things simpler. Patches are
the cornerstone of the process.

So we need to define the format of the "patch for review" SSB message
which "kit mail" will form and push. It should be mostly easy
(patch itself, base revision, ID, CC, reference to previous version,
Fixes, etc). But there may be some more interesting aspects,
e.g. we will need some notion of "subsystems" for notifications,
some representation of comments on code and probably some other
things that I can't think of now.
Other developers will "reply" to the patch with "acked", "reviewed",
"merged", "review delegated" meta messages. Referring to the recent
"Notification of your branch being tested by zero day bot?"
discussion [2] CI systems will post "testing started" (with a link
to their status page or something), "testing finished" (with clear
OK/FAIL signal, and a link for FAIL).

If/when we have this, most of the mentioned features should be almost
trivial to implement. E.g. collecting all of Acked/Reviewed tags,
adding them and forming final patch; or showing version-to-version diff;
or doing "local patchwork" with nice features like "don't show it to
me if I already reviewed it"; or presenting "testing on CI X started
1 hour ago" when you are looking at a patch wondering about its status.
I guess generally you don't want this as a separate notification as long
as you can get access to this bit of info whenever you need to. This may
also be relevant for e.g. "don't notify me about Acked-by somebody
else if I am just a reviewer of the patch", instead we could deliver
Acked-by only to author and maintainer. Not saying that we should do
exactly this, just some examples of nice things that become very easy
to add for everybody (and very hard to add with emails).

The next important thing we will need is email bridge.
I see it as separate service that receives all SSB messages and e.g.
flattens "patch for review" message and sends as email. It will also
form "Acked-by" email from "acked" SSB message, etc. It will also
need to proxy incoming emails. In some cases it may be possible to
figure out the semantics of the email (e.g. only "Reviewed-by" line),
for other cases it probably should be injected as a "freeform comment"
message).
After sending a patch email, the bridge could send "email Message-ID/
lore link" SSB message for tracking purposes, which will link both
systems together.
This email bridge is also a nice point for opt-in for all optional
notifications. E.g. CIs always send "testing started" SSB message,
but for emails you can opt-in/out as you want.

It seems that all other services could operate in roughly the same way.
Namely, a CI system will receive push notifications about all patches,
inject "testing started" message back, then "testing finished" message
later. A new version of the patch can easily abort testing of the previous
version, or at least prevent notifications on the stale version.
A small thing Linus mentioned as annoying is getting "your patch broken"
notifications for patches known to be broken; this can easily be
addressed with a "don't bother testing" bit on the patch.

Similarly, a number of people mentioned that having all patches/series
in git would be very useful. So a git bridge could receive push
notifications about all patches, import them into some git tree
on kernel.org and inject a reply with git branch name back.

One requirement Konstantin mentioned is that it would be good if the
system will be able to operate in some kind of global doom scenarios
(e.g. a remote Linux code execution affecting all versions and being
actively exploited). From this point of view, I think it's important
that these bridges are separate from the core part, if any of these
goes down the system partially degrades but keeps core functions.

Regarding "state machines" in the protocol (Radicle/IPFS), I think
it's not just "sipping the blockchain cool-aid a bit too much",
it's a wrong tool for our needs. Smart contracts are used for
crypto-currencies where one does want to carve the rules in the
blockchain. But we don't want and don't need this.
The blockchain itself (passive data) can't merge changes, so we will
need some kind of active service for this. Now this active service
is also a good place to do the required checks (reviewed+tested).
So we do not need these rules in the blockchain itself.
We also don't want them to be carved because they may change.
Consider, you require "CI X to pass". Now CI X goes down and
the process stalls because this requirement is carved in stone.
What we would want to do instead is to change the service config
to ignore CI X for now.
Not saying that removing smart contracts from the protocol will
significantly simplify its design, requirements for formal verification,
number of tricky corner cases and general understandability.

Another important part of the system is user identities.
Do we go with a public/private key pair? Or we have some other realistic
alternatives? Assuming we go with key pairs for now, "kit init"
will generate a local key pair for you (a new developer). But a user
should be able to evacuate/export the private key later and pass
an existing key (to bootstrap a new machine with the same identity).
However, we will probably need another identity that is slightly
easier to remember and type in patch CC line than 256-char hash.
And that probably needs to be an email address (required for sending
email notifications anyway). But I don't know how to ensure uniqueness
of emails in this system. An alternative would be to use usernames
(e.g. "torvalds" or "tytso") and then a user can map that to own email
as they want. But this does not remove the requirement for uniqueness.

Two more interesting/controversial possibilities.
If we have an email bridge, we could also have a github bridge!
Don't get me wrong, I am not saying we need to do this now or at all.
I am saying that if UI part is abstracted enough, then it may be
theoretically possible to take a PR on a special dedicated github
project, convert it to "patch for review" SSB message and inject
into the system. Comments on the patch will be proxied back to github.
Andrew will receive this over email bridge and review and merge,
not even suspecting he is reviewing a github PR (w00t!).

Second controversial idea: the local rich GUI/patchwork is actually
web-based _but_ it talks to a local web server (so fast and no internet
connection required) _and_ it resembles terminal UI and has tons of
hotkeys and terminal-like navigation (so it kinda feels like terminal).
You start it with "kit gui" which starts a browser for you.
The advantage of this: we build 1 UI instead of 2, so immediate 2x
time savings. Also consistency between the UIs: you go to web, you see
exactly the same UI that you used to work with locally (now it's just
powered by a remote web server).

Phew! I think that's it. Does any of this make sense to you?
Thanks for your attention!

* "kit" is short and easy to remember, stands for "equipment/tool kit",
also refers to "git" with "k" for "kernel", or "kernel it" ("kernel thingy")

[1] https://lore.kernel.org/workflows/5072394.GngetUhsyG@kreacher/T/
[2] https://lore.kernel.org/workflows/20190919032100.GC7453@intel.com/T/

Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] Reflections on kernel development processes

[Daniel Borkmann]

On 9/22/19 2:02 PM, Dmitry Vyukov wrote:
[...]
> Also adding people from the "Kernel development collaboration platform
> wish list" discussion on the workflows list [1].
> (Rafael et al, thanks for collecting the requirements, that's very useful!)
> 
> I second the idea expressed by several people that addressing the
> contributor side is a very important part of this effort.
> 
> While I understand the intention to provide something useful as fast
> as possible, I also a bit afraid that the Stage 1 ("putt") diverges
> us into investing into particular UI, tying capabilities with this UI
> and not addressing the fundamental problems.
> People are expressing a preference for different types of UIs
> (CL, terminal GUI, web, scripting), I think eventually we will have
> several. So I would suggest to untie features/capabilities from
> any particular UI as much as possible, and start with addressing more
> fundamental aspects. Building richer features on top of the current
> human-oriented emails is also going to be much harder, and that's the
> work that we eventually will (hopefully) throw away.
> 
>  From UI perspective I think we should start with a CL interface because
> (1) it's the simplest to build (we don't invest too much into it,
> don't shift focus and will shake down more important things faster),
> (2) there are some important actions that are best done with CL
> anyway (e.g. mailing a patch). Later it may serve as an
> entry point for starting the richer terminal GUI or other types of GUIs.

+1, agree.

> There are 3 groups of people we should be looking at:
> - contributors (important special case: sending first patch)
> - maintainers
> - reviewers
> 
> I would set the first milestone as having the CL utility (let's call
> it "kit"*) which can do:
> 
> $ kit init
> # Does some necessary one-time initialization, executed from the
> # kernel git checkout.
> 
> $ kit mail
> # Sends the top commit in the current branch for review.
> 
> So that would be the workflow for sending your first kernel patch.
> 
> Later "kit mail" can also run checkpatch, check SOB tag, add some kind
> of change ID or anything else we will consider necessary. It may be
> necessary to be able to force-override some of the checks, but by default
> you are now getting patches that have SOB, checkpatch-clean, etc.
> 
> If there is an easy way to make it work with the current email-based
> process (i.e. send email on your behalf and you receive incoming emails),
> then we could do that first and give it to new developers to relief from
> setting up email client. Otherwise, we should continue developing it
> based on something like SSB (or whatever protocol we will choose).
> 
> Obviously, the intention is that if you do "kit mail" second time
> with a changed patch, it sends "V2". Or if you have multiple local
> commits it will properly mail the series (or V2 of the series).
> 
> Most (all) of the "kit" functionality should be separated from the UI
> and be available for scripting/automation/other UIs. Whether it's
> done as "libgit" or as "shell out" is discussable.
[...]
On that note, such a tool would also need to co-exist with the current
email based process for some (long?) time in order to allow a smooth
transition period. Last week I spent a few of nights hacking a small tool
which is regularly pulling the lore git trees I'm interested in and checking
out all [new] mails into maildir format so they can be read naturally by
UIs like mutt et al [0]. As an experiment, in case of bpf vger mailing list,
it extracts all current ~8k mails in under a second:

$ ./l2md
general.maildir = /home/foo/.l2md/maildir/common
general.period = 30
repos.bpf.maildir = /home/foo/.l2md/maildir/bpf
repos.bpf.initial_import = 0
repos.bpf.url = https://lore.kernel.org/bpf/0
repos.bpf.oid_maildir = [unknown]
Initial repository check.
Cloning: https://lore.kernel.org/bpf/0
Remote: Counting objects: 23859, done.
Remote: Compressing objects: 100% (14476/14476), done.
Remote: Total 23859 (delta 1561), reused 22125 (delta 1430)
Initial repository check completed.
Bootstrap done.
Resyncing maildirs.
Processed 7953 new mails for https://lore.kernel.org/bpf/0 in 0.586466s.
Sync done. Sleeping 30s.
Resyncing repositories.
Fetching: https://lore.kernel.org/bpf/0
Merging: refs/heads/master commit d608393d011aa0c0fc5983059052362cebeafc91
Resyncing maildirs.
Processed 0 new mails for https://lore.kernel.org/bpf/0 in 0.53611s.
[...]

I've started using it daily now and it appears to work nicely so far.
Given that, I'm wondering whether for a "kit" tool and beyond, we could use
something like lore git trees as a basis. The 'receive' side is already
there today, but I'm wondering if it's feasible to have a sendmail compatible
interface that would allow to push new "mails" into lore git trees as well,
where a bridge from lore git to vger list then distributes the message to
all email-based subscribers. Obviously this needs similar spam-filtering and
sanity checks as with vger lists, but eventually it wouldn't be any different
than anyone being able to send to vger via email. The nice thing would be
that the trees are mirrored on a wide basis already (and wouldn't need any
additional transport) and CI infrastructure can just pull them, work with
git sha ids, and reply with the results by pushing (implicitly by mentioned
sendmail compat tool, or "kit" etc). "kit" tool for contributors/reviewers
(I'd probably put both into the same bucket on a high level) and maintainers
could be built around the trees as foundation which could already solve the
issues around mail pointed out by Konstantin some time ago [1] while allowing
max compatibility to current workflows as a transition period. Just a thought.

Cheers,
Daniel

   [0] https://git.kernel.org/pub/scm/linux/kernel/git/dborkman/l2md.git/
   [1] https://people.kernel.org/monsieuricon/patches-carved-into-developer-sigchains
_______________________________________________
Ksummit-discuss mailing list
Ksummit-discuss@lists.linuxfoundation.org
https://lists.linuxfoundation.org/mailman/listinfo/ksummit-discuss

Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] Reflections on kernel development processes

[Daniel Borkmann]

On 9/22/19 2:02 PM, Dmitry Vyukov wrote:
[...]
> Also adding people from the "Kernel development collaboration platform
> wish list" discussion on the workflows list [1].
> (Rafael et al, thanks for collecting the requirements, that's very useful!)
> 
> I second the idea expressed by several people that addressing the
> contributor side is a very important part of this effort.
> 
> While I understand the intention to provide something useful as fast
> as possible, I also a bit afraid that the Stage 1 ("putt") diverges
> us into investing into particular UI, tying capabilities with this UI
> and not addressing the fundamental problems.
> People are expressing a preference for different types of UIs
> (CL, terminal GUI, web, scripting), I think eventually we will have
> several. So I would suggest to untie features/capabilities from
> any particular UI as much as possible, and start with addressing more
> fundamental aspects. Building richer features on top of the current
> human-oriented emails is also going to be much harder, and that's the
> work that we eventually will (hopefully) throw away.
> 
>  From UI perspective I think we should start with a CL interface because
> (1) it's the simplest to build (we don't invest too much into it,
> don't shift focus and will shake down more important things faster),
> (2) there are some important actions that are best done with CL
> anyway (e.g. mailing a patch). Later it may serve as an
> entry point for starting the richer terminal GUI or other types of GUIs.

+1, agree.

> There are 3 groups of people we should be looking at:
> - contributors (important special case: sending first patch)
> - maintainers
> - reviewers
> 
> I would set the first milestone as having the CL utility (let's call
> it "kit"*) which can do:
> 
> $ kit init
> # Does some necessary one-time initialization, executed from the
> # kernel git checkout.
> 
> $ kit mail
> # Sends the top commit in the current branch for review.
> 
> So that would be the workflow for sending your first kernel patch.
> 
> Later "kit mail" can also run checkpatch, check SOB tag, add some kind
> of change ID or anything else we will consider necessary. It may be
> necessary to be able to force-override some of the checks, but by default
> you are now getting patches that have SOB, checkpatch-clean, etc.
> 
> If there is an easy way to make it work with the current email-based
> process (i.e. send email on your behalf and you receive incoming emails),
> then we could do that first and give it to new developers to relief from
> setting up email client. Otherwise, we should continue developing it
> based on something like SSB (or whatever protocol we will choose).
> 
> Obviously, the intention is that if you do "kit mail" second time
> with a changed patch, it sends "V2". Or if you have multiple local
> commits it will properly mail the series (or V2 of the series).
> 
> Most (all) of the "kit" functionality should be separated from the UI
> and be available for scripting/automation/other UIs. Whether it's
> done as "libgit" or as "shell out" is discussable.
[...]
On that note, such a tool would also need to co-exist with the current
email based process for some (long?) time in order to allow a smooth
transition period. Last week I spent a few of nights hacking a small tool
which is regularly pulling the lore git trees I'm interested in and checking
out all [new] mails into maildir format so they can be read naturally by
UIs like mutt et al [0]. As an experiment, in case of bpf vger mailing list,
it extracts all current ~8k mails in under a second:

$ ./l2md
general.maildir = /home/foo/.l2md/maildir/common
general.period = 30
repos.bpf.maildir = /home/foo/.l2md/maildir/bpf
repos.bpf.initial_import = 0
repos.bpf.url = https://lore.kernel.org/bpf/0
repos.bpf.oid_maildir = [unknown]
Initial repository check.
Cloning: https://lore.kernel.org/bpf/0
Remote: Counting objects: 23859, done.
Remote: Compressing objects: 100% (14476/14476), done.
Remote: Total 23859 (delta 1561), reused 22125 (delta 1430)
Initial repository check completed.
Bootstrap done.
Resyncing maildirs.
Processed 7953 new mails for https://lore.kernel.org/bpf/0 in 0.586466s.
Sync done. Sleeping 30s.
Resyncing repositories.
Fetching: https://lore.kernel.org/bpf/0
Merging: refs/heads/master commit d608393d011aa0c0fc5983059052362cebeafc91
Resyncing maildirs.
Processed 0 new mails for https://lore.kernel.org/bpf/0 in 0.53611s.
[...]

I've started using it daily now and it appears to work nicely so far.
Given that, I'm wondering whether for a "kit" tool and beyond, we could use
something like lore git trees as a basis. The 'receive' side is already
there today, but I'm wondering if it's feasible to have a sendmail compatible
interface that would allow to push new "mails" into lore git trees as well,
where a bridge from lore git to vger list then distributes the message to
all email-based subscribers. Obviously this needs similar spam-filtering and
sanity checks as with vger lists, but eventually it wouldn't be any different
than anyone being able to send to vger via email. The nice thing would be
that the trees are mirrored on a wide basis already (and wouldn't need any
additional transport) and CI infrastructure can just pull them, work with
git sha ids, and reply with the results by pushing (implicitly by mentioned
sendmail compat tool, or "kit" etc). "kit" tool for contributors/reviewers
(I'd probably put both into the same bucket on a high level) and maintainers
could be built around the trees as foundation which could already solve the
issues around mail pointed out by Konstantin some time ago [1] while allowing
max compatibility to current workflows as a transition period. Just a thought.

Cheers,
Daniel

   [0] https://git.kernel.org/pub/scm/linux/kernel/git/dborkman/l2md.git/
   [1] https://people.kernel.org/monsieuricon/patches-carved-into-developer-sigchains

Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] Reflections on kernel development processes

[Dmitry Vyukov via Ksummit-discuss]

On Mon, Sep 23, 2019 at 2:52 PM Daniel Borkmann <daniel@iogearbox.net> wrote:
>
> On 9/22/19 2:02 PM, Dmitry Vyukov wrote:
> [...]
> > Also adding people from the "Kernel development collaboration platform
> > wish list" discussion on the workflows list [1].
> > (Rafael et al, thanks for collecting the requirements, that's very useful!)
> >
> > I second the idea expressed by several people that addressing the
> > contributor side is a very important part of this effort.
> >
> > While I understand the intention to provide something useful as fast
> > as possible, I also a bit afraid that the Stage 1 ("putt") diverges
> > us into investing into particular UI, tying capabilities with this UI
> > and not addressing the fundamental problems.
> > People are expressing a preference for different types of UIs
> > (CL, terminal GUI, web, scripting), I think eventually we will have
> > several. So I would suggest to untie features/capabilities from
> > any particular UI as much as possible, and start with addressing more
> > fundamental aspects. Building richer features on top of the current
> > human-oriented emails is also going to be much harder, and that's the
> > work that we eventually will (hopefully) throw away.
> >
> >  From UI perspective I think we should start with a CL interface because
> > (1) it's the simplest to build (we don't invest too much into it,
> > don't shift focus and will shake down more important things faster),
> > (2) there are some important actions that are best done with CL
> > anyway (e.g. mailing a patch). Later it may serve as an
> > entry point for starting the richer terminal GUI or other types of GUIs.
>
> +1, agree.
>
> > There are 3 groups of people we should be looking at:
> > - contributors (important special case: sending first patch)
> > - maintainers
> > - reviewers
> >
> > I would set the first milestone as having the CL utility (let's call
> > it "kit"*) which can do:
> >
> > $ kit init
> > # Does some necessary one-time initialization, executed from the
> > # kernel git checkout.
> >
> > $ kit mail
> > # Sends the top commit in the current branch for review.
> >
> > So that would be the workflow for sending your first kernel patch.
> >
> > Later "kit mail" can also run checkpatch, check SOB tag, add some kind
> > of change ID or anything else we will consider necessary. It may be
> > necessary to be able to force-override some of the checks, but by default
> > you are now getting patches that have SOB, checkpatch-clean, etc.
> >
> > If there is an easy way to make it work with the current email-based
> > process (i.e. send email on your behalf and you receive incoming emails),
> > then we could do that first and give it to new developers to relief from
> > setting up email client. Otherwise, we should continue developing it
> > based on something like SSB (or whatever protocol we will choose).
> >
> > Obviously, the intention is that if you do "kit mail" second time
> > with a changed patch, it sends "V2". Or if you have multiple local
> > commits it will properly mail the series (or V2 of the series).
> >
> > Most (all) of the "kit" functionality should be separated from the UI
> > and be available for scripting/automation/other UIs. Whether it's
> > done as "libgit" or as "shell out" is discussable.
> [...]
> On that note, such a tool would also need to co-exist with the current
> email based process for some (long?) time in order to allow a smooth
> transition period. Last week I spent a few of nights hacking a small tool
> which is regularly pulling the lore git trees I'm interested in and checking
> out all [new] mails into maildir format so they can be read naturally by
> UIs like mutt et al [0]. As an experiment, in case of bpf vger mailing list,
> it extracts all current ~8k mails in under a second:
>
> $ ./l2md
> general.maildir = /home/foo/.l2md/maildir/common
> general.period = 30
> repos.bpf.maildir = /home/foo/.l2md/maildir/bpf
> repos.bpf.initial_import = 0
> repos.bpf.url = https://lore.kernel.org/bpf/0
> repos.bpf.oid_maildir = [unknown]
> Initial repository check.
> Cloning: https://lore.kernel.org/bpf/0
> Remote: Counting objects: 23859, done.
> Remote: Compressing objects: 100% (14476/14476), done.
> Remote: Total 23859 (delta 1561), reused 22125 (delta 1430)
> Initial repository check completed.
> Bootstrap done.
> Resyncing maildirs.
> Processed 7953 new mails for https://lore.kernel.org/bpf/0 in 0.586466s.
> Sync done. Sleeping 30s.
> Resyncing repositories.
> Fetching: https://lore.kernel.org/bpf/0
> Merging: refs/heads/master commit d608393d011aa0c0fc5983059052362cebeafc91
> Resyncing maildirs.
> Processed 0 new mails for https://lore.kernel.org/bpf/0 in 0.53611s.
> [...]
>
> I've started using it daily now and it appears to work nicely so far.
> Given that, I'm wondering whether for a "kit" tool and beyond, we could use
> something like lore git trees as a basis. The 'receive' side is already
> there today, but I'm wondering if it's feasible to have a sendmail compatible
> interface that would allow to push new "mails" into lore git trees as well,
> where a bridge from lore git to vger list then distributes the message to
> all email-based subscribers. Obviously this needs similar spam-filtering and
> sanity checks as with vger lists, but eventually it wouldn't be any different
> than anyone being able to send to vger via email. The nice thing would be
> that the trees are mirrored on a wide basis already (and wouldn't need any
> additional transport) and CI infrastructure can just pull them, work with
> git sha ids, and reply with the results by pushing (implicitly by mentioned
> sendmail compat tool, or "kit" etc). "kit" tool for contributors/reviewers
> (I'd probably put both into the same bucket on a high level) and maintainers
> could be built around the trees as foundation which could already solve the
> issues around mail pointed out by Konstantin some time ago [1] while allowing
> max compatibility to current workflows as a transition period. Just a thought.

Hi Daniel,

Do I understand correctly that you propose to use git as a transport
layer to store some structured data about changes? This was mentioned
during the maintainers summit [1], see "Git as the transport layer"
section.

Using git is nice in the sense that we could reuse lots of existing
infrastructure/code/tools rather than invent our own. And I think in
the end the exact transport layer does not matter much (for users) as
long as we can build the same interface on top of it.

However, Konstantin mentioned potential scalability problem with git
in such scenario. It wasn't supposed to store "journal" type info, but
rather small diffs to a code base. If we are considering such option
for real, I guess we just need to evaluate it in such scenario and
check how well/bad it works. We could also try to optimize git for
such a scenario, maybe there is a single bottleneck that we could
optimize...

There will probably also be some implications on possibility of local peering.

If everybody will be able to push into the single git, we will need to
figure out user auth story and single user screwing the whole thing
intentionally or by accident.

[1] https://lwn.net/Articles/799134/
_______________________________________________
Ksummit-discuss mailing list
Ksummit-discuss@lists.linuxfoundation.org
https://lists.linuxfoundation.org/mailman/listinfo/ksummit-discuss

Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] Reflections on kernel development processes

[Dmitry Vyukov]

On Mon, Sep 23, 2019 at 2:52 PM Daniel Borkmann <daniel@iogearbox.net> wrote:
>
> On 9/22/19 2:02 PM, Dmitry Vyukov wrote:
> [...]
> > Also adding people from the "Kernel development collaboration platform
> > wish list" discussion on the workflows list [1].
> > (Rafael et al, thanks for collecting the requirements, that's very useful!)
> >
> > I second the idea expressed by several people that addressing the
> > contributor side is a very important part of this effort.
> >
> > While I understand the intention to provide something useful as fast
> > as possible, I also a bit afraid that the Stage 1 ("putt") diverges
> > us into investing into particular UI, tying capabilities with this UI
> > and not addressing the fundamental problems.
> > People are expressing a preference for different types of UIs
> > (CL, terminal GUI, web, scripting), I think eventually we will have
> > several. So I would suggest to untie features/capabilities from
> > any particular UI as much as possible, and start with addressing more
> > fundamental aspects. Building richer features on top of the current
> > human-oriented emails is also going to be much harder, and that's the
> > work that we eventually will (hopefully) throw away.
> >
> >  From UI perspective I think we should start with a CL interface because
> > (1) it's the simplest to build (we don't invest too much into it,
> > don't shift focus and will shake down more important things faster),
> > (2) there are some important actions that are best done with CL
> > anyway (e.g. mailing a patch). Later it may serve as an
> > entry point for starting the richer terminal GUI or other types of GUIs.
>
> +1, agree.
>
> > There are 3 groups of people we should be looking at:
> > - contributors (important special case: sending first patch)
> > - maintainers
> > - reviewers
> >
> > I would set the first milestone as having the CL utility (let's call
> > it "kit"*) which can do:
> >
> > $ kit init
> > # Does some necessary one-time initialization, executed from the
> > # kernel git checkout.
> >
> > $ kit mail
> > # Sends the top commit in the current branch for review.
> >
> > So that would be the workflow for sending your first kernel patch.
> >
> > Later "kit mail" can also run checkpatch, check SOB tag, add some kind
> > of change ID or anything else we will consider necessary. It may be
> > necessary to be able to force-override some of the checks, but by default
> > you are now getting patches that have SOB, checkpatch-clean, etc.
> >
> > If there is an easy way to make it work with the current email-based
> > process (i.e. send email on your behalf and you receive incoming emails),
> > then we could do that first and give it to new developers to relief from
> > setting up email client. Otherwise, we should continue developing it
> > based on something like SSB (or whatever protocol we will choose).
> >
> > Obviously, the intention is that if you do "kit mail" second time
> > with a changed patch, it sends "V2". Or if you have multiple local
> > commits it will properly mail the series (or V2 of the series).
> >
> > Most (all) of the "kit" functionality should be separated from the UI
> > and be available for scripting/automation/other UIs. Whether it's
> > done as "libgit" or as "shell out" is discussable.
> [...]
> On that note, such a tool would also need to co-exist with the current
> email based process for some (long?) time in order to allow a smooth
> transition period. Last week I spent a few of nights hacking a small tool
> which is regularly pulling the lore git trees I'm interested in and checking
> out all [new] mails into maildir format so they can be read naturally by
> UIs like mutt et al [0]. As an experiment, in case of bpf vger mailing list,
> it extracts all current ~8k mails in under a second:
>
> $ ./l2md
> general.maildir = /home/foo/.l2md/maildir/common
> general.period = 30
> repos.bpf.maildir = /home/foo/.l2md/maildir/bpf
> repos.bpf.initial_import = 0
> repos.bpf.url = https://lore.kernel.org/bpf/0
> repos.bpf.oid_maildir = [unknown]
> Initial repository check.
> Cloning: https://lore.kernel.org/bpf/0
> Remote: Counting objects: 23859, done.
> Remote: Compressing objects: 100% (14476/14476), done.
> Remote: Total 23859 (delta 1561), reused 22125 (delta 1430)
> Initial repository check completed.
> Bootstrap done.
> Resyncing maildirs.
> Processed 7953 new mails for https://lore.kernel.org/bpf/0 in 0.586466s.
> Sync done. Sleeping 30s.
> Resyncing repositories.
> Fetching: https://lore.kernel.org/bpf/0
> Merging: refs/heads/master commit d608393d011aa0c0fc5983059052362cebeafc91
> Resyncing maildirs.
> Processed 0 new mails for https://lore.kernel.org/bpf/0 in 0.53611s.
> [...]
>
> I've started using it daily now and it appears to work nicely so far.
> Given that, I'm wondering whether for a "kit" tool and beyond, we could use
> something like lore git trees as a basis. The 'receive' side is already
> there today, but I'm wondering if it's feasible to have a sendmail compatible
> interface that would allow to push new "mails" into lore git trees as well,
> where a bridge from lore git to vger list then distributes the message to
> all email-based subscribers. Obviously this needs similar spam-filtering and
> sanity checks as with vger lists, but eventually it wouldn't be any different
> than anyone being able to send to vger via email. The nice thing would be
> that the trees are mirrored on a wide basis already (and wouldn't need any
> additional transport) and CI infrastructure can just pull them, work with
> git sha ids, and reply with the results by pushing (implicitly by mentioned
> sendmail compat tool, or "kit" etc). "kit" tool for contributors/reviewers
> (I'd probably put both into the same bucket on a high level) and maintainers
> could be built around the trees as foundation which could already solve the
> issues around mail pointed out by Konstantin some time ago [1] while allowing
> max compatibility to current workflows as a transition period. Just a thought.

Hi Daniel,

Do I understand correctly that you propose to use git as a transport
layer to store some structured data about changes? This was mentioned
during the maintainers summit [1], see "Git as the transport layer"
section.

Using git is nice in the sense that we could reuse lots of existing
infrastructure/code/tools rather than invent our own. And I think in
the end the exact transport layer does not matter much (for users) as
long as we can build the same interface on top of it.

However, Konstantin mentioned potential scalability problem with git
in such scenario. It wasn't supposed to store "journal" type info, but
rather small diffs to a code base. If we are considering such option
for real, I guess we just need to evaluate it in such scenario and
check how well/bad it works. We could also try to optimize git for
such a scenario, maybe there is a single bottleneck that we could
optimize...

There will probably also be some implications on possibility of local peering.

If everybody will be able to push into the single git, we will need to
figure out user auth story and single user screwing the whole thing
intentionally or by accident.

[1] https://lwn.net/Articles/799134/

Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] Reflections on kernel development processes

[Daniel Borkmann]

On 9/23/19 4:08 PM, Dmitry Vyukov wrote:
> On Mon, Sep 23, 2019 at 2:52 PM Daniel Borkmann <daniel@iogearbox.net> wrote:
>> On 9/22/19 2:02 PM, Dmitry Vyukov wrote:
>> [...]
>>> Also adding people from the "Kernel development collaboration platform
>>> wish list" discussion on the workflows list [1].
>>> (Rafael et al, thanks for collecting the requirements, that's very useful!)
>>>
>>> I second the idea expressed by several people that addressing the
>>> contributor side is a very important part of this effort.
>>>
>>> While I understand the intention to provide something useful as fast
>>> as possible, I also a bit afraid that the Stage 1 ("putt") diverges
>>> us into investing into particular UI, tying capabilities with this UI
>>> and not addressing the fundamental problems.
>>> People are expressing a preference for different types of UIs
>>> (CL, terminal GUI, web, scripting), I think eventually we will have
>>> several. So I would suggest to untie features/capabilities from
>>> any particular UI as much as possible, and start with addressing more
>>> fundamental aspects. Building richer features on top of the current
>>> human-oriented emails is also going to be much harder, and that's the
>>> work that we eventually will (hopefully) throw away.
>>>
>>>   From UI perspective I think we should start with a CL interface because
>>> (1) it's the simplest to build (we don't invest too much into it,
>>> don't shift focus and will shake down more important things faster),
>>> (2) there are some important actions that are best done with CL
>>> anyway (e.g. mailing a patch). Later it may serve as an
>>> entry point for starting the richer terminal GUI or other types of GUIs.
>>
>> +1, agree.
>>
>>> There are 3 groups of people we should be looking at:
>>> - contributors (important special case: sending first patch)
>>> - maintainers
>>> - reviewers
>>>
>>> I would set the first milestone as having the CL utility (let's call
>>> it "kit"*) which can do:
>>>
>>> $ kit init
>>> # Does some necessary one-time initialization, executed from the
>>> # kernel git checkout.
>>>
>>> $ kit mail
>>> # Sends the top commit in the current branch for review.
>>>
>>> So that would be the workflow for sending your first kernel patch.
>>>
>>> Later "kit mail" can also run checkpatch, check SOB tag, add some kind
>>> of change ID or anything else we will consider necessary. It may be
>>> necessary to be able to force-override some of the checks, but by default
>>> you are now getting patches that have SOB, checkpatch-clean, etc.
>>>
>>> If there is an easy way to make it work with the current email-based
>>> process (i.e. send email on your behalf and you receive incoming emails),
>>> then we could do that first and give it to new developers to relief from
>>> setting up email client. Otherwise, we should continue developing it
>>> based on something like SSB (or whatever protocol we will choose).
>>>
>>> Obviously, the intention is that if you do "kit mail" second time
>>> with a changed patch, it sends "V2". Or if you have multiple local
>>> commits it will properly mail the series (or V2 of the series).
>>>
>>> Most (all) of the "kit" functionality should be separated from the UI
>>> and be available for scripting/automation/other UIs. Whether it's
>>> done as "libgit" or as "shell out" is discussable.
>> [...]
>> On that note, such a tool would also need to co-exist with the current
>> email based process for some (long?) time in order to allow a smooth
>> transition period. Last week I spent a few of nights hacking a small tool
>> which is regularly pulling the lore git trees I'm interested in and checking
>> out all [new] mails into maildir format so they can be read naturally by
>> UIs like mutt et al [0]. As an experiment, in case of bpf vger mailing list,
>> it extracts all current ~8k mails in under a second:
>>
>> $ ./l2md
>> general.maildir = /home/foo/.l2md/maildir/common
>> general.period = 30
>> repos.bpf.maildir = /home/foo/.l2md/maildir/bpf
>> repos.bpf.initial_import = 0
>> repos.bpf.url = https://lore.kernel.org/bpf/0
>> repos.bpf.oid_maildir = [unknown]
>> Initial repository check.
>> Cloning: https://lore.kernel.org/bpf/0
>> Remote: Counting objects: 23859, done.
>> Remote: Compressing objects: 100% (14476/14476), done.
>> Remote: Total 23859 (delta 1561), reused 22125 (delta 1430)
>> Initial repository check completed.
>> Bootstrap done.
>> Resyncing maildirs.
>> Processed 7953 new mails for https://lore.kernel.org/bpf/0 in 0.586466s.
>> Sync done. Sleeping 30s.
>> Resyncing repositories.
>> Fetching: https://lore.kernel.org/bpf/0
>> Merging: refs/heads/master commit d608393d011aa0c0fc5983059052362cebeafc91
>> Resyncing maildirs.
>> Processed 0 new mails for https://lore.kernel.org/bpf/0 in 0.53611s.
>> [...]
>>
>> I've started using it daily now and it appears to work nicely so far.
>> Given that, I'm wondering whether for a "kit" tool and beyond, we could use
>> something like lore git trees as a basis. The 'receive' side is already
>> there today, but I'm wondering if it's feasible to have a sendmail compatible
>> interface that would allow to push new "mails" into lore git trees as well,
>> where a bridge from lore git to vger list then distributes the message to
>> all email-based subscribers. Obviously this needs similar spam-filtering and
>> sanity checks as with vger lists, but eventually it wouldn't be any different
>> than anyone being able to send to vger via email. The nice thing would be
>> that the trees are mirrored on a wide basis already (and wouldn't need any
>> additional transport) and CI infrastructure can just pull them, work with
>> git sha ids, and reply with the results by pushing (implicitly by mentioned
>> sendmail compat tool, or "kit" etc). "kit" tool for contributors/reviewers
>> (I'd probably put both into the same bucket on a high level) and maintainers
>> could be built around the trees as foundation which could already solve the
>> issues around mail pointed out by Konstantin some time ago [1] while allowing
>> max compatibility to current workflows as a transition period. Just a thought.
> 
> Hi Daniel,
> 
> Do I understand correctly that you propose to use git as a transport
> layer to store some structured data about changes? This was mentioned
> during the maintainers summit [1], see "Git as the transport layer"
> section.
> 
> Using git is nice in the sense that we could reuse lots of existing
> infrastructure/code/tools rather than invent our own. And I think in
> the end the exact transport layer does not matter much (for users) as
> long as we can build the same interface on top of it.
> 
> However, Konstantin mentioned potential scalability problem with git
> in such scenario. It wasn't supposed to store "journal" type info, but
> rather small diffs to a code base. If we are considering such option
> for real, I guess we just need to evaluate it in such scenario and
> check how well/bad it works. We could also try to optimize git for
> such a scenario, maybe there is a single bottleneck that we could
> optimize...
>
> There will probably also be some implications on possibility of local peering.
> 
> If everybody will be able to push into the single git, we will need to
> figure out user auth story and single user screwing the whole thing
> intentionally or by accident.

Yep all true. The way lore git works is that there's a single file called 'm'
in the repo which always contains the top most mail from the list [0], and [1]
contains basically the diff to the previous mail from the list. Extracting the
actual git object blobs is sufficiently fast (imho, see above and [2]), and
basically the receive side integration into a mail client is already there with
a mechanism like l2md tool has. A push mechanism would need more server side
logic that ensures to not screw up the repo, never allow force push, and always
have the pushed 'm' file(s) conflict-free at the top of the tree (along with the
usual mail/spam filtering there is in place on vger). The way 'm' is handled
today may not be nicest or most natural way for git indeed, and despite that
it may seem more of an implementation detail at this point I thought I mention
it here along with l2md since it can be used today for reading and it feels the
'push' side may not be too far off to realize. It wouldn't solve all the other
discussed items from the wish list you/others brought up in your original thread
(which would need to work on top of the transport), but could be an alternative
way with regards to the email problem.

Thanks,
Daniel

  [0] https://git.kernel.org/pub/scm/public-inbox/vger.kernel.org/bpf/0.git/tree/m
  [1] https://git.kernel.org/pub/scm/public-inbox/vger.kernel.org/bpf/0.git/commit/?id=0c0e0a248293aa382c3451680e696bbeaf38c47e
  [2] Converting the entire netdev history (if anyone ever needs it in their mail client)
      with l2md from 03/2002 till 09/2019 takes on my laptop:
      [...]
      Processed 137396 new mails for https://lore.kernel.org/netdev/1 in 11.198241s.
      Processed 498001 new mails for https://lore.kernel.org/netdev/0 in 41.414329s.
      [...]

> [1] https://lwn.net/Articles/799134/
_______________________________________________
Ksummit-discuss mailing list
Ksummit-discuss@lists.linuxfoundation.org
https://lists.linuxfoundation.org/mailman/listinfo/ksummit-discuss

Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] Reflections on kernel development processes

[Daniel Borkmann]

On 9/23/19 4:08 PM, Dmitry Vyukov wrote:
> On Mon, Sep 23, 2019 at 2:52 PM Daniel Borkmann <daniel@iogearbox.net> wrote:
>> On 9/22/19 2:02 PM, Dmitry Vyukov wrote:
>> [...]
>>> Also adding people from the "Kernel development collaboration platform
>>> wish list" discussion on the workflows list [1].
>>> (Rafael et al, thanks for collecting the requirements, that's very useful!)
>>>
>>> I second the idea expressed by several people that addressing the
>>> contributor side is a very important part of this effort.
>>>
>>> While I understand the intention to provide something useful as fast
>>> as possible, I also a bit afraid that the Stage 1 ("putt") diverges
>>> us into investing into particular UI, tying capabilities with this UI
>>> and not addressing the fundamental problems.
>>> People are expressing a preference for different types of UIs
>>> (CL, terminal GUI, web, scripting), I think eventually we will have
>>> several. So I would suggest to untie features/capabilities from
>>> any particular UI as much as possible, and start with addressing more
>>> fundamental aspects. Building richer features on top of the current
>>> human-oriented emails is also going to be much harder, and that's the
>>> work that we eventually will (hopefully) throw away.
>>>
>>>   From UI perspective I think we should start with a CL interface because
>>> (1) it's the simplest to build (we don't invest too much into it,
>>> don't shift focus and will shake down more important things faster),
>>> (2) there are some important actions that are best done with CL
>>> anyway (e.g. mailing a patch). Later it may serve as an
>>> entry point for starting the richer terminal GUI or other types of GUIs.
>>
>> +1, agree.
>>
>>> There are 3 groups of people we should be looking at:
>>> - contributors (important special case: sending first patch)
>>> - maintainers
>>> - reviewers
>>>
>>> I would set the first milestone as having the CL utility (let's call
>>> it "kit"*) which can do:
>>>
>>> $ kit init
>>> # Does some necessary one-time initialization, executed from the
>>> # kernel git checkout.
>>>
>>> $ kit mail
>>> # Sends the top commit in the current branch for review.
>>>
>>> So that would be the workflow for sending your first kernel patch.
>>>
>>> Later "kit mail" can also run checkpatch, check SOB tag, add some kind
>>> of change ID or anything else we will consider necessary. It may be
>>> necessary to be able to force-override some of the checks, but by default
>>> you are now getting patches that have SOB, checkpatch-clean, etc.
>>>
>>> If there is an easy way to make it work with the current email-based
>>> process (i.e. send email on your behalf and you receive incoming emails),
>>> then we could do that first and give it to new developers to relief from
>>> setting up email client. Otherwise, we should continue developing it
>>> based on something like SSB (or whatever protocol we will choose).
>>>
>>> Obviously, the intention is that if you do "kit mail" second time
>>> with a changed patch, it sends "V2". Or if you have multiple local
>>> commits it will properly mail the series (or V2 of the series).
>>>
>>> Most (all) of the "kit" functionality should be separated from the UI
>>> and be available for scripting/automation/other UIs. Whether it's
>>> done as "libgit" or as "shell out" is discussable.
>> [...]
>> On that note, such a tool would also need to co-exist with the current
>> email based process for some (long?) time in order to allow a smooth
>> transition period. Last week I spent a few of nights hacking a small tool
>> which is regularly pulling the lore git trees I'm interested in and checking
>> out all [new] mails into maildir format so they can be read naturally by
>> UIs like mutt et al [0]. As an experiment, in case of bpf vger mailing list,
>> it extracts all current ~8k mails in under a second:
>>
>> $ ./l2md
>> general.maildir = /home/foo/.l2md/maildir/common
>> general.period = 30
>> repos.bpf.maildir = /home/foo/.l2md/maildir/bpf
>> repos.bpf.initial_import = 0
>> repos.bpf.url = https://lore.kernel.org/bpf/0
>> repos.bpf.oid_maildir = [unknown]
>> Initial repository check.
>> Cloning: https://lore.kernel.org/bpf/0
>> Remote: Counting objects: 23859, done.
>> Remote: Compressing objects: 100% (14476/14476), done.
>> Remote: Total 23859 (delta 1561), reused 22125 (delta 1430)
>> Initial repository check completed.
>> Bootstrap done.
>> Resyncing maildirs.
>> Processed 7953 new mails for https://lore.kernel.org/bpf/0 in 0.586466s.
>> Sync done. Sleeping 30s.
>> Resyncing repositories.
>> Fetching: https://lore.kernel.org/bpf/0
>> Merging: refs/heads/master commit d608393d011aa0c0fc5983059052362cebeafc91
>> Resyncing maildirs.
>> Processed 0 new mails for https://lore.kernel.org/bpf/0 in 0.53611s.
>> [...]
>>
>> I've started using it daily now and it appears to work nicely so far.
>> Given that, I'm wondering whether for a "kit" tool and beyond, we could use
>> something like lore git trees as a basis. The 'receive' side is already
>> there today, but I'm wondering if it's feasible to have a sendmail compatible
>> interface that would allow to push new "mails" into lore git trees as well,
>> where a bridge from lore git to vger list then distributes the message to
>> all email-based subscribers. Obviously this needs similar spam-filtering and
>> sanity checks as with vger lists, but eventually it wouldn't be any different
>> than anyone being able to send to vger via email. The nice thing would be
>> that the trees are mirrored on a wide basis already (and wouldn't need any
>> additional transport) and CI infrastructure can just pull them, work with
>> git sha ids, and reply with the results by pushing (implicitly by mentioned
>> sendmail compat tool, or "kit" etc). "kit" tool for contributors/reviewers
>> (I'd probably put both into the same bucket on a high level) and maintainers
>> could be built around the trees as foundation which could already solve the
>> issues around mail pointed out by Konstantin some time ago [1] while allowing
>> max compatibility to current workflows as a transition period. Just a thought.
> 
> Hi Daniel,
> 
> Do I understand correctly that you propose to use git as a transport
> layer to store some structured data about changes? This was mentioned
> during the maintainers summit [1], see "Git as the transport layer"
> section.
> 
> Using git is nice in the sense that we could reuse lots of existing
> infrastructure/code/tools rather than invent our own. And I think in
> the end the exact transport layer does not matter much (for users) as
> long as we can build the same interface on top of it.
> 
> However, Konstantin mentioned potential scalability problem with git
> in such scenario. It wasn't supposed to store "journal" type info, but
> rather small diffs to a code base. If we are considering such option
> for real, I guess we just need to evaluate it in such scenario and
> check how well/bad it works. We could also try to optimize git for
> such a scenario, maybe there is a single bottleneck that we could
> optimize...
>
> There will probably also be some implications on possibility of local peering.
> 
> If everybody will be able to push into the single git, we will need to
> figure out user auth story and single user screwing the whole thing
> intentionally or by accident.

Yep all true. The way lore git works is that there's a single file called 'm'
in the repo which always contains the top most mail from the list [0], and [1]
contains basically the diff to the previous mail from the list. Extracting the
actual git object blobs is sufficiently fast (imho, see above and [2]), and
basically the receive side integration into a mail client is already there with
a mechanism like l2md tool has. A push mechanism would need more server side
logic that ensures to not screw up the repo, never allow force push, and always
have the pushed 'm' file(s) conflict-free at the top of the tree (along with the
usual mail/spam filtering there is in place on vger). The way 'm' is handled
today may not be nicest or most natural way for git indeed, and despite that
it may seem more of an implementation detail at this point I thought I mention
it here along with l2md since it can be used today for reading and it feels the
'push' side may not be too far off to realize. It wouldn't solve all the other
discussed items from the wish list you/others brought up in your original thread
(which would need to work on top of the transport), but could be an alternative
way with regards to the email problem.

Thanks,
Daniel

  [0] https://git.kernel.org/pub/scm/public-inbox/vger.kernel.org/bpf/0.git/tree/m
  [1] https://git.kernel.org/pub/scm/public-inbox/vger.kernel.org/bpf/0.git/commit/?id=0c0e0a248293aa382c3451680e696bbeaf38c47e
  [2] Converting the entire netdev history (if anyone ever needs it in their mail client)
      with l2md from 03/2002 till 09/2019 takes on my laptop:
      [...]
      Processed 137396 new mails for https://lore.kernel.org/netdev/1 in 11.198241s.
      Processed 498001 new mails for https://lore.kernel.org/netdev/0 in 41.414329s.
      [...]

> [1] https://lwn.net/Articles/799134/

Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] Reflections on kernel development processes

[Konstantin Ryabitsev]

On Mon, Sep 23, 2019 at 02:52:01PM +0200, Daniel Borkmann wrote:
> > Most (all) of the "kit" functionality should be separated from the UI
> > and be available for scripting/automation/other UIs. Whether it's
> > done as "libgit" or as "shell out" is discussable.
> [...]
> On that note, such a tool would also need to co-exist with the current
> email based process for some (long?) time in order to allow a smooth
> transition period. Last week I spent a few of nights hacking a small tool
> which is regularly pulling the lore git trees I'm interested in and checking
> out all [new] mails into maildir format so they can be read naturally by
> UIs like mutt et al [0]. As an experiment, in case of bpf vger mailing list,
> it extracts all current ~8k mails in under a second:

Thanks for working on this -- I've started on a similar tool in the
past, but got distracted and never completed it. In my implementation,
it was piping messages to procmail, which allowed writing complex rules
for folders/pre-processing, etc. May I suggest that your tool also
offers a stdout that can be piped to procmail?

-K
_______________________________________________
Ksummit-discuss mailing list
Ksummit-discuss@lists.linuxfoundation.org
https://lists.linuxfoundation.org/mailman/listinfo/ksummit-discuss

Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] Reflections on kernel development processes

[Konstantin Ryabitsev]

On Mon, Sep 23, 2019 at 02:52:01PM +0200, Daniel Borkmann wrote:
> > Most (all) of the "kit" functionality should be separated from the UI
> > and be available for scripting/automation/other UIs. Whether it's
> > done as "libgit" or as "shell out" is discussable.
> [...]
> On that note, such a tool would also need to co-exist with the current
> email based process for some (long?) time in order to allow a smooth
> transition period. Last week I spent a few of nights hacking a small tool
> which is regularly pulling the lore git trees I'm interested in and checking
> out all [new] mails into maildir format so they can be read naturally by
> UIs like mutt et al [0]. As an experiment, in case of bpf vger mailing list,
> it extracts all current ~8k mails in under a second:

Thanks for working on this -- I've started on a similar tool in the
past, but got distracted and never completed it. In my implementation,
it was piping messages to procmail, which allowed writing complex rules
for folders/pre-processing, etc. May I suggest that your tool also
offers a stdout that can be piped to procmail?

-K

Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] Reflections on kernel development processes

[Daniel Borkmann]

On 9/30/19 11:24 PM, Konstantin Ryabitsev wrote:
> On Mon, Sep 23, 2019 at 02:52:01PM +0200, Daniel Borkmann wrote:
>>> Most (all) of the "kit" functionality should be separated from the UI
>>> and be available for scripting/automation/other UIs. Whether it's
>>> done as "libgit" or as "shell out" is discussable.
>> [...]
>> On that note, such a tool would also need to co-exist with the current
>> email based process for some (long?) time in order to allow a smooth
>> transition period. Last week I spent a few of nights hacking a small tool
>> which is regularly pulling the lore git trees I'm interested in and checking
>> out all [new] mails into maildir format so they can be read naturally by
>> UIs like mutt et al [0]. As an experiment, in case of bpf vger mailing list,
>> it extracts all current ~8k mails in under a second:
> 
> Thanks for working on this -- I've started on a similar tool in the
> past, but got distracted and never completed it. In my implementation,
> it was piping messages to procmail, which allowed writing complex rules
> for folders/pre-processing, etc. May I suggest that your tool also
> offers a stdout that can be piped to procmail?

I have a rough version working now. ;-) Just pushed to [0]. Let me know if that
does the trick on your side, I've added example configs for procmail to the repo
as well for getting started. Did a quick run for l2md -> procmail -> mutt and
seems fine as far as I can tell. (Patches always welcome of course.)

Thanks,
Daniel

   [0] https://git.kernel.org/pub/scm/linux/kernel/git/dborkman/l2md.git/
_______________________________________________
Ksummit-discuss mailing list
Ksummit-discuss@lists.linuxfoundation.org
https://lists.linuxfoundation.org/mailman/listinfo/ksummit-discuss

Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] Reflections on kernel development processes

[Daniel Borkmann]

On 9/30/19 11:24 PM, Konstantin Ryabitsev wrote:
> On Mon, Sep 23, 2019 at 02:52:01PM +0200, Daniel Borkmann wrote:
>>> Most (all) of the "kit" functionality should be separated from the UI
>>> and be available for scripting/automation/other UIs. Whether it's
>>> done as "libgit" or as "shell out" is discussable.
>> [...]
>> On that note, such a tool would also need to co-exist with the current
>> email based process for some (long?) time in order to allow a smooth
>> transition period. Last week I spent a few of nights hacking a small tool
>> which is regularly pulling the lore git trees I'm interested in and checking
>> out all [new] mails into maildir format so they can be read naturally by
>> UIs like mutt et al [0]. As an experiment, in case of bpf vger mailing list,
>> it extracts all current ~8k mails in under a second:
> 
> Thanks for working on this -- I've started on a similar tool in the
> past, but got distracted and never completed it. In my implementation,
> it was piping messages to procmail, which allowed writing complex rules
> for folders/pre-processing, etc. May I suggest that your tool also
> offers a stdout that can be piped to procmail?

I have a rough version working now. ;-) Just pushed to [0]. Let me know if that
does the trick on your side, I've added example configs for procmail to the repo
as well for getting started. Did a quick run for l2md -> procmail -> mutt and
seems fine as far as I can tell. (Patches always welcome of course.)

Thanks,
Daniel

   [0] https://git.kernel.org/pub/scm/linux/kernel/git/dborkman/l2md.git/

Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] Reflections on kernel development processes

[Konstantin Ryabitsev]

On Tue, Oct 01, 2019 at 11:33:06PM +0200, Daniel Borkmann wrote:
> > Thanks for working on this -- I've started on a similar tool in the
> > past, but got distracted and never completed it. In my implementation,
> > it was piping messages to procmail, which allowed writing complex rules
> > for folders/pre-processing, etc. May I suggest that your tool also
> > offers a stdout that can be piped to procmail?
> 
> I have a rough version working now. ;-) Just pushed to [0]. Let me know if that
> does the trick on your side, I've added example configs for procmail to the repo
> as well for getting started. Did a quick run for l2md -> procmail -> mutt and
> seems fine as far as I can tell. (Patches always welcome of course.)

Nice, thanks! Until we have a better tool for holistic patch/series
management backed by public-inbox, this should allow folks to pre-filter
their messages.

Best,
-K
_______________________________________________
Ksummit-discuss mailing list
Ksummit-discuss@lists.linuxfoundation.org
https://lists.linuxfoundation.org/mailman/listinfo/ksummit-discuss

Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] Reflections on kernel development processes

[Konstantin Ryabitsev]

On Tue, Oct 01, 2019 at 11:33:06PM +0200, Daniel Borkmann wrote:
> > Thanks for working on this -- I've started on a similar tool in the
> > past, but got distracted and never completed it. In my implementation,
> > it was piping messages to procmail, which allowed writing complex rules
> > for folders/pre-processing, etc. May I suggest that your tool also
> > offers a stdout that can be piped to procmail?
> 
> I have a rough version working now. ;-) Just pushed to [0]. Let me know if that
> does the trick on your side, I've added example configs for procmail to the repo
> as well for getting started. Did a quick run for l2md -> procmail -> mutt and
> seems fine as far as I can tell. (Patches always welcome of course.)

Nice, thanks! Until we have a better tool for holistic patch/series
management backed by public-inbox, this should allow folks to pre-filter
their messages.

Best,
-K

Re: [MAINTAINERS SUMMIT] Reflections on kernel development processes

[Konstantin Ryabitsev]

Hi, all:

In retrospect, taking a week offline vacation right after getting
everyone excited about workflow tooling was not the best timing. :) I'm
slowly catching up to all the good conversations that took place while I
was enjoying the fine beaches of South-West Iberian peninsula.

On Sun, Sep 22, 2019 at 02:02:48PM +0200, Dmitry Vyukov wrote:
> While I understand the intention to provide something useful as fast
> as possible, I also a bit afraid that the Stage 1 ("putt") diverges
> us into investing into particular UI, tying capabilities with this UI
> and not addressing the fundamental problems.
> People are expressing a preference for different types of UIs
> (CL, terminal GUI, web, scripting), I think eventually we will have
> several. So I would suggest to untie features/capabilities from
> any particular UI as much as possible, and start with addressing more
> fundamental aspects. Building richer features on top of the current
> human-oriented emails is also going to be much harder, and that's the
> work that we eventually will (hopefully) throw away.

Yes, I fully agree. My perspective is a bit limited in this regard,
because I am not a kernel contributor, so I only hear what the
maintainers complain about. That said, I am also prioritizing the kind
of effort that we can get funded -- and high-profile Linux kernel
developers have access to a lot of important ears. I am working through
my backlog of emails to fine-tune my tool development proposal, which I
hope will allow me to get some funds via LF channels.

> There are 3 groups of people we should be looking at:
> - contributors (important special case: sending first patch)
> - maintainers
> - reviewers
> 
> I would set the first milestone as having the CL utility (let's call
> it "kit"*) which can do:

I think we should call it kthul (pronounced k-tool, easy to type, and
powerful enough to summon the Great Ones). :) Mostly kidding, except
perhaps that "kit" is not websearch-friendly, so people googling for
docs would have an easier time finding hits for "kthul".

> $ kit init
> # Does some necessary one-time initialization, executed from the
> # kernel git checkout.
> 
> $ kit mail
> # Sends the top commit in the current branch for review.
> 
> So that would be the workflow for sending your first kernel patch.

I would argue that "sending your first patch" should be a hosted web
service. A while back we discussed this in some detail -- it would be a
simple site that would basically accept a handful of fields:

- your name
- your email address
- the URL of your linux fork (e.g. on Github/Gitlab)
- (optionally) the commit id, if not master
- (optionally) a text field for the cover letter
- (optionally) message-id of the previous patch/series revision

The service would create the patch/series based on this input, run
checkpatch, validate that the provided email address is valid, and then
submit the patch/series to the proper list(s) based on get_maintainer
results.

The reasons why I think this should be a web service as opposed to a CLI
tool are pretty simple:

- it is easier to make updates to a web service instead of asking
  newbies to "git pull/make"
- it doesn't force us to stick to languages/libraries that are likely to
  be present on users' systems (which generally limits us to perl5,
  python, c)
- it would be better received by newbies than a CLI tool (arguable, but
  still)

> Later "kit mail" can also run checkpatch, check SOB tag, add some kind
> of change ID or anything else we will consider necessary. It may be
> necessary to be able to force-override some of the checks, but by default
> you are now getting patches that have SOB, checkpatch-clean, etc.

I would argue that this should be a totally separate tool from what
maintainers use, *or* that there should be two different tools -- a
simplified web bridge as described above for people submitting
occasional patches and a more powerful CLI tool used by regular
developers.

> If there is an easy way to make it work with the current email-based
> process (i.e. send email on your behalf and you receive incoming emails),
> then we could do that first and give it to new developers to relief from
> setting up email client. Otherwise, we should continue developing it
> based on something like SSB (or whatever protocol we will choose).

This slots easily into the "community bridges" workflow that I have in
mind, which is why I'm leaning towards it being a web tool.

> On the protocol side I don't have strong preference for SSB or
> something similar custom. It seems that we will use SSB in somewhat
> simplified way, i.e. global connected graph, rather than several large
> groups or small isolated groups. We won't need Facebook-like following
> nor Pubs announcements. You obviously don't want to be notified of all
> messages in the system (LKML), but still it's a global graph in the
> sense that you can receive anything if you want or CC anybody.
> That limited subset of SSB should be easier to implement.
> So as Konstantin said, we could fork SSB to better fit our needs.
> The more important part will be the application-level protocol that
> we will transfer inside of SSB messages, which is mostly transport
> protocol for our needs (at least for the majority, maybe not for
> Konstantin's concerns :)).

I'm all for taking the best parts of SSB and ditching the not-so-great
parts of it (like reliance on nodejs-formatted json), but we need to be
careful not to paint ourselves into the corner doing so. There's a lot
of careful thought behind SSB, especially its decentralized nature, and
people who are developing the v2 of the protocol are well-aware of the
downsides of v1. I think we should let them complete this work before
deciding to fork it or not.

Many people asked why not just use git as the channel/protocol instead
of something esoteric like SSB, and the easy answer is that git is
poorly suited for this purpose:

- git is fine-tuned for small revisions to large static codebases; it is
  not well-suited for applications where either the number of files
  grows indefinitely as records are added, or where the contents of the
  same file get completely rewritten between revisions (as with
  public-inbox v2). It *works*, but it's obviously not built for it.
- we don't need multiple branches/tags/etc for the purpose of creating a
  chain of records; we also don't need to worry about
  merging/rebasing/etc. Using git would be like using the scissors on
  a swiss army knife -- you *can*, but only out of desperation.
- similarly, git replication protocol is too chatty for what we need
  it to do -- it was written to support reconciling between multiple
  branches, some of which may have been rebased/deleted/etc. For
  replicating a chain of immutable records, we just need to compare the
  auto-incremented value of the latest record on the chain

> I would suggest to put bug/issue tracking aside for now (it's an
> important thing, but it should be possible to add it later) and also
> "bare chatting" (which can be done over emails for now) and
> concentrate on patches just to make things simpler. Patches are
> the cornerstone of the process.

Agreed.

> So we need to define the format of the "patch for review" SSB message
> which "kit mail" will form and push.

I think we shouldn't touch SSB yet -- we can write the maintainer tool
to support multiple sources of patches:

- public-inbox repositories (most natural choice)
- web tools that offer REST APIs (patchwork/patchew/etc)
- SSB sigchains, eventually

In my mind, these would be pluggable in the same way as one can choose
pop/imap and smtp/sendmail for mutt -- the rest of the app works the
same regardless of what fetch/delivery method you have configured.

> Regarding "state machines" in the protocol (Radicle/IPFS), I think
> it's not just "sipping the blockchain cool-aid a bit too much",
> it's a wrong tool for our needs. Smart contracts are used for
> crypto-currencies where one does want to carve the rules in the
> blockchain. But we don't want and don't need this.

Right, we need a solution where humans are the final deciding factors
about what makes it into their git trees. It's interesting to consider
futuristic situations where such decisions are made by smart contracts,
but it's a path I'd rather leave for someone more bright-eyed and
optimistic.

> Another important part of the system is user identities.
> Do we go with a public/private key pair? Or we have some other realistic
> alternatives? Assuming we go with key pairs for now, "kit init"
> will generate a local key pair for you (a new developer). But a user
> should be able to evacuate/export the private key later and pass
> an existing key (to bootstrap a new machine with the same identity).

Ah, now we are getting to *interesting* bits. :) I have a separate
conversation happening on the git list where I argue that git-send-email
and git-am should natively support minisign-type signatures:

https://public-inbox.org/git/20190930153741.GA6124@chatter.i7.local/

Other than cryptographic signing, there aren't really any "realistic
alternatives" -- and I would argue that we should continue to use PGP
for git tag/commit signing, but consider simpler alternatives for
patches, such as my minisign+TOFU proposal there.

Upsides of PGP:

- established standard with good support in git tools, including web
  tools like Gitlab/Github
- smartcard support, so PGP keys belonging to important maintainers are
  better protected than just by a passphrase
- cross-certification support with other PGP keys ("key signing")
  allowing us to establish a web of trust

Upsides of minisign:

- really fast
- really tiny signatures and public keys, such that adding them to patch
  submissions doesn't balloon things out of proportions, as with PGP
  keys and signatures
- good cross-platform support (minisign is a compatible reimplementation
  of OpenBSD's signify)

> However, we will probably need another identity that is slightly
> easier to remember and type in patch CC line than 256-char hash.
> And that probably needs to be an email address (required for sending
> email notifications anyway). But I don't know how to ensure uniqueness
> of emails in this system. An alternative would be to use usernames
> (e.g. "torvalds" or "tytso") and then a user can map that to own email
> as they want. But this does not remove the requirement for uniqueness.

My suggestion is that we continue to use email addresses for now, since
that's still required for git commits. Having a "unique global identity"
is a hard problem to solve (if it's at all solvable), so we shouldn't
try to come up with a solution while brainstorming this tool.

> Two more interesting/controversial possibilities.
> If we have an email bridge, we could also have a github bridge!
> Don't get me wrong, I am not saying we need to do this now or at all.
> I am saying that if UI part is abstracted enough, then it may be
> theoretically possible to take a PR on a special dedicated github
> project, convert it to "patch for review" SSB message and inject
> into the system. Comments on the patch will be proxied back to github.
> Andrew will receive this over email bridge and review and merge,
> not even suspecting he is reviewing a github PR (w00t!).

It's basically what I described above with my "patch submission web
service" tool. :)

> Second controversial idea: the local rich GUI/patchwork is actually
> web-based _but_ it talks to a local web server (so fast and no internet
> connection required) _and_ it resembles terminal UI and has tons of
> hotkeys and terminal-like navigation (so it kinda feels like terminal).
> You start it with "kit gui" which starts a browser for you.
> The advantage of this: we build 1 UI instead of 2, so immediate 2x
> time savings. Also consistency between the UIs: you go to web, you see
> exactly the same UI that you used to work with locally (now it's just
> powered by a remote web server).

It's not that controversial -- a web-ui that runs on a local port and is
accessible via a browser is something we should totally do, simply
because it offers a much richer interface than the terminal. The main
trouble would be designing a non-horrible web UI for this purpose...
without trying to offend anyone on this list, I'd argue that nobody
present is any good at web UIs. :) So, we should enlist a group of
people who are willing to put a good amount of effort into getting it
done.

Thanks for all of your work on this! I am trying to finalize a proposal
that I can submit for review to the Proper People -- I just wish the
rest of my work would stop demanding so much attention at the moment, so
I can spend more time working on cool things like this, and less time on
putting together spreadsheets and approving travel expenses. :)

Best,
-K

Re: [MAINTAINERS SUMMIT] Reflections on kernel development processes

[Dmitry Vyukov]

On Mon, Sep 30, 2019 at 10:24 PM Konstantin Ryabitsev
<konstantin@linuxfoundation.org> wrote:
>
> Hi, all:
>
> In retrospect, taking a week offline vacation right after getting
> everyone excited about workflow tooling was not the best timing. :) I'm
> slowly catching up to all the good conversations that took place while I
> was enjoying the fine beaches of South-West Iberian peninsula.

Hi Konstantin,

Are you going to attend OSS Summit EU? If yes, I think we should sync
up there. I know Dave and Greg are there too. Perhaps we could
schedule some kind of semi-formal meeting?


> On Sun, Sep 22, 2019 at 02:02:48PM +0200, Dmitry Vyukov wrote:
> > While I understand the intention to provide something useful as fast
> > as possible, I also a bit afraid that the Stage 1 ("putt") diverges
> > us into investing into particular UI, tying capabilities with this UI
> > and not addressing the fundamental problems.
> > People are expressing a preference for different types of UIs
> > (CL, terminal GUI, web, scripting), I think eventually we will have
> > several. So I would suggest to untie features/capabilities from
> > any particular UI as much as possible, and start with addressing more
> > fundamental aspects. Building richer features on top of the current
> > human-oriented emails is also going to be much harder, and that's the
> > work that we eventually will (hopefully) throw away.
>
> Yes, I fully agree. My perspective is a bit limited in this regard,
> because I am not a kernel contributor, so I only hear what the
> maintainers complain about. That said, I am also prioritizing the kind
> of effort that we can get funded -- and high-profile Linux kernel
> developers have access to a lot of important ears. I am working through
> my backlog of emails to fine-tune my tool development proposal, which I
> hope will allow me to get some funds via LF channels.
>
> > There are 3 groups of people we should be looking at:
> > - contributors (important special case: sending first patch)
> > - maintainers
> > - reviewers
> >
> > I would set the first milestone as having the CL utility (let's call
> > it "kit"*) which can do:
>
> I think we should call it kthul (pronounced k-tool, easy to type, and
> powerful enough to summon the Great Ones). :) Mostly kidding, except
> perhaps that "kit" is not websearch-friendly, so people googling for
> docs would have an easier time finding hits for "kthul".
>
> > $ kit init
> > # Does some necessary one-time initialization, executed from the
> > # kernel git checkout.
> >
> > $ kit mail
> > # Sends the top commit in the current branch for review.
> >
> > So that would be the workflow for sending your first kernel patch.
>
> I would argue that "sending your first patch" should be a hosted web
> service. A while back we discussed this in some detail -- it would be a
> simple site that would basically accept a handful of fields:
>
> - your name
> - your email address
> - the URL of your linux fork (e.g. on Github/Gitlab)
> - (optionally) the commit id, if not master
> - (optionally) a text field for the cover letter
> - (optionally) message-id of the previous patch/series revision
>
> The service would create the patch/series based on this input, run
> checkpatch, validate that the provided email address is valid, and then
> submit the patch/series to the proper list(s) based on get_maintainer
> results.
>
> The reasons why I think this should be a web service as opposed to a CLI
> tool are pretty simple:
>
> - it is easier to make updates to a web service instead of asking
>   newbies to "git pull/make"
> - it doesn't force us to stick to languages/libraries that are likely to
>   be present on users' systems (which generally limits us to perl5,
>   python, c)
> - it would be better received by newbies than a CLI tool (arguable, but
>   still)
>
> > Later "kit mail" can also run checkpatch, check SOB tag, add some kind
> > of change ID or anything else we will consider necessary. It may be
> > necessary to be able to force-override some of the checks, but by default
> > you are now getting patches that have SOB, checkpatch-clean, etc.
>
> I would argue that this should be a totally separate tool from what
> maintainers use, *or* that there should be two different tools -- a
> simplified web bridge as described above for people submitting
> occasional patches and a more powerful CLI tool used by regular
> developers.
>
> > If there is an easy way to make it work with the current email-based
> > process (i.e. send email on your behalf and you receive incoming emails),
> > then we could do that first and give it to new developers to relief from
> > setting up email client. Otherwise, we should continue developing it
> > based on something like SSB (or whatever protocol we will choose).
>
> This slots easily into the "community bridges" workflow that I have in
> mind, which is why I'm leaning towards it being a web tool.
>
> > On the protocol side I don't have strong preference for SSB or
> > something similar custom. It seems that we will use SSB in somewhat
> > simplified way, i.e. global connected graph, rather than several large
> > groups or small isolated groups. We won't need Facebook-like following
> > nor Pubs announcements. You obviously don't want to be notified of all
> > messages in the system (LKML), but still it's a global graph in the
> > sense that you can receive anything if you want or CC anybody.
> > That limited subset of SSB should be easier to implement.
> > So as Konstantin said, we could fork SSB to better fit our needs.
> > The more important part will be the application-level protocol that
> > we will transfer inside of SSB messages, which is mostly transport
> > protocol for our needs (at least for the majority, maybe not for
> > Konstantin's concerns :)).
>
> I'm all for taking the best parts of SSB and ditching the not-so-great
> parts of it (like reliance on nodejs-formatted json), but we need to be
> careful not to paint ourselves into the corner doing so. There's a lot
> of careful thought behind SSB, especially its decentralized nature, and
> people who are developing the v2 of the protocol are well-aware of the
> downsides of v1. I think we should let them complete this work before
> deciding to fork it or not.
>
> Many people asked why not just use git as the channel/protocol instead
> of something esoteric like SSB, and the easy answer is that git is
> poorly suited for this purpose:
>
> - git is fine-tuned for small revisions to large static codebases; it is
>   not well-suited for applications where either the number of files
>   grows indefinitely as records are added, or where the contents of the
>   same file get completely rewritten between revisions (as with
>   public-inbox v2). It *works*, but it's obviously not built for it.
> - we don't need multiple branches/tags/etc for the purpose of creating a
>   chain of records; we also don't need to worry about
>   merging/rebasing/etc. Using git would be like using the scissors on
>   a swiss army knife -- you *can*, but only out of desperation.
> - similarly, git replication protocol is too chatty for what we need
>   it to do -- it was written to support reconciling between multiple
>   branches, some of which may have been rebased/deleted/etc. For
>   replicating a chain of immutable records, we just need to compare the
>   auto-incremented value of the latest record on the chain
>
> > I would suggest to put bug/issue tracking aside for now (it's an
> > important thing, but it should be possible to add it later) and also
> > "bare chatting" (which can be done over emails for now) and
> > concentrate on patches just to make things simpler. Patches are
> > the cornerstone of the process.
>
> Agreed.
>
> > So we need to define the format of the "patch for review" SSB message
> > which "kit mail" will form and push.
>
> I think we shouldn't touch SSB yet -- we can write the maintainer tool
> to support multiple sources of patches:
>
> - public-inbox repositories (most natural choice)
> - web tools that offer REST APIs (patchwork/patchew/etc)
> - SSB sigchains, eventually
>
> In my mind, these would be pluggable in the same way as one can choose
> pop/imap and smtp/sendmail for mutt -- the rest of the app works the
> same regardless of what fetch/delivery method you have configured.
>
> > Regarding "state machines" in the protocol (Radicle/IPFS), I think
> > it's not just "sipping the blockchain cool-aid a bit too much",
> > it's a wrong tool for our needs. Smart contracts are used for
> > crypto-currencies where one does want to carve the rules in the
> > blockchain. But we don't want and don't need this.
>
> Right, we need a solution where humans are the final deciding factors
> about what makes it into their git trees. It's interesting to consider
> futuristic situations where such decisions are made by smart contracts,
> but it's a path I'd rather leave for someone more bright-eyed and
> optimistic.
>
> > Another important part of the system is user identities.
> > Do we go with a public/private key pair? Or we have some other realistic
> > alternatives? Assuming we go with key pairs for now, "kit init"
> > will generate a local key pair for you (a new developer). But a user
> > should be able to evacuate/export the private key later and pass
> > an existing key (to bootstrap a new machine with the same identity).
>
> Ah, now we are getting to *interesting* bits. :) I have a separate
> conversation happening on the git list where I argue that git-send-email
> and git-am should natively support minisign-type signatures:
>
> https://public-inbox.org/git/20190930153741.GA6124@chatter.i7.local/
>
> Other than cryptographic signing, there aren't really any "realistic
> alternatives" -- and I would argue that we should continue to use PGP
> for git tag/commit signing, but consider simpler alternatives for
> patches, such as my minisign+TOFU proposal there.
>
> Upsides of PGP:
>
> - established standard with good support in git tools, including web
>   tools like Gitlab/Github
> - smartcard support, so PGP keys belonging to important maintainers are
>   better protected than just by a passphrase
> - cross-certification support with other PGP keys ("key signing")
>   allowing us to establish a web of trust
>
> Upsides of minisign:
>
> - really fast
> - really tiny signatures and public keys, such that adding them to patch
>   submissions doesn't balloon things out of proportions, as with PGP
>   keys and signatures
> - good cross-platform support (minisign is a compatible reimplementation
>   of OpenBSD's signify)
>
> > However, we will probably need another identity that is slightly
> > easier to remember and type in patch CC line than 256-char hash.
> > And that probably needs to be an email address (required for sending
> > email notifications anyway). But I don't know how to ensure uniqueness
> > of emails in this system. An alternative would be to use usernames
> > (e.g. "torvalds" or "tytso") and then a user can map that to own email
> > as they want. But this does not remove the requirement for uniqueness.
>
> My suggestion is that we continue to use email addresses for now, since
> that's still required for git commits. Having a "unique global identity"
> is a hard problem to solve (if it's at all solvable), so we shouldn't
> try to come up with a solution while brainstorming this tool.
>
> > Two more interesting/controversial possibilities.
> > If we have an email bridge, we could also have a github bridge!
> > Don't get me wrong, I am not saying we need to do this now or at all.
> > I am saying that if UI part is abstracted enough, then it may be
> > theoretically possible to take a PR on a special dedicated github
> > project, convert it to "patch for review" SSB message and inject
> > into the system. Comments on the patch will be proxied back to github.
> > Andrew will receive this over email bridge and review and merge,
> > not even suspecting he is reviewing a github PR (w00t!).
>
> It's basically what I described above with my "patch submission web
> service" tool. :)
>
> > Second controversial idea: the local rich GUI/patchwork is actually
> > web-based _but_ it talks to a local web server (so fast and no internet
> > connection required) _and_ it resembles terminal UI and has tons of
> > hotkeys and terminal-like navigation (so it kinda feels like terminal).
> > You start it with "kit gui" which starts a browser for you.
> > The advantage of this: we build 1 UI instead of 2, so immediate 2x
> > time savings. Also consistency between the UIs: you go to web, you see
> > exactly the same UI that you used to work with locally (now it's just
> > powered by a remote web server).
>
> It's not that controversial -- a web-ui that runs on a local port and is
> accessible via a browser is something we should totally do, simply
> because it offers a much richer interface than the terminal. The main
> trouble would be designing a non-horrible web UI for this purpose...
> without trying to offend anyone on this list, I'd argue that nobody
> present is any good at web UIs. :) So, we should enlist a group of
> people who are willing to put a good amount of effort into getting it
> done.
>
> Thanks for all of your work on this! I am trying to finalize a proposal
> that I can submit for review to the Proper People -- I just wish the
> rest of my work would stop demanding so much attention at the moment, so
> I can spend more time working on cool things like this, and less time on
> putting together spreadsheets and approving travel expenses. :)
>
> Best,
> -K

Re: [MAINTAINERS SUMMIT] Reflections on kernel development processes

[Konstantin Ryabitsev]

On Tue, Oct 08, 2019 at 08:46:02AM +0200, Dmitry Vyukov wrote:
>On Mon, Sep 30, 2019 at 10:24 PM Konstantin Ryabitsev
><konstantin@linuxfoundation.org> wrote:
>>
>> Hi, all:
>>
>> In retrospect, taking a week offline vacation right after getting
>> everyone excited about workflow tooling was not the best timing. :) I'm
>> slowly catching up to all the good conversations that took place while I
>> was enjoying the fine beaches of South-West Iberian peninsula.
>
>Hi Konstantin,
>
>Are you going to attend OSS Summit EU? If yes, I think we should sync
>up there. I know Dave and Greg are there too. Perhaps we could
>schedule some kind of semi-formal meeting?

Unfortunately, no -- we have limited budget for traveling and I burned 
through mine by going to Lisbon. I'd be happy to work on a draft 
document that expands on my tooling vision (I'm doing so anyway), and I 
can time it to be ready before the summit so you can review and improve 
it. My hope is to submit it formally to the LF by the end of the month.

-K

Re: [MAINTAINERS SUMMIT] Reflections on kernel development processes

[Konstantin Ryabitsev]

On Tue, 8 Oct 2019 at 12:51, Konstantin Ryabitsev
<konstantin@linuxfoundation.org> wrote:
> >Hi Konstantin,
> >
> >Are you going to attend OSS Summit EU? If yes, I think we should sync
> >up there. I know Dave and Greg are there too. Perhaps we could
> >schedule some kind of semi-formal meeting?
>
> Unfortunately, no -- we have limited budget for traveling and I burned
> through mine by going to Lisbon. I'd be happy to work on a draft
> document that expands on my tooling vision (I'm doing so anyway), and I
> can time it to be ready before the summit so you can review and improve
> it. My hope is to submit it formally to the LF by the end of the month.

Oh, hey, change of plans -- looks like I *will* be going to Lyon after
all. I think sitting down together for a semi-formal meeting sounds
like a good idea, considering all the active discussions happening
lately. Should we hash out a tentative agenda about topics that we
should and shouldn't cover? Who should be in attendance?

Best Regards,
-- 
Konstantin Ryabitsev
Director, Projects IT
The Linux Foundation
Montréal, Québec

Re: [MAINTAINERS SUMMIT] Reflections on kernel development processes

[Steven Rostedt]

On Thu, 10 Oct 2019 22:16:42 -0400
Konstantin Ryabitsev <konstantin@linuxfoundation.org> wrote:
> 
> Oh, hey, change of plans -- looks like I *will* be going to Lyon after

Nice.

> all. I think sitting down together for a semi-formal meeting sounds
> like a good idea, considering all the active discussions happening
> lately. Should we hash out a tentative agenda about topics that we
> should and shouldn't cover? Who should be in attendance?

I'd like to be there. (I'll be in Lyon)

-- Steve

Re: [MAINTAINERS SUMMIT] Reflections on kernel development processes

[Greg Kroah-Hartman]

On Thu, Oct 10, 2019 at 10:16:42PM -0400, Konstantin Ryabitsev wrote:
> On Tue, 8 Oct 2019 at 12:51, Konstantin Ryabitsev
> <konstantin@linuxfoundation.org> wrote:
> > >Hi Konstantin,
> > >
> > >Are you going to attend OSS Summit EU? If yes, I think we should sync
> > >up there. I know Dave and Greg are there too. Perhaps we could
> > >schedule some kind of semi-formal meeting?
> >
> > Unfortunately, no -- we have limited budget for traveling and I burned
> > through mine by going to Lisbon. I'd be happy to work on a draft
> > document that expands on my tooling vision (I'm doing so anyway), and I
> > can time it to be ready before the summit so you can review and improve
> > it. My hope is to submit it formally to the LF by the end of the month.
> 
> Oh, hey, change of plans -- looks like I *will* be going to Lyon after
> all. I think sitting down together for a semi-formal meeting sounds
> like a good idea, considering all the active discussions happening
> lately. Should we hash out a tentative agenda about topics that we
> should and shouldn't cover? Who should be in attendance?

I'd like to be there, as I will be at the conference.

thanks,

greg k-h

Re: [MAINTAINERS SUMMIT] Reflections on kernel development processes

[Dmitry Vyukov]

On Fri, Oct 11, 2019 at 10:31 AM Greg Kroah-Hartman
<gregkh@linuxfoundation.org> wrote:
>
> On Thu, Oct 10, 2019 at 10:16:42PM -0400, Konstantin Ryabitsev wrote:
> > On Tue, 8 Oct 2019 at 12:51, Konstantin Ryabitsev
> > <konstantin@linuxfoundation.org> wrote:
> > > >Hi Konstantin,
> > > >
> > > >Are you going to attend OSS Summit EU? If yes, I think we should sync
> > > >up there. I know Dave and Greg are there too. Perhaps we could
> > > >schedule some kind of semi-formal meeting?
> > >
> > > Unfortunately, no -- we have limited budget for traveling and I burned
> > > through mine by going to Lisbon. I'd be happy to work on a draft
> > > document that expands on my tooling vision (I'm doing so anyway), and I
> > > can time it to be ready before the summit so you can review and improve
> > > it. My hope is to submit it formally to the LF by the end of the month.
> >
> > Oh, hey, change of plans -- looks like I *will* be going to Lyon after
> > all. I think sitting down together for a semi-formal meeting sounds
> > like a good idea, considering all the active discussions happening
> > lately. Should we hash out a tentative agenda about topics that we
> > should and shouldn't cover? Who should be in attendance?
>
> I'd like to be there, as I will be at the conference.


Cool! Konstantin, glad you will make it!

Let's try to figure out time then.
I propose the first day to not leave it to the last day and we may
need some follow up discussions.
On Monday I see Greg's keynote at 09:50 - 10:10. Steven's talk at
12:20 - 12:55. Sasha's talk at 12:20 - 12:55. Any other conflicts for
people who want to attend?
What about 10:30-11:30 (this covers a coffee break at 10:45-11:00, can
prolong it to 12:00)? Any other time slot proposals?

FTR here is the schedule link:
https://events19.linuxfoundation.org/events/open-source-summit-europe-2019/program/schedule/

Anybody knows how we can find a room, etc? Or knows somebody who
knows? Should it be on a BoF track? Should we also tentatively reserve
a slot on subsequent days?

Re: [MAINTAINERS SUMMIT] Reflections on kernel development processes

[Dmitry Vyukov]

On Fri, Oct 11, 2019 at 10:59 AM Dmitry Vyukov <dvyukov@google.com> wrote:
>
> On Fri, Oct 11, 2019 at 10:31 AM Greg Kroah-Hartman
> <gregkh@linuxfoundation.org> wrote:
> >
> > On Thu, Oct 10, 2019 at 10:16:42PM -0400, Konstantin Ryabitsev wrote:
> > > On Tue, 8 Oct 2019 at 12:51, Konstantin Ryabitsev
> > > <konstantin@linuxfoundation.org> wrote:
> > > > >Hi Konstantin,
> > > > >
> > > > >Are you going to attend OSS Summit EU? If yes, I think we should sync
> > > > >up there. I know Dave and Greg are there too. Perhaps we could
> > > > >schedule some kind of semi-formal meeting?
> > > >
> > > > Unfortunately, no -- we have limited budget for traveling and I burned
> > > > through mine by going to Lisbon. I'd be happy to work on a draft
> > > > document that expands on my tooling vision (I'm doing so anyway), and I
> > > > can time it to be ready before the summit so you can review and improve
> > > > it. My hope is to submit it formally to the LF by the end of the month.
> > >
> > > Oh, hey, change of plans -- looks like I *will* be going to Lyon after
> > > all. I think sitting down together for a semi-formal meeting sounds
> > > like a good idea, considering all the active discussions happening
> > > lately. Should we hash out a tentative agenda about topics that we
> > > should and shouldn't cover? Who should be in attendance?
> >
> > I'd like to be there, as I will be at the conference.
>
>
> Cool! Konstantin, glad you will make it!
>
> Let's try to figure out time then.
> I propose the first day to not leave it to the last day and we may
> need some follow up discussions.
> On Monday I see Greg's keynote at 09:50 - 10:10. Steven's talk at
> 12:20 - 12:55. Sasha's talk at 12:20 - 12:55. Any other conflicts for
> people who want to attend?
> What about 10:30-11:30 (this covers a coffee break at 10:45-11:00, can
> prolong it to 12:00)? Any other time slot proposals?

+Christian also expressed interest, his talks are 14:25 - 15:50 and
16:20 - 16:55 on Monday.

>
> FTR here is the schedule link:
> https://events19.linuxfoundation.org/events/open-source-summit-europe-2019/program/schedule/
>
> Anybody knows how we can find a room, etc? Or knows somebody who
> knows? Should it be on a BoF track? Should we also tentatively reserve
> a slot on subsequent days?

Re: [MAINTAINERS SUMMIT] Reflections on kernel development processes

[Christian Brauner]

On Fri, Oct 11, 2019 at 11:33:35AM +0200, Dmitry Vyukov wrote:
> On Fri, Oct 11, 2019 at 10:59 AM Dmitry Vyukov <dvyukov@google.com> wrote:
> >
> > On Fri, Oct 11, 2019 at 10:31 AM Greg Kroah-Hartman
> > <gregkh@linuxfoundation.org> wrote:
> > >
> > > On Thu, Oct 10, 2019 at 10:16:42PM -0400, Konstantin Ryabitsev wrote:
> > > > On Tue, 8 Oct 2019 at 12:51, Konstantin Ryabitsev
> > > > <konstantin@linuxfoundation.org> wrote:
> > > > > >Hi Konstantin,
> > > > > >
> > > > > >Are you going to attend OSS Summit EU? If yes, I think we should sync
> > > > > >up there. I know Dave and Greg are there too. Perhaps we could
> > > > > >schedule some kind of semi-formal meeting?
> > > > >
> > > > > Unfortunately, no -- we have limited budget for traveling and I burned
> > > > > through mine by going to Lisbon. I'd be happy to work on a draft
> > > > > document that expands on my tooling vision (I'm doing so anyway), and I
> > > > > can time it to be ready before the summit so you can review and improve
> > > > > it. My hope is to submit it formally to the LF by the end of the month.
> > > >
> > > > Oh, hey, change of plans -- looks like I *will* be going to Lyon after
> > > > all. I think sitting down together for a semi-formal meeting sounds
> > > > like a good idea, considering all the active discussions happening
> > > > lately. Should we hash out a tentative agenda about topics that we
> > > > should and shouldn't cover? Who should be in attendance?
> > >
> > > I'd like to be there, as I will be at the conference.
> >
> >
> > Cool! Konstantin, glad you will make it!
> >
> > Let's try to figure out time then.
> > I propose the first day to not leave it to the last day and we may
> > need some follow up discussions.
> > On Monday I see Greg's keynote at 09:50 - 10:10. Steven's talk at
> > 12:20 - 12:55. Sasha's talk at 12:20 - 12:55. Any other conflicts for
> > people who want to attend?
> > What about 10:30-11:30 (this covers a coffee break at 10:45-11:00, can
> > prolong it to 12:00)? Any other time slot proposals?
> 
> +Christian also expressed interest, his talks are 14:25 - 15:50 and
> 16:20 - 16:55 on Monday.

Thanks Dmitry!
Christian

Re: [MAINTAINERS SUMMIT] Reflections on kernel development processes

[Steven Rostedt]

On Fri, 11 Oct 2019 11:33:35 +0200
Dmitry Vyukov <dvyukov@google.com> wrote:

> > Let's try to figure out time then.
> > I propose the first day to not leave it to the last day and we may
> > need some follow up discussions.
> > On Monday I see Greg's keynote at 09:50 - 10:10. Steven's talk at
> > 12:20 - 12:55. Sasha's talk at 12:20 - 12:55. Any other conflicts for
> > people who want to attend?
> > What about 10:30-11:30 (this covers a coffee break at 10:45-11:00, can
> > prolong it to 12:00)? Any other time slot proposals?  
> 
> +Christian also expressed interest, his talks are 14:25 - 15:50 and
> 16:20 - 16:55 on Monday.

If everyone's giving a talk on Monday, can we do this meeting on
Tuesday?

It's much easier to concentrate if you don't have to worry about a talk
you are about to give (or just finished).

-- Steve

Re: [MAINTAINERS SUMMIT] Reflections on kernel development processes

[Christian Brauner]

On Fri, Oct 11, 2019 at 09:18:12AM -0400, Steven Rostedt wrote:
> On Fri, 11 Oct 2019 11:33:35 +0200
> Dmitry Vyukov <dvyukov@google.com> wrote:
> 
> > > Let's try to figure out time then.
> > > I propose the first day to not leave it to the last day and we may
> > > need some follow up discussions.
> > > On Monday I see Greg's keynote at 09:50 - 10:10. Steven's talk at
> > > 12:20 - 12:55. Sasha's talk at 12:20 - 12:55. Any other conflicts for
> > > people who want to attend?
> > > What about 10:30-11:30 (this covers a coffee break at 10:45-11:00, can
> > > prolong it to 12:00)? Any other time slot proposals?  
> > 
> > +Christian also expressed interest, his talks are 14:25 - 15:50 and
> > 16:20 - 16:55 on Monday.
> 
> If everyone's giving a talk on Monday, can we do this meeting on
> Tuesday?
> 
> It's much easier to concentrate if you don't have to worry about a talk
> you are about to give (or just finished).

Seconded. :)

Christian

Re: [MAINTAINERS SUMMIT] Reflections on kernel development processes

[Dmitry Vyukov]

On Fri, Oct 11, 2019 at 3:19 PM Christian Brauner
<christian.brauner@ubuntu.com> wrote:
>
> On Fri, Oct 11, 2019 at 09:18:12AM -0400, Steven Rostedt wrote:
> > On Fri, 11 Oct 2019 11:33:35 +0200
> > Dmitry Vyukov <dvyukov@google.com> wrote:
> >
> > > > Let's try to figure out time then.
> > > > I propose the first day to not leave it to the last day and we may
> > > > need some follow up discussions.
> > > > On Monday I see Greg's keynote at 09:50 - 10:10. Steven's talk at
> > > > 12:20 - 12:55. Sasha's talk at 12:20 - 12:55. Any other conflicts for
> > > > people who want to attend?
> > > > What about 10:30-11:30 (this covers a coffee break at 10:45-11:00, can
> > > > prolong it to 12:00)? Any other time slot proposals?
> > >
> > > +Christian also expressed interest, his talks are 14:25 - 15:50 and
> > > 16:20 - 16:55 on Monday.
> >
> > If everyone's giving a talk on Monday, can we do this meeting on
> > Tuesday?
> >
> > It's much easier to concentrate if you don't have to worry about a talk
> > you are about to give (or just finished).
>
> Seconded. :)

It's not that there is nothing on Tue, it's just that I looked only on
Monday schedule :)
For Tue I see for potentially interested people:
09:00 - 09:20 Linus keynote
11:00 - 12:00 Shuah Office Hours
12:20 - 12:55 Rafael talk
Anything else?

Tue is fine with me, I just did not want to leave it to the last day
because most likely it will take me a day to digest all info, so I
will be able to reply something meaningful only the next day. It would
be good to have some in-person conversations next day as well.

Tue 14:00-15:00?

Re: [MAINTAINERS SUMMIT] Reflections on kernel development processes

[Laurent Pinchart]

On Fri, Oct 11, 2019 at 03:30:29PM +0200, Dmitry Vyukov wrote:
> On Fri, Oct 11, 2019 at 3:19 PM Christian Brauner wrote:
> > On Fri, Oct 11, 2019 at 09:18:12AM -0400, Steven Rostedt wrote:
> > > On Fri, 11 Oct 2019 11:33:35 +0200 Dmitry Vyukov wrote:
> > >
> > > > > Let's try to figure out time then.
> > > > > I propose the first day to not leave it to the last day and we may
> > > > > need some follow up discussions.
> > > > > On Monday I see Greg's keynote at 09:50 - 10:10. Steven's talk at
> > > > > 12:20 - 12:55. Sasha's talk at 12:20 - 12:55. Any other conflicts for
> > > > > people who want to attend?
> > > > > What about 10:30-11:30 (this covers a coffee break at 10:45-11:00, can
> > > > > prolong it to 12:00)? Any other time slot proposals?
> > > >
> > > > +Christian also expressed interest, his talks are 14:25 - 15:50 and
> > > > 16:20 - 16:55 on Monday.
> > >
> > > If everyone's giving a talk on Monday, can we do this meeting on
> > > Tuesday?
> > >
> > > It's much easier to concentrate if you don't have to worry about a talk
> > > you are about to give (or just finished).
> >
> > Seconded. :)
> 
> It's not that there is nothing on Tue, it's just that I looked only on
> Monday schedule :)
> For Tue I see for potentially interested people:
> 09:00 - 09:20 Linus keynote
> 11:00 - 12:00 Shuah Office Hours
> 12:20 - 12:55 Rafael talk
> Anything else?

I need to attend the libcamera talk at 15:15.

> Tue is fine with me, I just did not want to leave it to the last day
> because most likely it will take me a day to digest all info, so I
> will be able to reply something meaningful only the next day. It would
> be good to have some in-person conversations next day as well.
> 
> Tue 14:00-15:00?

Works for me. I wonder however if we should try to start the discussions
in the morning, in order to have time in the afternoon to handle the
leftovers (which, I think, will be orders of magnitude larger than the
initial discussions :-))

-- 
Regards,

Laurent Pinchart

Re: [MAINTAINERS SUMMIT] Reflections on kernel development processes

[Jonathan Corbet]

On Fri, 11 Oct 2019 15:30:29 +0200
Dmitry Vyukov <dvyukov@google.com> wrote:

> For Tue I see for potentially interested people:
> 09:00 - 09:20 Linus keynote
> 11:00 - 12:00 Shuah Office Hours
> 12:20 - 12:55 Rafael talk

16:20-16:55 Jonathan Corbet :)

> Anything else?
> 
> Tue is fine with me, I just did not want to leave it to the last day
> because most likely it will take me a day to digest all info, so I
> will be able to reply something meaningful only the next day. It would
> be good to have some in-person conversations next day as well.
> 
> Tue 14:00-15:00?

I'd like to be there too.

jon

Re: [MAINTAINERS SUMMIT] Reflections on kernel development processes

[Nicolas Belouin]

On 10/11/19 3:30 PM, Dmitry Vyukov wrote:
> On Fri, Oct 11, 2019 at 3:19 PM Christian Brauner
> <christian.brauner@ubuntu.com> wrote:
>> On Fri, Oct 11, 2019 at 09:18:12AM -0400, Steven Rostedt wrote:
>>> On Fri, 11 Oct 2019 11:33:35 +0200
>>> Dmitry Vyukov <dvyukov@google.com> wrote:
>>>
>>>>> Let's try to figure out time then.
>>>>> I propose the first day to not leave it to the last day and we may
>>>>> need some follow up discussions.
>>>>> On Monday I see Greg's keynote at 09:50 - 10:10. Steven's talk at
>>>>> 12:20 - 12:55. Sasha's talk at 12:20 - 12:55. Any other conflicts for
>>>>> people who want to attend?
>>>>> What about 10:30-11:30 (this covers a coffee break at 10:45-11:00, can
>>>>> prolong it to 12:00)? Any other time slot proposals?
>>>> +Christian also expressed interest, his talks are 14:25 - 15:50 and
>>>> 16:20 - 16:55 on Monday.
>>> If everyone's giving a talk on Monday, can we do this meeting on
>>> Tuesday?
>>>
>>> It's much easier to concentrate if you don't have to worry about a talk
>>> you are about to give (or just finished).
>> Seconded. :)
> It's not that there is nothing on Tue, it's just that I looked only on
> Monday schedule :)
> For Tue I see for potentially interested people:
> 09:00 - 09:20 Linus keynote
> 11:00 - 12:00 Shuah Office Hours
> 12:20 - 12:55 Rafael talk
> Anything else?
>
> Tue is fine with me, I just did not want to leave it to the last day
> because most likely it will take me a day to digest all info, so I
> will be able to reply something meaningful only the next day. It would
> be good to have some in-person conversations next day as well.
>
> Tue 14:00-15:00?
I'd like to be here too and this proposition fits well in my planned
schedule

Re: [MAINTAINERS SUMMIT] Reflections on kernel development processes

[Daniel Vetter]

On Mon, Oct 14, 2019 at 9:42 AM Nicolas Belouin
<nicolas.belouin@gandi.net> wrote:
>
>
>
> On 10/11/19 3:30 PM, Dmitry Vyukov wrote:
> > On Fri, Oct 11, 2019 at 3:19 PM Christian Brauner
> > <christian.brauner@ubuntu.com> wrote:
> >> On Fri, Oct 11, 2019 at 09:18:12AM -0400, Steven Rostedt wrote:
> >>> On Fri, 11 Oct 2019 11:33:35 +0200
> >>> Dmitry Vyukov <dvyukov@google.com> wrote:
> >>>
> >>>>> Let's try to figure out time then.
> >>>>> I propose the first day to not leave it to the last day and we may
> >>>>> need some follow up discussions.
> >>>>> On Monday I see Greg's keynote at 09:50 - 10:10. Steven's talk at
> >>>>> 12:20 - 12:55. Sasha's talk at 12:20 - 12:55. Any other conflicts for
> >>>>> people who want to attend?
> >>>>> What about 10:30-11:30 (this covers a coffee break at 10:45-11:00, can
> >>>>> prolong it to 12:00)? Any other time slot proposals?
> >>>> +Christian also expressed interest, his talks are 14:25 - 15:50 and
> >>>> 16:20 - 16:55 on Monday.
> >>> If everyone's giving a talk on Monday, can we do this meeting on
> >>> Tuesday?
> >>>
> >>> It's much easier to concentrate if you don't have to worry about a talk
> >>> you are about to give (or just finished).
> >> Seconded. :)
> > It's not that there is nothing on Tue, it's just that I looked only on
> > Monday schedule :)
> > For Tue I see for potentially interested people:
> > 09:00 - 09:20 Linus keynote
> > 11:00 - 12:00 Shuah Office Hours
> > 12:20 - 12:55 Rafael talk
> > Anything else?
> >
> > Tue is fine with me, I just did not want to leave it to the last day
> > because most likely it will take me a day to digest all info, so I
> > will be able to reply something meaningful only the next day. It would
> > be good to have some in-person conversations next day as well.
> >
> > Tue 14:00-15:00?
> I'd like to be here too and this proposition fits well in my planned
> schedule

Me too on both interest and slot.
-Daniel
-- 
Daniel Vetter
Software Engineer, Intel Corporation
+41 (0) 79 365 57 48 - http://blog.ffwll.ch

Re: [MAINTAINERS SUMMIT] Reflections on kernel development processes

[Dmitry Vyukov]

On Mon, Oct 14, 2019 at 9:53 AM Daniel Vetter <daniel@ffwll.ch> wrote:
>
> On Mon, Oct 14, 2019 at 9:42 AM Nicolas Belouin
> <nicolas.belouin@gandi.net> wrote:
> >
> >
> >
> > On 10/11/19 3:30 PM, Dmitry Vyukov wrote:
> > > On Fri, Oct 11, 2019 at 3:19 PM Christian Brauner
> > > <christian.brauner@ubuntu.com> wrote:
> > >> On Fri, Oct 11, 2019 at 09:18:12AM -0400, Steven Rostedt wrote:
> > >>> On Fri, 11 Oct 2019 11:33:35 +0200
> > >>> Dmitry Vyukov <dvyukov@google.com> wrote:
> > >>>
> > >>>>> Let's try to figure out time then.
> > >>>>> I propose the first day to not leave it to the last day and we may
> > >>>>> need some follow up discussions.
> > >>>>> On Monday I see Greg's keynote at 09:50 - 10:10. Steven's talk at
> > >>>>> 12:20 - 12:55. Sasha's talk at 12:20 - 12:55. Any other conflicts for
> > >>>>> people who want to attend?
> > >>>>> What about 10:30-11:30 (this covers a coffee break at 10:45-11:00, can
> > >>>>> prolong it to 12:00)? Any other time slot proposals?
> > >>>> +Christian also expressed interest, his talks are 14:25 - 15:50 and
> > >>>> 16:20 - 16:55 on Monday.
> > >>> If everyone's giving a talk on Monday, can we do this meeting on
> > >>> Tuesday?
> > >>>
> > >>> It's much easier to concentrate if you don't have to worry about a talk
> > >>> you are about to give (or just finished).
> > >> Seconded. :)
> > > It's not that there is nothing on Tue, it's just that I looked only on
> > > Monday schedule :)
> > > For Tue I see for potentially interested people:
> > > 09:00 - 09:20 Linus keynote
> > > 11:00 - 12:00 Shuah Office Hours
> > > 12:20 - 12:55 Rafael talk
> > > Anything else?
> > >
> > > Tue is fine with me, I just did not want to leave it to the last day
> > > because most likely it will take me a day to digest all info, so I
> > > will be able to reply something meaningful only the next day. It would
> > > be good to have some in-person conversations next day as well.
> > >
> > > Tue 14:00-15:00?
> > I'd like to be here too and this proposition fits well in my planned
> > schedule
>
> Me too on both interest and slot.
> -Daniel


Re logistics, it would be nice to get some room for this. Looking at
the program committee [1], I see Greg, Steve, Shuah there. Do you know
if it's possible to get a room? And who can help to organize it?

[1] https://events19.linuxfoundation.org/events/open-source-summit-europe-2019/program/program-committee/

Re: [MAINTAINERS SUMMIT] Reflections on kernel development processes

[Konstantin Ryabitsev]

On Tue, Oct 15, 2019 at 09:31:23AM +0200, Dmitry Vyukov wrote:
>> Me too on both interest and slot.
>
>Re logistics, it would be nice to get some room for this. Looking at
>the program committee [1], I see Greg, Steve, Shuah there. Do you know
>if it's possible to get a room? And who can help to organize it?

There are usually "hacker space" rooms made available during OSS events, 
so finding a place to meet for an hour shouldn't be too hard. I will 
email the LF events folks to double-check and follow up here.

-K

Re: [MAINTAINERS SUMMIT] Reflections on kernel development processes

[Dmitry Vyukov]

On Fri, Oct 11, 2019 at 4:16 AM Konstantin Ryabitsev
<konstantin@linuxfoundation.org> wrote:
>
> On Tue, 8 Oct 2019 at 12:51, Konstantin Ryabitsev
> <konstantin@linuxfoundation.org> wrote:
> > >Hi Konstantin,
> > >
> > >Are you going to attend OSS Summit EU? If yes, I think we should sync
> > >up there. I know Dave and Greg are there too. Perhaps we could
> > >schedule some kind of semi-formal meeting?
> >
> > Unfortunately, no -- we have limited budget for traveling and I burned
> > through mine by going to Lisbon. I'd be happy to work on a draft
> > document that expands on my tooling vision (I'm doing so anyway), and I
> > can time it to be ready before the summit so you can review and improve
> > it. My hope is to submit it formally to the LF by the end of the month.
>
> Oh, hey, change of plans -- looks like I *will* be going to Lyon after
> all. I think sitting down together for a semi-formal meeting sounds
> like a good idea, considering all the active discussions happening
> lately. Should we hash out a tentative agenda about topics that we
> should and shouldn't cover? Who should be in attendance?

Re agenda.
For me the main bullet on the agenda would be figuring out high-level
final destination for the effort. I think we need to take a step back,
forget about implementation costs for a moment and try to imagine how
we want kernel development process to look like in 5-10 years. Once we
do that, most likely we will decide to cut some corners and compromise
on practical grounds, and definitely have some incremental process of
getting there. But still making small steps with a clear vision of the
final destination is very different from making conflicting small
steps in random directions.
It should help us to avoid conflicting, duplicate and throwaway
efforts. Personally I would be much more willing to contribute to such
unified effort with clear destination rather than random projects used
by a fraction of the community. It seems that lots of current efforts
suffer from an inability to get enough resources behind because of an
unofficial status. If we discard something based on the implementation
costs, it would be useful to make that an explicit informed decision.
Implementing something solid from scratch may actually be less effort
than trying to connect together two pieces that were never meant to
work together, it's important to consider short term vs long term
costs here.

But for this final destination we also need to figure out concrete
technical foundations: email vs git vs ssb vs forge vs something else;
identity/user account story, etc. But it's not just picking one name,
because e.g. for ssb it's currently unclear to me how to deal with its
inability to provide global data consistency. But git-based solution
has comparable fundamental problems too.
Without these decisions we won't be able to make any actual forward progress.

Re: [MAINTAINERS SUMMIT] Reflections on kernel development processes

[Laurent Pinchart]

Hi Konstantin,

On Thu, Oct 10, 2019 at 10:16:42PM -0400, Konstantin Ryabitsev wrote:
> On Tue, 8 Oct 2019 at 12:51, Konstantin Ryabitsev wrote:
> > > Hi Konstantin,
> > >
> > > Are you going to attend OSS Summit EU? If yes, I think we should sync
> > > up there. I know Dave and Greg are there too. Perhaps we could
> > > schedule some kind of semi-formal meeting?
> >
> > Unfortunately, no -- we have limited budget for traveling and I burned
> > through mine by going to Lisbon. I'd be happy to work on a draft
> > document that expands on my tooling vision (I'm doing so anyway), and I
> > can time it to be ready before the summit so you can review and improve
> > it. My hope is to submit it formally to the LF by the end of the month.
> 
> Oh, hey, change of plans -- looks like I *will* be going to Lyon after
> all. I think sitting down together for a semi-formal meeting sounds
> like a good idea, considering all the active discussions happening
> lately. Should we hash out a tentative agenda about topics that we
> should and shouldn't cover? Who should be in attendance?

I'll be there as well, and would like to attend if possible (assuming no
agenda clash).

Regarding the agenda, I think the most urgent task now is figuring out
what we want to achieve and find an as large as possible consensus. The
"centralisation resistance" movement is quite fragmented, leading to
little traction and a high level of doubt from third parties that we
would be able to achieve anything.

Crucial to test effort is, I believe, a need to acknowledge all the pain
experienced by developers and maintainers who are considering moving to
forges, and making sure we take into account their use cases.

I also would like to use that opportunity to discuss what we really mean
by "e-mail workflow". Many developers on this list have expressed a need
for an e-mail-compatible solution, and I really wonder if they meant
e-mail as such, or if e-mail is more of an umbrella term that summarises
the current advantages of e-mail that could also be provided by new
tools that we would develop.

-- 
Regards,

Laurent Pinchart

Re: [MAINTAINERS SUMMIT] Reflections on kernel development processes

[Theodore Y. Ts'o]

On Fri, Oct 11, 2019 at 04:29:21PM +0300, Laurent Pinchart wrote:
> I also would like to use that opportunity to discuss what we really mean
> by "e-mail workflow". Many developers on this list have expressed a need
> for an e-mail-compatible solution, and I really wonder if they meant
> e-mail as such, or if e-mail is more of an umbrella term that summarises
> the current advantages of e-mail that could also be provided by new
> tools that we would develop.

Unfortunately, it's unlikely I'll be able to arrange a trip to Lyon at
this point, so hopefully folks can take good notes and a summary
posted to workflows@ afterwards?

As far as "e-mail workflow" is concerned, I'd suggest that a
reasonable place to end up is that things that currently can be done
via e-mail should be continue to be viable over e-mail.  That way, we
can do a graduate cutover without people who are still using e-mail
don't lose functionality.   This means:

1) People should be able to submit a patch via e-mail
2) People should be able to comment on a patch via e-mail
       (with the comments reflected on the web review UI)
3) Comments made via the web review UI, and changes in the patch
   status (the patch gets a +1 or +2 rating; the patch gets submitted
   into a git tree, etc.) should be reflected via e-mail.

Gerrit does #3 already.  Patchwork does #1 and #2.  There has been a
proof of concept for #2 a Gerrit-like tool where the tool can look at
the quoted patch hunk, or the quoted texted which is being replied to,
which allows the comment to be assigned to the correct place in the
web review UI.

I do *not* think that administrative actions (e.g., those those
currently being done via the patchwork web or CLI UI) should be doable
via e-mail, because e-mail is painful to authenticate.  It's true that
the Debian Bug Tracking System (BTS) uses no authentication at all,
but for projects (like the Linux kernel) which are much higher
visibility, the ability to have patches be marked as abandoned or
automatically merged into a git repository without any authentication
at all is ripe for abuse.

A similar discussion should be had over what sort of operations need
to be off-line versus only doable when you are on-line and connected
to some service.  For example, if you are going to request that tests
get run on a test branch, to the extent that the tests are going to be
run on some set of test hardware, or test VM's, you have to be on-line
anyway.

Other operations, such as signing off on a patch and marking as
approved, probably *do* make sense to be doable when you are
disconnected from the internet (for example, while you are on an
airplane flying between North America and Lyon.  :-)

Cheers,

						- Ted