Pain points in Git's patch flow

Pain points in Git's patch flow

[Jonathan Nieder]

Hi,

I'd like to introduce Raxel (cc-ed), who is starting an internship
this June with the Git team at Google.

He'll be working on a bit of an experimental project: we want to take
Patchwork[1], which in principle can be a helpful addition to a
mailing list centric workflow[2], and improve it to be something that
people in the Git open source project get day-to-day benefit from.
Raxel's previous successes in making changes to tools to support a
better user experience make me excited for the potential for this
work.

Anyway, yesterday[3] Junio, Taylor, and Emily were discussing how to
encourage more reviews:

 <gitster> this week, i'd be thinking about ways to get topics, that
           are not reviewed sufficiently, reviewed. I can act as the
           last-resort fallback reviewer, but that's not sufficient.
 <ttaylorr> gitster: I share your concern.
 <nasamuffin> gitster: yep, agree, on both counts

That reminded me that it would be useful preparation to collect
descriptions of pain points we are having with our existing patch
flow.  For example:

- As a reviewer, I want to be able to easily find a series that needs
  review.  Using patchwork, I can see some recent patch series; or
  using a hierarchical threaded mail reader, I can find a neglected
  thread or one that seems to be likely to have an interesting
  discussion going on.  But without reading in detail, there is no
  easy way to see whether the series has reached a review, whether
  someone else intends to review it, and what the author believes its
  status to be.

- Relatedly, as a patch author or reviewer, I want to be able to
  easily tell whether a topic has been sufficiently reviewed.  Today,
  the signals for this are implicit: I have to judge consensus, or to
  check the Git repository for whether the patch has been merged, or
  to check the maintainer's latest "What's cooking in git.git"
  message.

- As a potential reviewer or interested user, I want to be able to
  follow all relevant discussion for a patch series, while also
  having the ability to stop following it if the discussion goes on
  too long and starts overwhelming my email inbox.  Today, I can join
  the discussion and then (1) it is hit-or-miss whether the patch
  author ccs me on later iterations of the patch and (2) there is no
  easy way without aggressive email filtering to stop watching it if
  I am cc-ed.

- After having diagnosed an issue to be due to a patch, I want to be
  able to easily find all relevant review discussion.  Today I can
  use the mailing list archive[4] or patchwork to find review
  discussion on the latest version of the series that patch was in,
  but tracing back to previous iterations of that same series can be
  non-trivial.  Moreover, if I'm interested in a particular puzzling
  line of code, finding which iteration introduced it can take a long
  time.

Those four are important in my everyday life.  Questions:

 1. What pain points in the patch flow for git.git are important to
    you?

 2. What tricks do you use to get by with those existing pain points?

 3. Do you think patchwork goes in a direction that is likely to help
    with these?

 4. What other tools would you like to see that could help?

Thanks,
Jonathan

[1] http://jk.ozlabs.org/projects/patchwork/; you can see an instance
for Git at https://patchwork.kernel.org/project/git/list/
[2] https://kernel-recipes.org/en/2016/talks/patches-carved-into-stone-tablets/,
https://www.kernel.org/pub/software/scm/git/docs/gitworkflows.html#_patch_workflow
[3] https://colabti.org/irclogger/irclogger_log/git-devel?date=2021-04-12#l40
[4] https://lore.kernel.org/git/

Re: Pain points in Git's patch flow

[Bagas Sanjaya]

On 14/04/21 13.13, Jonathan Nieder wrote:
> Those four are important in my everyday life.  Questions:
> 
>   1. What pain points in the patch flow for git.git are important to
>      you?

There is no lists of "beginner-friendly" issues that can be worked on by
new contributors. They had to search this ML archive for bug report
issues and determine themselves which are beginner-friendly.

>   2. What tricks do you use to get by with those existing pain points?
> 
>   3. Do you think patchwork goes in a direction that is likely to help
>      with these?

No, unrelated to beginner-friendly issues above.

>   4. What other tools would you like to see that could help?

Some sort of bug tracker systems like Bugzilla (used by Linux kernel
and many other projects) and Debbugs [1] (which is mail-centric and used by
Debian for its BTS).

[1]: https://bugs.debian.org/debbugs-source/

-- 
An old man doll... just what I always wanted! - Clara

Re: Pain points in Git's patch flow

[Junio C Hamano]

Bagas Sanjaya <bagasdotme@gmail.com> writes:

> There is no lists of "beginner-friendly" issues that can be worked on by
> new contributors. They had to search this ML archive for bug report
> issues and determine themselves which are beginner-friendly.

Yeah, looking for "#leftoverbits" or "low-hanging" on the list
archive is often cited as a way, and it does seem easy enough to
do.  You go to https://lore.kernel.org/git/, type "leftoverbits"
or "low-hanging" in the text input and press SEARCH.

But that is only half of the story.

Anybody can throw random ideas and label them "#leftoverbits" or
"low-hanging fruit", but some of these ideas might turn out to be
ill-conceived or outright nonsense.  Limiting search to the
utterances by those with known good taste does help, but as a
newbie, you do not know who these people with good taste are.

It might help to have a curated list of starter tasks, but I suspect
that they tend to get depleted rather quickly---by definition the
ones on the list are easy to do and there is nothing to stop an
eager newbie from eating all of them in one sitting X-(.

So, I dunno.  We seem to suffer from the same lack of good starter
tasks before each GSoC begins.

Re: Pain points in Git's patch flow

[Junio C Hamano]

Junio C Hamano <gitster@pobox.com> writes:

> So, I dunno.  We seem to suffer from the same lack of good starter
> tasks before each GSoC begins.

And long after sending the response, I realize that this has very
little to do with "Git's patch flow" issue that you wanted to
discuss.  Helping newbies wet their toes may be a topic worth
discussing, but that is not the focus of this thread.

Let me send a response to describe my pain points separately.
Sorry for the noise.

Re: Pain points in Git's patch flow

[Denton Liu]

Hi Bagas,

On Wed, Apr 14, 2021 at 02:22:51PM +0700, Bagas Sanjaya wrote:
> 
> On 14/04/21 13.13, Jonathan Nieder wrote:
> > Those four are important in my everyday life.  Questions:
> > 
> >   1. What pain points in the patch flow for git.git are important to
> >      you?
> 
> There is no lists of "beginner-friendly" issues that can be worked on by
> new contributors. They had to search this ML archive for bug report
> issues and determine themselves which are beginner-friendly.

An unofficial and semi-curated list of issues exists at [0]. I've seen
many people new to git development pick up some beginner-friendly issues
from there.

-Denton

[0]: https://github.com/gitgitgadget/git/issues

Re: Pain points in Git's patch flow

[Junio C Hamano]

Jonathan Nieder <jrnieder@gmail.com> writes:

> That reminded me that it would be useful preparation to collect
> descriptions of pain points we are having with our existing patch
> flow.

As the maintainer, I want to ensure that what I queue on 'seen' is
kept reasonably up to date.  Not picking up the latest every time a
topic is rerolled is OK, but before declaring the topic will hit
'next' in a few days, it must be (1) the latest and (2) the greatest
(meaning: what reviewers are happy with).

This requires a few features from any tracking system.  It must be
able to:

 - tell which round of patches are in 'seen'.

 - tell which e-mail messages are the newer round than what is
   queued, and which ones are the latest round.

 - tell which patch have been commented on, and been updated in
   response.

 - tell which patch have been positively accepted with an Ack or a
   Reviewed-by, and if a patch in a newer round is identical to an
   already accepted one (in which case, reviewers would not bother
   to send "this step still looks good to me").

The system I use for the first two points is to rely on the list
archive and the mapping from each individual commit made out of a
patch on the list (implemented as git notes, that records a blob
with the Message-Id for each commit [*1*]).  So

   $ git log --notes=amlog --no-merges --oneline master..$topic

would give a list of commits with the original Message-ID, and I can
see which piece of e-mail each commit came from.

The third and fourth are maintained mostly manual, with me keeping
notes in the draft of "What's cooking" report (which I send out from
time to time).  Having to notice, pick up and squeeze in Acked-by's
and Reviewed-by's is quite painful and cumbersome, especially for a
long series, and the buggy "git rebase -x" does not help, either
[*2*].

If we run a patchwork instance for our project, the first two could
be largely automated.  Automation built around Patchwork should be
able to, or at least should be able to help me to:

 - notice when a new round of an existing topic is posted.

 - fetch the "amlog" notes, together with a copy of daily 'seen', to
   see if a topic that is queued has an update, and notify me and
   others when the topic queued is stale [*3*].

 - tie a step in the latest round with a corresponding step in the
   previous round, and show Ack's and Reviewed-By's that are still
   valid [*4*].


[Footnotes]

*1* "git fetch https://github.com/gitster/git +refs/notes/amlog:refs/notes/amlog"
    should give you a copy of this database.  Then, for example you
    can ask where a commit came from:

    $ git show -s --notes=amlog format="%N%s" 61a7660516
    Message-Id: <pull.1001.git.git.1618254757074.gitgitgadget@gmail.com>
    reftable: document an alternate cleanup method on Windows

    Note that this is not a one-to-one mapping.  I may initially
    apply patches to an inappropriate base and push the integration
    result out that has it in 'seen', but I may realize that the
    series needs to be queued on a different commit and rebase the
    topic the next day.  Both commits before and after such a
    rebasing have come from the single piece of e-mail, so you can
    say "this commit came from this message", but it is impossible
    to expect a single answer to "which commit is the result of this
    message"---there will be multiple.

    Strictly speaking, when two rounds of the same topic had patches
    that were unchanged between the iterations in their earliest
    parts, two pieces of e-mail may convey the same patch, so in the
    ideal world, it might be more useful to record "this commit came
    from this and that messges, both of which record an identical
    patch".  I currently do not do so, though.

*2* It would be ideal if "rebase -i -x 'add-trailer -r peff@'" can
    be used to stop at each commit, run the 'add-trailer -r peff@'
    script that amends HEAD to add "Reviewed-by: peff@", and
    continue, while honoring the "notes.rewriteref" configuration
    variable (in my repository, set to "refs/notes/amlog").  That
    way, I can queue with "git am", at which time "amlog" gets
    populated to map each commit to the original message, find
    Reviewed-by: and do the above rebase, while carrying the message
    IDs to resulting commits.  Alas, "rebase -i -x" is buggy and
    loses the notes during this process (doing s/pick/reword/ and
    manually squeezing Reviewed-by: into the log message is a poor
    but workable workaround).
    cf. https://lore.kernel.org/git/xmqq8s6tcuxc.fsf@gitster.g/

*3* It is perfectly normal if a topic is left stale, if the newer
    iteration breaks integration.  So the stale notification going
    directly to a contributor from an automated system would not
    help very much, but it needs to come with the reason why it is
    kept out of 'seen', which must be supplied by human, not by an
    automation.

    If an automation around Patchwork can send, instead of "hey,
    here is an updated series available" notification, a mbox
    readily usable by "git am -s3c" to me, with Acked-by's and
    Reviewed-by's already incorporated, that might be ideal (these
    trailers may have to be filtered/curated to avoid spams,
    though).
 

*4* Judging a step in the latest round and the corresponding step in
    the previous round has not substantially changed may not be
    easily automatable, and carrying Ack's and Reviewed-by's forward
    would require human curator of the "patchwork" database.

Re: Pain points in Git's patch flow

[Son Luong Ngoc]

Hi there,

I'm not a regular contributor but I have started to subscribe to the
Git's Mailing List recently.  So I thought it might be worth sharing my
personal view on this.

After writting all the below, I do realize that I have written quite a
rant, some of which I think some might consider to be off topic.  For
that, I do want to appologize before hand.

 Tue, Apr 13, 2021 at 11:13:26PM -0700, Jonathan Nieder wrote:
> Hi,
> 
...
> 
> Those four are important in my everyday life.  Questions:
> 
>  1. What pain points in the patch flow for git.git are important to
>     you?

There are several points I want to highlight:

1. Issue about reading the Mailing List:

- Subscribing to Git's Mailing List is not trivial:
  It takes a lot of time to setup the email subscription.  I remember
  having to google through a few documents to get my subscription
  working.

- And even after having subscribed, I was bombarded with a set
  of spam emails that was sent to the mailing list address.  These spams
  range anywhere from absurd to disguising themselves as legitimate
  users trying to contact you about a new shiny tech product.

2. Issue about joining the conversation in the Maling List:

- Setting up email client to reply to the Mailing List was definitely
  not trivial.  It's not trivial to send a reply without subscribing to
  the ML(i.e. using a Header provided from one of the archive).
  The list does not accept HTML emails, which many clients
  use as default format.  Getting the formatting to work for line
  wrapping is also a challenge depends on the client that you use.

- It's a bit intimidating to ask 'trivial questions' about the patch and
  create 'noise' in the ML.

3. Isssue with archive:

- I don't find the ML archive trivial for new comers.  It took me a bit
  of time to realize: 'Oh if I scroll to bottom and find the "Thread 
  overview" then I can navigate a mailing thread a lot easier'.

- The lack of labeling / categorization that I can filter while browsing
  through the archive make the 'browse' experience to be quite
  unpleasant.  Search is one way to do it, but a new comers would not be
  knowledgable enough to craft search query to get the archive view just
  right.  Perhaps a way to provide a curate set of categories would be
  nice.

- Lost track of issues / discussion:
  A quick example would be me searching for Git's zstd support
  recently with 

  > https://lore.kernel.org/git/?q=zstandard 

  and got next to no relevant result.  However if I were to query

  > 'https://lore.kernel.org/git/?q=zstd'

  then a very relevant thread from Peff appeared.  I think this could be
  avoided if the search in ML archive do more than just matching exact
  text.

4. Lack of way to run test suite / CI:

  It would be nice if we can discuss patches while having CI result as
  part of the conversation.  Right now mostly I see that we have to
  manually running benchmarks/tests and share the paste the results.

  But for folks who don't have a dev environment ready at hand (new
  comers, during travel with only phone access), it would be nice to
  have a way to run tests without a dev environment.

  This was mostly solved in the context of works spent on Github's
  Action Workflow.  But if we are discussing about pure patch flow, this
  is a miss.

>  2. What tricks do you use to get by with those existing pain points?

For (1):
- I had to invested a lot of time into setting up a set of Gmail search
  filter.  Move mails with topics that Im interested in into a special
  tag while the rest into archive.  Regularly check if anything
  interesting went to archive by accident.

For (2):
- I had to setup Mutt + Tmux to have a compatible experience sending
  replies like this one.

- All the patches I have submitted were through
  > https://github.com/gitgitgadget/git/pulls
  and it was not directly trivial to get permission to send email from a
  PR.

For (3):
- Spending time reading git blame / git log / commit message helps
  identifying the keywords I need to refine my search result in the ML
  archive.  This requires some commitments and is a barrier to entry for
  new comers.

- Using service like Github Search or SourceGraph helped a lot in term
  of navigating through the commit message / git blame.

For (4):
- I leverage both Github action and a patch that added Gitlab CI to run
  the test suite.

>  3. Do you think patchwork goes in a direction that is likely to help
>     with these?
>
>  4. What other tools would you like to see that could help?

With all that said, I don't know if patchwork will solve the problems
above.  I do understand that the current patch workflow comes with a
certain set of advantages, and adopting another tool will most likely be
a trade-off.

Personally I have been spending more and more time reading through
git.git via Sourcegraph Web UI and I would love for the search feature
to be able to extend to be able to search in the Mailing List from
relevant commit if possible.  I have also tried both Github's Codespace
and Microsoft's DevContainer to setup an opionated IDE with predefined
tasks that help executing the test suite.  I think these tools (or
their competitors such as GitPod) are quite ideal to quickly onboard
new contributors onto a history-rich codebase such as git.git.

Perhaps some configure a set of sane default, including editor extensions
that would handle email config for first time users.

As for code review and issue tracking toolings, I don't think there are
a perfect solution.  Any solutions: Github PR, Gitlab MR, Gerrit,
Phabricator would come with their own set of tradeoffs.  I like the
prospect of PatchWork gona improve the patch workflow though.  Perhaps I
will give it a try.

> 
> Thanks,
> Jonathan

Thanks,
Son Luong.

Re: Pain points in Git's patch flow

[Eric Wong]

Son Luong Ngoc <sluongng@gmail.com> wrote:
> Hi there,
> 
> I'm not a regular contributor but I have started to subscribe to the
> Git's Mailing List recently.  So I thought it might be worth sharing my
> personal view on this.
> 
> After writting all the below, I do realize that I have written quite a
> rant, some of which I think some might consider to be off topic.  For
> that, I do want to appologize before hand.

Thanks for the feedback, some points below.

>  Tue, Apr 13, 2021 at 11:13:26PM -0700, Jonathan Nieder wrote:
> > Hi,
> > 
> ...
> > 
> > Those four are important in my everyday life.  Questions:
> > 
> >  1. What pain points in the patch flow for git.git are important to
> >     you?
> 
> There are several points I want to highlight:
> 
> 1. Issue about reading the Mailing List:
> 
> - Subscribing to Git's Mailing List is not trivial:
>   It takes a lot of time to setup the email subscription.  I remember
>   having to google through a few documents to get my subscription
>   working.
> 
> - And even after having subscribed, I was bombarded with a set
>   of spam emails that was sent to the mailing list address.  These spams
>   range anywhere from absurd to disguising themselves as legitimate
>   users trying to contact you about a new shiny tech product.

Note that subscription is totally optional.

Gmail's mail filters probably aren't very good, perhaps
SpamAssassin or similar filters can be added locally to improve
things for you.

Spam filtering is a complex topic and Google's monopolistic
power probably doesn't inspire them to do better.

> 2. Issue about joining the conversation in the Maling List:
> 
> - Setting up email client to reply to the Mailing List was definitely
>   not trivial.  It's not trivial to send a reply without subscribing to
>   the ML(i.e. using a Header provided from one of the archive).
>   The list does not accept HTML emails, which many clients
>   use as default format.  Getting the formatting to work for line
>   wrapping is also a challenge depends on the client that you use.

The spam (and phishing) problem would be worse if HTML mail were
accepted.  Obfuscation/misdirection techniques used by spammers
and phishers aren't available in plain-text.

It's also more expensive to filter + archive HTML mail due to
decoding and size overheads, which makes it more expensive for
others to mirror/fork things.

> - It's a bit intimidating to ask 'trivial questions' about the patch and
>   create 'noise' in the ML.

I'm sorry you feel that way.  I understand the Internet and its
persistence (especially with mail archives :x) can have a
chilling effect on people.  I think the way to balance things is
to allow/encourage anonymity or pseudonyms, but some folks here
might disagree with me for copyright reasons.  OTOH, don't ask,
don't tell :)

(I am not speaking as a representative of the git project)

> 3. Isssue with archive:
> 
> - I don't find the ML archive trivial for new comers.  It took me a bit
>   of time to realize: 'Oh if I scroll to bottom and find the "Thread 
>   overview" then I can navigate a mailing thread a lot easier'.

(I'm the maintainer of public-inbox, the archival software you
seem to be referring to).

I'm not sure how to make "Thread overview" easier to find
without cluttering the display near the top.  Maybe I'll try
aria labels in the Subject: link...

> - The lack of labeling / categorization that I can filter while browsing
>   through the archive make the 'browse' experience to be quite
>   unpleasant.  Search is one way to do it, but a new comers would not be
>   knowledgable enough to craft search query to get the archive view just
>   right.  Perhaps a way to provide a curate set of categories would be
>   nice.

Perhaps TODO files/comments in the source tree are acceptable;
or a regularly-posted mail similar to "What's cooking".

Having a centralized website/tracker would give too much power
and influence to people/orgs who run the site.  It would like
either require network access or require learning more software
to synchronize.

> - Lost track of issues / discussion:
>   A quick example would be me searching for Git's zstd support
>   recently with 
> 
>   > https://lore.kernel.org/git/?q=zstandard 
> 
>   and got next to no relevant result.  However if I were to query
> 
>   > 'https://lore.kernel.org/git/?q=zstd'
> 
>   then a very relevant thread from Peff appeared.  I think this could be
>   avoided if the search in ML archive do more than just matching exact
>   text.

I'm planning to support Xapian synonyms for that, but haven't
gotten around to making it configurable+reproducible by admins.
Everything in public-inbox is designed to be reproducible+forkable.

> 4. Lack of way to run test suite / CI:
> 
>   It would be nice if we can discuss patches while having CI result as
>   part of the conversation.  Right now mostly I see that we have to
>   manually running benchmarks/tests and share the paste the results.
> 
>   But for folks who don't have a dev environment ready at hand (new
>   comers, during travel with only phone access), it would be nice to
>   have a way to run tests without a dev environment.

Fwiw, the GCC Farm project gives ssh accounts for all free
software contributors, not just gcc hackers: https://cfarm.tetaneutral.net
Perhaps there's other similar services, too.

Slow down and enjoy travel :)  There's very little in free
software urgent enough to require constant attention.  Email is
well-suited for asynchronous work, and nobody should expect
instant replies.  The always-on nature of the modern Internet
and smartphones increases stress and dangerous situations; so I
hope free software hackers aren't contributing to that.

>   This was mostly solved in the context of works spent on Github's
>   Action Workflow.  But if we are discussing about pure patch flow, this
>   is a miss.
> 
> >  2. What tricks do you use to get by with those existing pain points?
> 
> For (1):
> - I had to invested a lot of time into setting up a set of Gmail search
>   filter.  Move mails with topics that Im interested in into a special
>   tag while the rest into archive.  Regularly check if anything
>   interesting went to archive by accident.
> 
> For (2):
> - I had to setup Mutt + Tmux to have a compatible experience sending
>   replies like this one.

Fwiw, git-send-email works for non-patch mails, too.  I don't
want a monoculture around mutt or any particular clients, either.
(I've never used tmux and don't see why it's necessary, here).

Anyways, thanks again for the feedback.

Re: Pain points in Git's patch flow

[Theodore Ts'o]

On Mon, Apr 19, 2021 at 02:57:54AM +0000, Eric Wong wrote:
> >   But for folks who don't have a dev environment ready at hand (new
> >   comers, during travel with only phone access), it would be nice to
> >   have a way to run tests without a dev environment.
> 
> Fwiw, the GCC Farm project gives ssh accounts for all free
> software contributors, not just gcc hackers: https://cfarm.tetaneutral.net
> Perhaps there's other similar services, too.
> 
> Slow down and enjoy travel :)  There's very little in free
> software urgent enough to require constant attention.  Email is
> well-suited for asynchronous work, and nobody should expect
> instant replies.  The always-on nature of the modern Internet
> and smartphones increases stress and dangerous situations; so I
> hope free software hackers aren't contributing to that.

FWIW, I find the disconnected, e-mail based workflow using a
command-line interface to be *ideal* for working while travelling on
an airplane.  I'm mostly disconnected from the internet, because the
airplane wifi is so slow that you *really* don't want to use a
web-based interface, but I can use offlineimap to sync my e-mail onto
my laptop, and using the command-line interface and the lack of
distractions is great since you really can't surf the web on the
gogoonline's pathetically slow 'net access.

This also means I have an excuse to work on open source projects which
are using e-mail and off-line git, as opposed to $WORK which mandates
the use of gerrit.  :-)

(All of the above applies pre-pandemic, of course.  I've been working
from home and not travelling for the past year+, sigh.)

     	      	  	     	     	  - Ted

P.S.  Also, while working on the road, I find that web-based
interfaces are much more tolerable when I'm at my desk with a 40"
screen.  When I'm using a 13" laptop screen, I much prefer CLI
interfaces.  YMMV, of course.

Re: Pain points in Git's patch flow

[Ævar Arnfjörð Bjarmason]

On Mon, Apr 19 2021, Eric Wong wrote:

> Son Luong Ngoc <sluongng@gmail.com> wrote:
>> [...]
>> 3. Isssue with archive:
>> 
>> - I don't find the ML archive trivial for new comers.  It took me a bit
>>   of time to realize: 'Oh if I scroll to bottom and find the "Thread 
>>   overview" then I can navigate a mailing thread a lot easier'.
>
> (I'm the maintainer of public-inbox, the archival software you
> seem to be referring to).
>
> I'm not sure how to make "Thread overview" easier to find
> without cluttering the display near the top.  Maybe I'll try
> aria labels in the Subject: link...

I'd say the bare-bones style of it is probably jarring to most users
today. I had to check if the site even had any CSS at all.

I.e. I think a more intuitive UI to users today would probably be some
collapsible side-bar on the left of the screen, which would have a
threaded view. The "Archives are clonable" would probably belong in some
"help" tab in such a UI.

Re: Pain points in Git's patch flow

[Eric Wong]

Ævar Arnfjörð Bjarmason <avarab@gmail.com> wrote:
> On Mon, Apr 19 2021, Eric Wong wrote:
> > Son Luong Ngoc <sluongng@gmail.com> wrote:
> >> [...]
> >> 3. Isssue with archive:
> >> 
> >> - I don't find the ML archive trivial for new comers.  It took me a bit
> >>   of time to realize: 'Oh if I scroll to bottom and find the "Thread 
> >>   overview" then I can navigate a mailing thread a lot easier'.
> >
> > (I'm the maintainer of public-inbox, the archival software you
> > seem to be referring to).
> >
> > I'm not sure how to make "Thread overview" easier to find
> > without cluttering the display near the top.  Maybe I'll try
> > aria labels in the Subject: link...
> 
> I'd say the bare-bones style of it is probably jarring to most users
> today. I had to check if the site even had any CSS at all.
> 
> I.e. I think a more intuitive UI to users today would probably be some
> collapsible side-bar on the left of the screen, which would have a
> threaded view. The "Archives are clonable" would probably belong in some
> "help" tab in such a UI.

The plan is to support read-only JMAP, so it's a stable API that
users can build their own displays on top of (of course, NNTP
and IMAP support already exists).

I can't make drastic UI changes such as a sidebar without
breaking things for users who like the current UI.  I only know
about GNOME3 and Digg because they made drastic UI changes that
angered their existing userbase.

The current UI is designed to for a terminal with w3m|lynx since
it's the lowest common denominator.  Graphics drivers/stacks
seem to be most frequently broken thing on GNU/Linux systems, so
it's important users can find patches/configs/help easily with a
text-only browser in order to get graphics working.

Re: Pain points in Git's patch flow

[Eric Wong]

Eric Wong <e@80x24.org> wrote:
> Son Luong Ngoc <sluongng@gmail.com> wrote:
> > 3. Isssue with archive:
> > 
> > - I don't find the ML archive trivial for new comers.  It took me a bit
> >   of time to realize: 'Oh if I scroll to bottom and find the "Thread 
> >   overview" then I can navigate a mailing thread a lot easier'.
> 
> (I'm the maintainer of public-inbox, the archival software you
> seem to be referring to).
> 
> I'm not sure how to make "Thread overview" easier to find
> without cluttering the display near the top.  Maybe I'll try
> aria labels in the Subject: link...

I think I made [thread overview] easier-to-find without adding
more clutter:

	https://public-inbox.org/meta/20210428065522.12795-1-e@80x24.org/

Not sure about title attribute or aria labels (my version of w3m
doesn't support that, yet).  Anyways, an an example of it
deployed:

	https://public-inbox.org/git/20210419025754.GA26065@dcvr/

Re: Pain points in Git's patch flow

[Atharva Raykar]

On 14-Apr-2021, at 11:43, Jonathan Nieder <jrnieder@gmail.com> wrote:
> 
> Hi,
> 
> I'd like to introduce Raxel (cc-ed), who is starting an internship
> this June with the Git team at Google.
> 
> He'll be working on a bit of an experimental project: we want to take
> Patchwork[1], which in principle can be a helpful addition to a
> mailing list centric workflow[2], and improve it to be something that
> people in the Git open source project get day-to-day benefit from.
> Raxel's previous successes in making changes to tools to support a
> better user experience make me excited for the potential for this
> work.
> 
> Anyway, yesterday[3] Junio, Taylor, and Emily were discussing how to
> encourage more reviews:
> 
> <gitster> this week, i'd be thinking about ways to get topics, that
>           are not reviewed sufficiently, reviewed. I can act as the
>           last-resort fallback reviewer, but that's not sufficient.
> <ttaylorr> gitster: I share your concern.
> <nasamuffin> gitster: yep, agree, on both counts
> 
> That reminded me that it would be useful preparation to collect
> descriptions of pain points we are having with our existing patch
> flow.  For example:
> 
> - As a reviewer, I want to be able to easily find a series that needs
>  review.  Using patchwork, I can see some recent patch series; or
>  using a hierarchical threaded mail reader, I can find a neglected
>  thread or one that seems to be likely to have an interesting
>  discussion going on.  But without reading in detail, there is no
>  easy way to see whether the series has reached a review, whether
>  someone else intends to review it, and what the author believes its
>  status to be.
> 
> - Relatedly, as a patch author or reviewer, I want to be able to
>  easily tell whether a topic has been sufficiently reviewed.  Today,
>  the signals for this are implicit: I have to judge consensus, or to
>  check the Git repository for whether the patch has been merged, or
>  to check the maintainer's latest "What's cooking in git.git"
>  message.
> 
> - As a potential reviewer or interested user, I want to be able to
>  follow all relevant discussion for a patch series, while also
>  having the ability to stop following it if the discussion goes on
>  too long and starts overwhelming my email inbox.  Today, I can join
>  the discussion and then (1) it is hit-or-miss whether the patch
>  author ccs me on later iterations of the patch and (2) there is no
>  easy way without aggressive email filtering to stop watching it if
>  I am cc-ed.
> 
> - After having diagnosed an issue to be due to a patch, I want to be
>  able to easily find all relevant review discussion.  Today I can
>  use the mailing list archive[4] or patchwork to find review
>  discussion on the latest version of the series that patch was in,
>  but tracing back to previous iterations of that same series can be
>  non-trivial.  Moreover, if I'm interested in a particular puzzling
>  line of code, finding which iteration introduced it can take a long
>  time.

This is a great initiative!

While I do not have anything new to add in terms of pain
points, I just wanted to let you know that this is definitely
something that would have eased the process of bringing in a
new contributor like me.

> Those four are important in my everyday life.  Questions:
> 
> 1. What pain points in the patch flow for git.git are important to
>    you?

As a new contributor (and also someone new to the patch flow) I
would have especially liked the second and fourth point addressed.
When I was preparing my GSoC proposal, I wanted to gather the
status of a previous contributor's work and even though searching
the mailing list helped, it was hard to immediately know what were
the status of the patches, and which changes got introduced in
which version of the patch series.

Also with my first patch series that I sent to the mailing list,
I initially felt unsure about what the status of my patch was
after a few people discussed over it. The 'implicit signals' is
something that was not immediately obvious to me, and only after
reading other interactions in the mailing list did I start getting
a hold of how I should interpret the responses, and what my next
action should be.

> 2. What tricks do you use to get by with those existing pain points?

In order to learn about a previous patch series and what was added,
I used git blame on the relevant part of the codebase, and tried to
search the commit message in the mailing list archive. From there on
it was just opening a ton of tabs in order to see how the patches
developed over time.

The limitation with this trick is it will work only if the patch
actually landed in the codebase. A part of building my proposal
required me to read a patch that did not get merged, and I had to
just aggressively search the mailing list and hope I managed to catch
everything I wanted.

> 3. Do you think patchwork goes in a direction that is likely to help
>    with these?

I have noticed that a patchwork instance for this mailing list
already exists[1] so I decided to try it out. It definitely
addresses the problem of explicitly identifying the status of a
patch. I also liked that I could search for the previous
contributor that I spoke of and sort his contributions by date.
If I knew this existed, I would have saved a lot of time.

But as you also mentioned, it does not yet help me locate an
older version of a particular patch, and let me observe how it
developed over time. So that would definitely be a welcome
addition.

As Bagas mentioned in the thread, it seems to lend itself well
to identify beginner-friendly tasks. I did not personally have
too much difficulty with those thanks to the GitHub issue tracker
and the Git documentation, but if new contributors are anyway
going to refer to patchwork to study previous patches and learn
from it, it might be helpful to keep beginner issues accessible
there as well. Even a generic labelling system to help categorise
issues will do (a downside being that the labels will have to be
maintained and managed too).

Some small nits:
- The searching capability was not super obvious to me. My
  eyes naturally scan for a search box or search icon, it
  took me a few minutes of fiddling to realise that
  'show patches with' is a link that opens all the search
  filters.
- It would be nice (though probably out of scope) to allow
  me to do a code search in the patches.

> 4. What other tools would you like to see that could help?

(...I don't really have an opinion on this)

> Thanks,
> Jonathan
> 
> [1] http://jk.ozlabs.org/projects/patchwork/; you can see an instance
> for Git at https://patchwork.kernel.org/project/git/list/
> [2] https://kernel-recipes.org/en/2016/talks/patches-carved-into-stone-tablets/,
> https://www.kernel.org/pub/software/scm/git/docs/gitworkflows.html#_patch_workflow
> [3] https://colabti.org/irclogger/irclogger_log/git-devel?date=2021-04-12#l40
> [4] https://lore.kernel.org/git/

[1] https://patchwork.kernel.org/project/git/list/

Re: Pain points in Git's patch flow

[Junio C Hamano]

Jonathan Nieder <jrnieder@gmail.com> writes:

>  3. Do you think patchwork goes in a direction that is likely to help
>     with these?

So here is a real-life example.

Let's say somebody is looking at a "gentle ping" [*1*]

znh> The patch seems to have fallen into the crack.
zhn> Jeff and Junio, willing to help?

How would we figure out what happened to the patch today without
visiting patchwork would be:

 1. Visit the message at lore.kernel.org/git/ [*1*]

 2. Notice that it is a response to a message, and click the link to
    be taken to [*2*]

 3. Notice that nobody commented on the patch.

 4. Type "f:zhening ref-filter" to the search box and search, with
    suspicion that this was an updated version of something.

 5. Click one of them in the result [*3*]

 6. This time, we can tell that this seemed to have had two earlier
    iterations, and after reading the discussion through, the last
    one changed the course in a major way.  Not just a new helper
    introduced in the earlier rounds has gone away, but an existing
    helper got removed.

 7. All comments in the discussion for the earlier two rounds can be
    read as supporting the new direction the latest round takes.

 8. The fact remains that even if the direction has been endorsed
    (see 7. above) nobody took a look at the implementation for the
    latest round.

 9. Make the final verdict.

I use my newsreader to do pretty much the equivalent of the above
without hitting https://lore.kernel.org/git/ but the above is
written to use the web interface, in order to make it reproducible
more easily by anybody on the list.

Now, how can patchwork improve the above reviewer experience, out
of the box and possibly with new helpe rools around it?

I can see #3 would immediately become obvious, and I hope #4-#5
would become unnecessary.

Anything else?

At steps #6 and #7, there is human judgment involved that may not be
automatable, but would there be some mechanism to make it easy to
help these steps if the user visits patchwork (instead of staying
in my newsreader or web interface to the lore archive)?

I am of course not expecting to automate step #9 ;-)  It would be
nice though.

Thanks.


[References]

*1* https://lore.kernel.org/git/CAOLTT8Tis5Yjg8UR0c-i0BnqiFQvLXvDgxUQJ-WcP6jjQPu9cQ@mail.gmail.com/

*2* https://lore.kernel.org/git/pull.928.git.1617975348494.gitgitgadget@gmail.com/

*3* https://lore.kernel.org/git/pull.927.v2.git.1617809209164.gitgitgadget@gmail.com/

Re: Pain points in Git's patch flow

[Junio C Hamano]

Junio C Hamano <gitster@pobox.com> writes:

> So here is a real-life example.
>
> Let's say somebody is looking at a "gentle ping" [*1*]
>
> znh> The patch seems to have fallen into the crack.
> zhn> Jeff and Junio, willing to help?
>
> How would we figure out what happened to the patch today without
> visiting patchwork would be:
> ...
> Now, how can patchwork improve the above reviewer experience, out
> of the box and possibly with new helpe rools around it?

Also, it would be ideal if it is made easy for willing reviewers
with excess bandwidth to preemptively find and review patches that
need reviewing.  I think your original write-up upthread covered
this use case sufficiently.

Thanks.

Re: Pain points in Git's patch flow

[ZheNing Hu]

Haaaa, everyone, I am this example :)
Sorry I haven't checked these mails cc to me for a long time.

Junio C Hamano <gitster@pobox.com> 于2021年4月17日周六 上午3:50写道：
>
> Jonathan Nieder <jrnieder@gmail.com> writes:
>
> >  3. Do you think patchwork goes in a direction that is likely to help
> >     with these?
>
> So here is a real-life example.
>
> Let's say somebody is looking at a "gentle ping" [*1*]
>
> znh> The patch seems to have fallen into the crack.
> zhn> Jeff and Junio, willing to help?
>
> How would we figure out what happened to the patch today without
> visiting patchwork would be:
>
>  1. Visit the message at lore.kernel.org/git/ [*1*]
>
>  2. Notice that it is a response to a message, and click the link to
>     be taken to [*2*]
>
>  3. Notice that nobody commented on the patch.
>
>  4. Type "f:zhening ref-filter" to the search box and search, with
>     suspicion that this was an updated version of something.
>
>  5. Click one of them in the result [*3*]
>
>  6. This time, we can tell that this seemed to have had two earlier
>     iterations, and after reading the discussion through, the last
>     one changed the course in a major way.  Not just a new helper
>     introduced in the earlier rounds has gone away, but an existing
>     helper got removed.
>
>  7. All comments in the discussion for the earlier two rounds can be
>     read as supporting the new direction the latest round takes.
>
>  8. The fact remains that even if the direction has been endorsed
>     (see 7. above) nobody took a look at the implementation for the
>     latest round.
>
>  9. Make the final verdict.
>
> I use my newsreader to do pretty much the equivalent of the above
> without hitting https://lore.kernel.org/git/ but the above is
> written to use the web interface, in order to make it reproducible
> more easily by anybody on the list.
>
> Now, how can patchwork improve the above reviewer experience, out
> of the box and possibly with new helpe rools around it?
>
> I can see #3 would immediately become obvious, and I hope #4-#5
> would become unnecessary.
>

Here are my thoughts:

For the reviewers like Junio, after missing a new patch iteration, need to
review the past history to find the correct patch and related comments
from other reviewers. Just like I once read a github blog saying that
"patch" is also a special object in git. I would like to have a "new" tool
 which can link multiple related patches and comments.

1. Coder need Reviewers' help.
2. This new tool will obtained multiple different patches contents automatically
or coder provided those pathes versions links.
3. This tool will analyze the differences between multiple patches
versions, get all
the reviewers comments and coder comments related to the "patch stream",
organize it into "patch graph".
4. The tool will notify the reviewer(by email or something else) and
show the links
and patch graph or patch range-diff. It can visualize the entire patch process,
 It’s best that comments from different people can be displayed on one page.

In order to be more accurate, I made a picture [*1*].

Using this new tool, reviewers can choose to see or not see the range-diff
and diff in multiple different patch versions, Instead of the range-diff
automatically sent by GGG. When my second patch processing was greatly
changed from the previous one, I have to rebuild a new branch and create a new
PR, this is my pain point.

Thanks!
--
ZheNing Hu

> Anything else?
>
> At steps #6 and #7, there is human judgment involved that may not be
> automatable, but would there be some mechanism to make it easy to
> help these steps if the user visits patchwork (instead of staying
> in my newsreader or web interface to the lore archive)?
>
> I am of course not expecting to automate step #9 ;-)  It would be
> nice though.
>
> Thanks.
>
>
> [References]
>
> *1* https://lore.kernel.org/git/CAOLTT8Tis5Yjg8UR0c-i0BnqiFQvLXvDgxUQJ-WcP6jjQPu9cQ@mail.gmail.com/
>
> *2* https://lore.kernel.org/git/pull.928.git.1617975348494.gitgitgadget@gmail.com/
>
> *3* https://lore.kernel.org/git/pull.927.v2.git.1617809209164.gitgitgadget@gmail.com/

*1* https://github.com/adlternative/git/blob/pic/git-patch-pain-point-solve-idea.png

Re: Pain points in Git's patch flow

[Sebastian Schuberth]

On 2021-04-14 08:13, Jonathan Nieder wrote:

> Those four are important in my everyday life.  Questions:

Thanks for bringing up these questions in a dedicated format. I'll take 
this as an opportunity to share my thoughts on this topic, which have 
accompanied me for quite a while.

>   1. What pain points in the patch flow for git.git are important to
>      you?

Well, it's email-based. As a result it's error prone to things like 
formatting / quoting issues, putting the right people it CC, etc.

I have always wondered why Git core development does not start to make 
use of the Git ecosystem that we have by now, esp. in the form of review 
tools / platforms like GitHub (via pull-requests), GitLab (via 
merge-requests), or Gerrit (via patches). From these, Gerrit would IMO 
be the best fit for Git, due to its capability to cope well with 
rebase-workflows. Those tools avoid things like formatting / quoting 
issues completely, and shift the responsibility of assigning reviewers 
from the contributor to the tool, where people can subscribe to code 
changes or code ownership can be defined and automatically taken into 
account.

Sure, I get that that the contribution workflow to Git core has 
historically grown, but what concerns me is that the efforts to "bridge" 
the contribution workflow to the "modern world" seem to go into the 
wrong direction: Tools like submitgit [1], gitgitgadget [2] and now 
patchwork [3] were created / are considered for use to allow the legacy 
email path workflow to remain, but also allow more "GUI minded" people 
to contribute. While this has worked quite well for some time, and esp. 
gitgitgadget [2] seems to haven gotten popular, I wonder whether it's 
now the time to "swap the default", and make a patch / contribution tool 
with a GUI the standard, and bridge the legacy workflow by using / 
creating tooling that makes it convenient to use those modern tools from 
the CLI, instead of the opposite.

>   2. What tricks do you use to get by with those existing pain points?

None. I simply have stopped contributing to Git core, to be frank.

>   3. Do you think patchwork goes in a direction that is likely to help
>      with these?

No. To me, this is yet another effort that tries to come up with a 
work-around instead of fixing the root cause: It tries to lift the 
limitations of an email-based contribution workflow instead of getting 
rid of the email-based contribution workflow altogether.

>   4. What other tools would you like to see that could help?

Currently, only Gerrit [4] comes to my mind, as a complete substitute 
for the email-based contribution workflow.

[1] https://github.com/rtyley/submitgit
[2] https://github.com/gitgitgadget/gitgitgadget
[3] http://jk.ozlabs.org/projects/patchwork
[4] https://www.gerritcodereview.com

-- 
Sebastian Schuberth

Re: Pain points in Git's patch flow

[Ævar Arnfjörð Bjarmason]

On Sun, Apr 18 2021, Sebastian Schuberth wrote:

> On 2021-04-14 08:13, Jonathan Nieder wrote:
>
>> Those four are important in my everyday life.  Questions:
>
> Thanks for bringing up these questions in a dedicated format. I'll
> take this as an opportunity to share my thoughts on this topic, which
> have accompanied me for quite a while.

And thank you for participating in the discussion. I think it's
especially valuable to get a viewpoint like yours, i.e. someone who (per
this E-Mail below) gave up in frustration with the current development
flow.

The below isn't meant as a retort, but to hopefully clarify things a
bit.

>>   1. What pain points in the patch flow for git.git are important to
>>      you?
>
> Well, it's email-based. As a result it's error prone to things like
> formatting / quoting issues, putting the right people it CC, etc.
>
> I have always wondered why Git core development does not start to make
> use of the Git ecosystem that we have by now, esp. in the form of
> review tools / platforms like GitHub (via pull-requests), GitLab (via 
> merge-requests), or Gerrit (via patches). From these, Gerrit would IMO
> be the best fit for Git, due to its capability to cope well with 
> rebase-workflows. Those tools avoid things like formatting / quoting
> issues completely, and shift the responsibility of assigning reviewers 
> from the contributor to the tool, where people can subscribe to code
> changes or code ownership can be defined and automatically taken into 
> account.

I think it's important not to conflate tooling issues with social
issues. It's not that we e.g. couldn't whip up a quick script to
round-robin randomly assign reviewers on the basis of topics Junio has
picked up, which is basically the function of some of these "open a MR
end get on the review train" tools.

Rather it's that it's a volunteer project and people work on what
they're interested in.

So maybe having assigned reviewers would help move things along. But I
wonder if it wouldn't also lead down the rut of PRs/MRs languishing for
months, because the reviewers just want to spend their time in some
other way.

I.e. the design of many of these tools in this regard assumes you have a
workforce, not the cat-herding problem of volunteers working on whatever
strikes their fancy.

> Sure, I get that that the contribution workflow to Git core has
> historically grown, but what concerns me is that the efforts to
> "bridge" the contribution workflow to the "modern world" seem to go
> into the wrong direction: Tools like submitgit [1], gitgitgadget [2]
> and now patchwork [3] were created / are considered for use to allow
> the legacy email path workflow to remain, but also allow more "GUI
> minded" people to contribute. While this has worked quite well for
> some time, and esp. gitgitgadget [2] seems to haven gotten popular, I
> wonder whether it's now the time to "swap the default", and make a
> patch / contribution tool with a GUI the standard, and bridge the
> legacy workflow by using / creating tooling that makes it convenient
> to use those modern tools from the CLI, instead of the opposite.

I think characterizing E-Mail as a "legacy" workflow isn't accurate. All
of these proposed alternatives involve moving away from something that's
a distributed system today (E-Mail infrastructure, local clients), to
what's essentially some website run by a centralized entity, in some
cases proprietary.

Even in cases where the tool itself isn't proprietary (e.g. GitLab
instead of GitHub) using GitHub/GitLab/Gerrit/Atlassian Bitbucket
etc. means having some centralized infrastructure somewhere holding a
bunch of data only the operator of that infrastructure can realistically
access.

So really basic things that are comparatively trivial with E-Mail
(e.g. "I think the search sucks, try another client") run up against a
brick wall with those tools.

And to e.g. as one good example to use (as is the common convention on
this list) git-range-diff to display a diff to the "last rebased
revision" would mean some long feature cycle in those tools, if they're
even interested in implementing such a thing at all.

Because we're E-Mail based that's just something some people started
using (well, it was called "tbdiff" then), some others picked it up etc.

Which is not to say that one can't argue that on balance using those
tools isn't better overall, I'm just responding to the characterization
of E-Mail based development as "legacy", or something those tools
supersede. I think it's better to think of them as orthagonal ways to
reach similar aims.

Re: Pain points in Git's patch flow

[Eric Wong]

Ævar Arnfjörð Bjarmason <avarab@gmail.com> wrote:
> On Sun, Apr 18 2021, Sebastian Schuberth wrote:
> 
> > On 2021-04-14 08:13, Jonathan Nieder wrote:
> >
> >> Those four are important in my everyday life.  Questions:
> >
> > Thanks for bringing up these questions in a dedicated format. I'll
> > take this as an opportunity to share my thoughts on this topic, which
> > have accompanied me for quite a while.
> 
> And thank you for participating in the discussion. I think it's
> especially valuable to get a viewpoint like yours, i.e. someone who (per
> this E-Mail below) gave up in frustration with the current development
> flow.

Agreed

> The below isn't meant as a retort, but to hopefully clarify things a
> bit.

Some further clarifications on my part below.

<snip some of Ævar's excellent clarifications>

> > Sure, I get that that the contribution workflow to Git core has
> > historically grown, but what concerns me is that the efforts to
> > "bridge" the contribution workflow to the "modern world" seem to go
> > into the wrong direction: Tools like submitgit [1], gitgitgadget [2]
> > and now patchwork [3] were created / are considered for use to allow
> > the legacy email path workflow to remain, but also allow more "GUI
> > minded" people to contribute. While this has worked quite well for
> > some time, and esp. gitgitgadget [2] seems to haven gotten popular, I
> > wonder whether it's now the time to "swap the default", and make a
> > patch / contribution tool with a GUI the standard, and bridge the
> > legacy workflow by using / creating tooling that makes it convenient
> > to use those modern tools from the CLI, instead of the opposite.
> 
> I think characterizing E-Mail as a "legacy" workflow isn't accurate. All
> of these proposed alternatives involve moving away from something that's
> a distributed system today (E-Mail infrastructure, local clients), to
> what's essentially some website run by a centralized entity, in some
> cases proprietary.
> 
> Even in cases where the tool itself isn't proprietary (e.g. GitLab
> instead of GitHub) using GitHub/GitLab/Gerrit/Atlassian Bitbucket
> etc. means having some centralized infrastructure somewhere holding a
> bunch of data only the operator of that infrastructure can realistically
> access.

Thanks all for bringing this up.  I should add the mail archives
at lore.kernel.org are backed by public-inbox, thus all mail and
code are completely reproducible by anyone.  It even targets
old, slow, legacy hardware and tries to minimize bandwidth to
benefit the economically-disadvantaged.

Forking is the checks-and-balances system of free software to
prevent any central entity from becoming too powerful (remember:
power corrupts).  DVCS (e.g. git) makes forking easier,
public-inbox uses git to make text communications history
reproducible (and therefore, forkable).  My end goal is
completely forkable communities without any central arbiters.

Email has problems, of course.  Big players are constantly
introducing more complexity to squeeze out smaller players.
And most mail servers require DNS through ICANN, an organization
that tried to extort .org users and hence likely to attempt
further abuses of power in the future.  Again, power corrupts.

> So really basic things that are comparatively trivial with E-Mail
> (e.g. "I think the search sucks, try another client") run up against a
> brick wall with those tools.

I'm working on some local tooling based on public-inbox ;)
(of course, totally optional)

And public-inbox will support JMAP in coming months, so it'll
be a standardized API and hopefully compatible with a wider
variety of clients and frontends.  This should help users who
prefer other other layouts.


Anyways, I do what I can to keep hardware and bandwidth
requirements low for folks who:
a) can't afford to keep up with Moore's law
b) won't accept mystery firmware blobs in modern HW
I got into free software because HW was constantly obsoleted by
proprietary software; and I'm sad so much "modern" free software
has followed that path...

The modern web is largely unusable with HW I first tried git
with in 2005.  Despite being technically free software, the size
of "modern" browsers makes it impractical for people on slow
HW/connections to actually exercise software freedom.  IMHO, any
HW that worked well in 2005 when git started ought to work well
for git and anything associated with it today.

Re: Pain points in Git's patch flow

[Sebastian Schuberth]

On Sun, Apr 18, 2021 at 10:54 PM Ævar Arnfjörð Bjarmason
<avarab@gmail.com> wrote:

> And thank you for participating in the discussion. I think it's
> especially valuable to get a viewpoint like yours, i.e. someone who (per
> this E-Mail below) gave up in frustration with the current development
> flow.

To be fair, Git's contribution flow isn't the only reason why I chose
to stop contributing. Another reason is the very lengthy and tedious
discussions that too often spark from rather small changes.

Also, I wouldn't say I "gave up in frustration". It was a mostly
unemotional decision on which of the many OSS projects I contribute to
my rare spare time is spent best.

> I think it's important not to conflate tooling issues with social
> issues. It's not that we e.g. couldn't whip up a quick script to

I'm not sure I agree completely here, because some tools make it
easier to overcome social issues than others.

> Rather it's that it's a volunteer project and people work on what
> they're interested in.

Exactly. That's why I believe tooling should allow people to subscribe
to changes in code areas they're interested in, rather than a
contributor having to know which subsystem maintainer to put in CC
(e.g. for gitk changes). At least at the time when I contributed it
was sometimes hard to move things forward if you didn't reach out to
the right people.

> So maybe having assigned reviewers would help move things along. But I
> wonder if it wouldn't also lead down the rut of PRs/MRs languishing for
> months, because the reviewers just want to spend their time in some
> other way.

Having default assignees for reviews / code owners / however you want
to call it does not mean that only these people should review, or that
something cannot be merged without their review. It just makes it more
clear who's opinion would be the best to get, and who should execute a
"word of command" if things do not move forward.

> I.e. the design of many of these tools in this regard assumes you have a
> workforce, not the cat-herding problem of volunteers working on whatever
> strikes their fancy.

E.g. GitHub makes the distinction between "reviewers" for a PR and
"assignees" for a PR, and the former can be configured from a
CODEOWNERS file. In projects I contribute to on GitHub, "reviewers"
are used as an optional list of named reviewers, i.e. these people are
explicitly invited for a review. There's no obligation to review,
though. On the other hand, if there are additional "assignees", these
people are explicitly asked for a review. Assignees can also be
assigned only at a later stage of the review, to "settle" a
discussion.

The cat-herd of volunteers would neither be "reviewers" nor
"assignees", but they would just browse the list or open PRs can jump
it where they want to.

> I think characterizing E-Mail as a "legacy" workflow isn't accurate. All

I admit it was a deliberately provocative choice of words, well
knowing it's not reflecting the current state, to underline how I'm
feeling about the workflow. E-mail is great. Also plain text e-mail is
great (I've configured all my client to only send plain text), but
please, not for sending around code patches.

If you send around code patches by mail instead of directly working on
Git repos plus some UI, that feels to me like serializing a data class
instance to JSON, printing the JSON string to paper, taking that sheet
of paper to another PC with a scanner, using OCR to scan it into a
JSON string, and then deserialize it again to a new data class
instance, when you could have just a REST API to push the data from on
PC to the other.

> of these proposed alternatives involve moving away from something that's
> a distributed system today (E-Mail infrastructure, local clients), to
> what's essentially some website run by a centralized entity, in some
> cases proprietary.

That's a good point, I admit I haven't thought of that. Probably
because I also don't care much. So *does* it really matter? What
exactly concerns you about a "centralized entity"? Is it the technical
aspect of a single point of failure, or the political / social aspect
of being dependent on someone you do not want to get influenced by? I
guess it's a bit of both.

While these concerns could probably be addressed somewhat e.g. by
multiple independently operated Gerrit servers that are kept in sync,
I was curious and quickly search for more fitting "truly
decentralized" solutions, and came across radicle [1]. Just FYI.

> So really basic things that are comparatively trivial with E-Mail
> (e.g. "I think the search sucks, try another client") run up against a
> brick wall with those tools.

Not necessarily. As many of these tools have (REST) APIs, also
different API clients exist that you could try.

> And to e.g. as one good example to use (as is the common convention on
> this list) git-range-diff to display a diff to the "last rebased
> revision" would mean some long feature cycle in those tools, if they're
> even interested in implementing such a thing at all.

AFAIK Gerrit can already do that.

-- 
Sebastian Schuberth

Re: Pain points in Git's patch flow

[Sebastian Schuberth]

On Mon, Apr 19, 2021 at 7:54 AM Sebastian Schuberth
<sschuberth@gmail.com> wrote:

> While these concerns could probably be addressed somewhat e.g. by
> multiple independently operated Gerrit servers that are kept in sync,
> I was curious and quickly search for more fitting "truly
> decentralized" solutions, and came across radicle [1]. Just FYI.

Forgot to add the link:

[1] https://radicle.xyz/

-- 
Sebastian Schuberth

Re: Pain points in Git's patch flow

[Ævar Arnfjörð Bjarmason]

On Mon, Apr 19 2021, Sebastian Schuberth wrote:

> On Sun, Apr 18, 2021 at 10:54 PM Ævar Arnfjörð Bjarmason
> <avarab@gmail.com> wrote:
>
>> And thank you for participating in the discussion. I think it's
>> especially valuable to get a viewpoint like yours, i.e. someone who (per
>> this E-Mail below) gave up in frustration with the current development
>> flow.
>
> To be fair, Git's contribution flow isn't the only reason why I chose
> to stop contributing. Another reason is the very lengthy and tedious
> discussions that too often spark from rather small changes.
>
> Also, I wouldn't say I "gave up in frustration". It was a mostly
> unemotional decision on which of the many OSS projects I contribute to
> my rare spare time is spent best.
>
>> I think it's important not to conflate tooling issues with social
>> issues. It's not that we e.g. couldn't whip up a quick script to
>
> I'm not sure I agree completely here, because some tools make it
> easier to overcome social issues than others.

Indeed, to clarify I'm not dismissing that. E.g. is an easier UX or
other "prodding" going to result in better collaboration? Maybe. I'm
just pointing out that it may not mostly/entirely be a tooling issue.

>> Rather it's that it's a volunteer project and people work on what
>> they're interested in.
>
> Exactly. That's why I believe tooling should allow people to subscribe
> to changes in code areas they're interested in, rather than a
> contributor having to know which subsystem maintainer to put in CC
> (e.g. for gitk changes). At least at the time when I contributed it
> was sometimes hard to move things forward if you didn't reach out to
> the right people.
>
>> So maybe having assigned reviewers would help move things along. But I
>> wonder if it wouldn't also lead down the rut of PRs/MRs languishing for
>> months, because the reviewers just want to spend their time in some
>> other way.
>
> Having default assignees for reviews / code owners / however you want
> to call it does not mean that only these people should review, or that
> something cannot be merged without their review. It just makes it more
> clear who's opinion would be the best to get, and who should execute a
> "word of command" if things do not move forward.
>
>> I.e. the design of many of these tools in this regard assumes you have a
>> workforce, not the cat-herding problem of volunteers working on whatever
>> strikes their fancy.
>
> E.g. GitHub makes the distinction between "reviewers" for a PR and
> "assignees" for a PR, and the former can be configured from a
> CODEOWNERS file[...]

CODEOWNERS and default assignment etc. is something that tends to
over-assign things, which is fine for a workforce you're paying, but
it's important to realize that the practice in the git project is to put
the onus on the submitter of the change to manually find who they should
CC.

Because it's not a problem you can really solve automatically without
the a trade-off of feeding potentially uninterested parties a bunch of
patches they're not interested in, which could be another thing that
makes them give up contributing.

So for example I've got a lot of code in git I consider that I "own" in
the sense that I'm responsible for creating it, have been involved in
past design discussions about it etc.

But very little of that cleanly maps to a file as the CODEOWNERS
workflow expects. E.g. there's parts of grep.c that I'd definitely like
to be CC'd on, but others not. Even a "git blame" of the specific lines
you're touching isn't always what you want.

Having used CODEOWNERS in a corporate setting I think it's most useful
for e.g. when you have a monorepo with different subdirectories that do
(at least mostly) map o different teams or peope, think drivers/usb/* or
arch/s390/*.

>> I think characterizing E-Mail as a "legacy" workflow isn't accurate. All
>
> I admit it was a deliberately provocative choice of words, well
> knowing it's not reflecting the current state, to underline how I'm
> feeling about the workflow. E-mail is great. Also plain text e-mail is
> great (I've configured all my client to only send plain text), but
> please, not for sending around code patches.
>
> If you send around code patches by mail instead of directly working on
> Git repos plus some UI, that feels to me like serializing a data class
> instance to JSON, printing the JSON string to paper, taking that sheet
> of paper to another PC with a scanner, using OCR to scan it into a
> JSON string, and then deserialize it again to a new data class
> instance, when you could have just a REST API to push the data from on
> PC to the other.

That's not inherent with the E-Mail workflow, e.g. Linus on the LKML
also pulls from remotes.

It does ensure that e.g. if someone submits patches and then deletes
their GitHub account the patches are still on the ML.

>> of these proposed alternatives involve moving away from something that's
>> a distributed system today (E-Mail infrastructure, local clients), to
>> what's essentially some website run by a centralized entity, in some
>> cases proprietary.
>
> That's a good point, I admit I haven't thought of that. Probably
> because I also don't care much. So *does* it really matter? What
> exactly concerns you about a "centralized entity"? Is it the technical
> aspect of a single point of failure, or the political / social aspect
> of being dependent on someone you do not want to get influenced by? I
> guess it's a bit of both.

To begin with if we'd have used the bugtracker solution from the
beginning we'd probably be talking about moving away from Bugzilla
now. I.e. using those things means your data becomes entangled with the
their opinionated data models.

> While these concerns could probably be addressed somewhat e.g. by
> multiple independently operated Gerrit servers that are kept in sync,
> I was curious and quickly search for more fitting "truly
> decentralized" solutions, and came across radicle [1]. Just FYI.

That's interesting, but I haven't looked into that tool. Browsing their
documentation earlier many of the links were 404s.

>> So really basic things that are comparatively trivial with E-Mail
>> (e.g. "I think the search sucks, try another client") run up against a
>> brick wall with those tools.
>
> Not necessarily. As many of these tools have (REST) APIs, also
> different API clients exist that you could try.

API use that usually (always?) requires an account/EULA with some entity
holding the data, and as a practical concern getting all the data is
usually some huge number of API requests.

>> And to e.g. as one good example to use (as is the common convention on
>> this list) git-range-diff to display a diff to the "last rebased
>> revision" would mean some long feature cycle in those tools, if they're
>> even interested in implementing such a thing at all.
>
> AFAIK Gerrit can already do that.

Sure, FWIW the point was that you needed Gerrit to implement that, and
to suggest what if they weren't interested. Would you need to maintain a
forked Gerrit?

Not to say that's a dealbreaker, just trying to bridge the understanding
of why some people prefer the E-Mail workflow.

Anyway, as before don't take any of the above as arguing, FWIW I
wouldn't mind using one of these websites overall if it helped
development velocity in the project.

Ultimately those things are up to Junio though, which these discussions
always come down to.

I just wanted to help bridge the gap between the distributed E-Mail v.s
centralized website flow.

Re: Pain points in Git's patch flow

[Sebastian Schuberth]

On Mon, Apr 19, 2021 at 10:26 AM Ævar Arnfjörð Bjarmason
<avarab@gmail.com> wrote:

> > If you send around code patches by mail instead of directly working on
> > Git repos plus some UI, that feels to me like serializing a data class
> > instance to JSON, printing the JSON string to paper, taking that sheet
> > of paper to another PC with a scanner, using OCR to scan it into a
> > JSON string, and then deserialize it again to a new data class
> > instance, when you could have just a REST API to push the data from on
> > PC to the other.
>
> That's not inherent with the E-Mail workflow, e.g. Linus on the LKML
> also pulls from remotes.

Yeah, I was vaguely aware of this. To me, the question is why "also"?
Why not *only* pull from remotes? What's the feature gap email patches
try to close?

> It does ensure that e.g. if someone submits patches and then deletes
> their GitHub account the patches are still on the ML.

Ah, so it's basically just about a backup? That could also be solved
differently by forking / syncing Git repos.

> To begin with if we'd have used the bugtracker solution from the
> beginning we'd probably be talking about moving away from Bugzilla
> now. I.e. using those things means your data becomes entangled with the
> their opinionated data models.

Indeed, it's an art to choose the right tool at the time, and to
ensure you're not running into some "vendor-lock-in" if data export is
made too hard. And aligning on someone's "opinionated data model" is
not necessarily a bad thing, as standardization can also help
interoperability and to smoothen workflows.

> > Not necessarily. As many of these tools have (REST) APIs, also
> > different API clients exist that you could try.
>
> API use that usually (always?) requires an account/EULA with some entity
> holding the data, and as a practical concern getting all the data is
> usually some huge number of API requests.

I'm not sure how relevant that concern really is, but in any cause it
would be irrelevant for a self-hosted solution.

> >> And to e.g. as one good example to use (as is the common convention on
> >> this list) git-range-diff to display a diff to the "last rebased
> >> revision" would mean some long feature cycle in those tools, if they're
> >> even interested in implementing such a thing at all.
> >
> > AFAIK Gerrit can already do that.
>
> Sure, FWIW the point was that you needed Gerrit to implement that, and
> to suggest what if they weren't interested. Would you need to maintain a
> forked Gerrit?

Sorry, I can't follow that. Why would you need to maintain a fork of
Gerrit if Gerrit already has the feature you're looking for? Is it a
hypothetical question about what to do if Gerrit would not have the
feature yet?

> Anyway, as before don't take any of the above as arguing, FWIW I
> wouldn't mind using one of these websites overall if it helped
> development velocity in the project.

I appreciate that open mindset of yours here.

> I just wanted to help bridge the gap between the distributed E-Mail v.s
> centralized website flow.

Maybe, instead of jumping into something like an email vs Gerrit
discussion, what would help is to get back one step and gather the
abstract requirements. Then, with a fresh and unbiased mind, look at
all the tools and infrastructure out there that are able to fulfill
the needs, and then make a choice.

-- 
Sebastian Schuberth

Re: Pain points in Git's patch flow

[Theodore Ts'o]

On Mon, Apr 19, 2021 at 09:23:14PM +0200, Sebastian Schuberth wrote:
> > That's not inherent with the E-Mail workflow, e.g. Linus on the LKML
> > also pulls from remotes.
> 
> Yeah, I was vaguely aware of this. To me, the question is why "also"?
> Why not *only* pull from remotes? What's the feature gap email patches
> try to close?

Linus mostly pulls from git trees.  The e-mail workflow tends to be
used by maintainers, who are reviewing submissions from their
contributors.  People submitting changes relating to ext4 know to send
it to the linux-ext4 mailing list; people who are submitting changes
to the xfs file system send it to linux-xfs, etc.

> > It does ensure that e.g. if someone submits patches and then deletes
> > their GitHub account the patches are still on the ML.
> 
> Ah, so it's basically just about a backup? That could also be solved
> differently by forking / syncing Git repos.

The primary reason why the kernel uses mailing lists is because code
reviews are fundamentally *discussions*, and people are used to using
inboxes.  Sure, you can have a gerrit server send e-mail notifications
about code reviews, but then you have to reply by going to the gerrit
server (and gerrit really doesn't work well on slow network link such
as those found on airplanes and cruise ships).  I'd say that most
maintainers simply find e-mail reviews to simply be more *convenient*
than using gerrit.  And over time, we've used other tools to track
metadata over the status of a patch, such as patchwork, which are
optional.

> > I just wanted to help bridge the gap between the distributed E-Mail v.s
> > centralized website flow.
> 
> Maybe, instead of jumping into something like an email vs Gerrit
> discussion, what would help is to get back one step and gather the
> abstract requirements. Then, with a fresh and unbiased mind, look at
> all the tools and infrastructure out there that are able to fulfill
> the needs, and then make a choice.

I'll note that the kernel folks have done this, starting with a 2019
Kernel Summit talk at the Linux Plumbers Conference in Lisbon.  A
description of the follow-up discussions from that talk can be found
here:

	https://lwn.net/Articles/803619/

There was a collection of requirements on a thread on the newly
created workflows@vger.kernel.org mailing list.  This has led to a
number of proposals to make improvements to git, public-inbox,
patchwork, the kernel.org infrastructures, etc., some of which were
funded by the Linux Foundation last year.

Konstantin Ryabitsev has been driving a large amount of that work, and
one of the things that has come out of that is b4.  (Yes, that's a
Star Trek reference...  https://memory-alpha.fandom.com/wiki/B-4)

  https://people.kernel.org/monsieuricon/introducing-b4-and-patch-attestation

Obviously, this isn't intended to be a solution for everyone, and I'm
sure there are many projects that are happy forcing developers to use,
say, Gerrit, which might be a better solution for them.

However, there are a number of core kernel developers who are
super-allergic to solutions which force users to use web interfaces.
So solutions that have a combination of CLI's as well as web interface
is probably going to be the right approach.  Things like pwclient and
b4 are exciting starting points for improved kernel workflows.

Of course, we've gone a bit farther afield from the original question
which is what should git's development workflows should be.  Given
that git is using some of the kernel.org infrastructures, certainly
some of the kernel workflow tools are options for the git development
community to consider.

One of the advantages of the kernel workflows model is that we don't
force users to use github or gitlab or gerrit, without having to make
a global decision for the entire community.  For example, if some
developers want to start using b4 to download patch series for git,
they could start doing that today.

Cheers,

						- Ted

Re: Pain points in Git's patch flow

[Sebastian Schuberth]

On Tue, Apr 20, 2021 at 12:34 AM Theodore Ts'o <tytso@mit.edu> wrote:

> The primary reason why the kernel uses mailing lists is because code
> reviews are fundamentally *discussions*, and people are used to using
> inboxes.  Sure, you can have a gerrit server send e-mail notifications

[...]

> maintainers simply find e-mail reviews to simply be more *convenient*
> than using gerrit.  And over time, we've used other tools to track

That still sounds to me as if people are stuck to what they know.
Maintainers are "used to using inboxes'', and that's *why* they find
e-mail reviews to be convenient.

Of course, there's basically nothing wrong with sticking to a flow
that works. But I understood the start of this discussion as a sign
that you guys acknowledge that something does *not* work. At least not
when it comes to attracting new contributors.

> I'll note that the kernel folks have done this, starting with a 2019
> Kernel Summit talk at the Linux Plumbers Conference in Lisbon.  A
> description of the follow-up discussions from that talk can be found
> here:

I'm reading a lot about "maintainers" and "kernel developers" here.
But what I believe is important to accept is that Git is not only
about kernel development anymore. While I'm well aware of Git's
history, there are by far more people using Git than there are kernel
developers, and also by lines of code (or whatever "stupid" metric you
want to choose) the kernel is not the biggest project maintained in
Git. Maybe not even the most important one, but that's highly
subjective anyway. So asked in a heretic way, why should the opinion
of a kernel developer count more than the opinion of, say, an Eclipse
Foundation developer when it comes to Git workflow questions?

To me, that means if you want to make contributions to Git more
attractive to the Git community beyond the kernel, you need to stop
making incremental improvements to existing tools and start thinking
out of the box by looking at the tools that are most popular in that
"other side" of the community.

And, please don't take anything I've written as a try to talk you into
anything. But as the lead of a team who's day job it is to contribute
to various Open Source projects, I believe to have a good feeling
about what the pain points of developers are to start contributing.
I'm just trying to foster some appreciation for the thinking of the
"other side".

-- 
Sebastian Schuberth

Re: Pain points in Git's patch flow

[Theodore Ts'o]

On Tue, Apr 20, 2021 at 08:30:57AM +0200, Sebastian Schuberth wrote:
> 
> I'm reading a lot about "maintainers" and "kernel developers" here.
> But what I believe is important to accept is that Git is not only
> about kernel development anymore. While I'm well aware of Git's
> history, there are by far more people using Git than there are kernel
> developers, and also by lines of code (or whatever "stupid" metric you
> want to choose) the kernel is not the biggest project maintained in
> Git. Maybe not even the most important one, but that's highly
> subjective anyway. So asked in a heretic way, why should the opinion
> of a kernel developer count more than the opinion of, say, an Eclipse
> Foundation developer when it comes to Git workflow questions?

I think it should be up to each development community to decide what
workflows makes sense for that community.

The kernel community has been taking requirements for what works well
for *that* community, and we've found companies who are willing to
fund improvements in the tools that we use (which include git,
public-inbox, patchwork, etc.) so that it can meet our needs.

Our approach is that we don't want to *force* everyone to switch to
some web interface.  Instead, instead of either-or, we're looking for
some kind both/and, so we don't have to insult people who are
*extremely* productive with an e-mail workflow by calling them
dinosaurs, and force them to use a web interface which would make them
much less productive, on the hope that maybe we would get some more
new contributors.  (And at least for the kernel, we're blessed by the
fact that there is no shortage of new contributors; so our goals are
to make *everyone* more productive, and not have an attitude of "you
shall use gerrit/github and everyone else can go suck wind".)

Git is going to have to decide what development workflows will work
well for its development community.  Historically, since git was
originally authored by Linus Torvalds, it's not surprising that there
is a bias towards an e-mail workflow.  Indeed, git commands like "git
send-email" and "git apply-mbox" are there from the very beginning
because it was *designed* to work well with the kernel workflow.

> To me, that means if you want to make contributions to Git more
> attractive to the Git community beyond the kernel, you need to stop
> making incremental improvements to existing tools and start thinking
> out of the box by looking at the tools that are most popular in that
> "other side" of the community.

I'm not sure that's what was meant by the question of pain points in
Git's patch flow.  Your perspective is certainly a valid one, and it's
certainly easier to choose sides and make an opinioned decision which
disenfranches "one side" of the community in favor of the "other side"
of the community.

It's not the approach that was adopted by the folks who are working on
improving the Kernel development workflows.  The git development
community will need what approach makes sense for it.

Cheers,

						- Ted

Re: Pain points in Git's patch flow

[Felipe Contreras]

Sebastian Schuberth wrote:
> On Tue, Apr 20, 2021 at 12:34 AM Theodore Ts'o <tytso@mit.edu> wrote:
> 
> > The primary reason why the kernel uses mailing lists is because code
> > reviews are fundamentally *discussions*, and people are used to using
> > inboxes.  Sure, you can have a gerrit server send e-mail notifications
> 
> [...]
> 
> > maintainers simply find e-mail reviews to simply be more *convenient*
> > than using gerrit.  And over time, we've used other tools to track
> 
> That still sounds to me as if people are stuck to what they know.
> Maintainers are "used to using inboxes'', and that's *why* they find
> e-mail reviews to be convenient.

I'm not stuck with what I know.

The reason why I use the email workflow is not because "I'm stuck" with
it, it's because I've tried every approach out there, and they are *all*
inferior.

You tell me of an approach and I will tell you all the ways in which
it's inferior to email.

If some people want to use Gerrit, and/or Patchwork on top of email,
that's fine. You can use inferior approaches if you want, just don't
force the rest of us to stop using the superior approach.

Cheers.

-- 
Felipe Contreras

Re: Pain points in Git's patch flow

[Ævar Arnfjörð Bjarmason]

On Mon, Apr 19 2021, Sebastian Schuberth wrote:

> On Mon, Apr 19, 2021 at 10:26 AM Ævar Arnfjörð Bjarmason
> <avarab@gmail.com> wrote:
>
>> > If you send around code patches by mail instead of directly working on
>> > Git repos plus some UI, that feels to me like serializing a data class
>> > instance to JSON, printing the JSON string to paper, taking that sheet
>> > of paper to another PC with a scanner, using OCR to scan it into a
>> > JSON string, and then deserialize it again to a new data class
>> > instance, when you could have just a REST API to push the data from on
>> > PC to the other.
>>
>> That's not inherent with the E-Mail workflow, e.g. Linus on the LKML
>> also pulls from remotes.
>
> Yeah, I was vaguely aware of this. To me, the question is why "also"?
> Why not *only* pull from remotes? What's the feature gap email patches
> try to close?
>
>> It does ensure that e.g. if someone submits patches and then deletes
>> their GitHub account the patches are still on the ML.
>
> Ah, so it's basically just about a backup? That could also be solved
> differently by forking / syncing Git repos.

I just mentioned that as an example, FWIW I think Linus mainly uses that
method for pulling "for-linus" tags from lieutenants, but that those
"forks" in turn are developed over E-Mail (e.g. on the linux-usb list).

So they're supporting a flow the Git ML doesn't, of e.g. needing
hundreds of patches from a subsystem maintainer to bring that subsystem
up-to-date for a release.

Aside from that:

 1. Once you have patches sent to the list for review / commenting you
    might as well use them to apply the changes, why also require a
    repository to pull from?

    I may just not be understanding what you're suggesting here.

 2. One thing you may have missed is that patches != sending someone a
    repo URL to "pull" / run "log" etc. on because:

    2.1: You can provide different context (-U<n> -W) and diff algorithm
         for patches, and submitters sometimes do on this list to make
         the job of reviewers easier. I.e. sometimes I'll want to
         manually extend the context to some relevant code related to
         what's being changed.

    2.2: There's also the convention of a free-form (e.g. extra
         commentary, a reply to another E-Mail, whatever) after the
         "---" in the patch.

>> To begin with if we'd have used the bugtracker solution from the
>> beginning we'd probably be talking about moving away from Bugzilla
>> now. I.e. using those things means your data becomes entangled with the
>> their opinionated data models.
>
> Indeed, it's an art to choose the right tool at the time, and to
> ensure you're not running into some "vendor-lock-in" if data export is
> made too hard. And aligning on someone's "opinionated data model" is
> not necessarily a bad thing, as standardization can also help
> interoperability and to smoothen workflows.
>
>> > Not necessarily. As many of these tools have (REST) APIs, also
>> > different API clients exist that you could try.
>>
>> API use that usually (always?) requires an account/EULA with some entity
>> holding the data, and as a practical concern getting all the data is
>> usually some huge number of API requests.
>
> I'm not sure how relevant that concern really is, but in any cause it
> would be irrelevant for a self-hosted solution.

Yes and no, getting kicked off the service due to e.g. being from Iran
would probably be a non-issue (as was the case with GitHub until
recently).

I've still done various searching/for-looping over the ML archive in a
way that would if translated to API requests against some remote service
probably be considered a DoS, so being distributed helps there.

>> >> And to e.g. as one good example to use (as is the common convention on
>> >> this list) git-range-diff to display a diff to the "last rebased
>> >> revision" would mean some long feature cycle in those tools, if they're
>> >> even interested in implementing such a thing at all.
>> >
>> > AFAIK Gerrit can already do that.
>>
>> Sure, FWIW the point was that you needed Gerrit to implement that, and
>> to suggest what if they weren't interested. Would you need to maintain a
>> forked Gerrit?
>
> Sorry, I can't follow that. Why would you need to maintain a fork of
> Gerrit if Gerrit already has the feature you're looking for? Is it a
> hypothetical question about what to do if Gerrit would not have the
> feature yet?

My example of range-diff upthread in
<87fszn48lh.fsf@evledraar.gmail.com> which started this discussion
wasn't to make a point about range-diff per-se, but just mention it
offhand as a "feature" that in an E-Mail based flow is simply a matter
of someone including a thing like that in their cover letter and sending
a patch.

So for that example the specifics of range-diff being a part of Gerrit
now don't matter that much, the point is that the next thing lik
range-diff first has to be made part of such a service, it can't easily
emerge "bottom-up".

Of course there's disadvantages to that too, I'm just pointing out that
the free-form of text also has advantages that shouldn't be overlooked.

>> Anyway, as before don't take any of the above as arguing, FWIW I
>> wouldn't mind using one of these websites overall if it helped
>> development velocity in the project.
>
> I appreciate that open mindset of yours here.
>
>> I just wanted to help bridge the gap between the distributed E-Mail v.s
>> centralized website flow.
>
> Maybe, instead of jumping into something like an email vs Gerrit
> discussion, what would help is to get back one step and gather the
> abstract requirements. Then, with a fresh and unbiased mind, look at
> all the tools and infrastructure out there that are able to fulfill
> the needs, and then make a choice.

The abstract requirement comes down to one thing: Whatever Junio
decides.

For a bit of context on the current discussion:

There have been versions of this discussion in-person in various "git
merge" developer meets over the years, Junio doesn't attend most of
those (I believe the last one was the one in April 2016 in NYC, but I
missed a couple since then, so don't take my word for it).

So those discussions re-hashing of the various pain points that have
been brought up over the years (and again in this thread), but have had
to tip-toe around the fact that most things can't make radical forward
progress (as is being proposed by some here) unless Junio is on board
with them, without the benefit of being able to get feedback from him in
real-time.

So the things that have come out of them are things like submitGit
(later GitGitGadget), which are very useful, but still something
implemented inside the narrow confines of fitting into the confines of
the existing E-Mail/ML-based workflow.

My reading of [1] and [2] (including some "between the lines", so I may
be wrong) is that Junio's very much interested in using his current
workflow as the primary "source of truth", and that any (issue) tracking
system will need to be shimmied on top of that.

Which I think means that any such "system on top" is going to have to be
bespoke and inevitably drift from the real "source of truth" which is
the mailing list, What's Cooking E-Mails etc.

As opposed to say, imagining some "light" in-between state where we
don't use a bugtracker, or have any discussion on some centralized
website, but would (or rather, Junio would) commit to creating one-line
tracking issues for what's now the topics in "next" and "seen"
branches. I.e. just the smaller step of answering "what's the status of
my topic" via a some bug tracker, nevermind using it for anything else.

Which at least for some contributors like myself means that I'm
uninterested in using any such system myself, not because I would mind
using it in theory, but as long as it's not the "source of truth" I
don't see the point. I'd still need to scour the ML for anything it may
have missed.

1. https://lore.kernel.org/git/xmqqv98orsj5.fsf@gitster.g/
2. https://lore.kernel.org/git/xmqqfszqko0k.fsf@gitster.g/

Re: Pain points in Git's patch flow

[Eric Wong]

Sebastian Schuberth <sschuberth@gmail.com> wrote:
> On Sun, Apr 18, 2021 at 10:54 PM Ævar Arnfjörð Bjarmason
> <avarab@gmail.com> wrote:
> 
> > And thank you for participating in the discussion. I think it's
> > especially valuable to get a viewpoint like yours, i.e. someone who (per
> > this E-Mail below) gave up in frustration with the current development
> > flow.
> 
> To be fair, Git's contribution flow isn't the only reason why I chose
> to stop contributing. Another reason is the very lengthy and tedious
> discussions that too often spark from rather small changes.
> 
> Also, I wouldn't say I "gave up in frustration". It was a mostly
> unemotional decision on which of the many OSS projects I contribute to
> my rare spare time is spent best.

I guess some things aren't for everybody.  When I started
git-svn, I never expected git to be the right tool for others.
I figured most folks could just continue using SVN since they
seem to like centralized things or at least have some sort of
"authority" to look to.

I'm largely uninvolved with git nowadays since I'm reasonably
satisfied with how it works; that and I prefer scripting
languages rather than ahead-of-time languages.

> > Rather it's that it's a volunteer project and people work on what
> > they're interested in.
> 
> Exactly. That's why I believe tooling should allow people to subscribe
> to changes in code areas they're interested in, rather than a
> contributor having to know which subsystem maintainer to put in CC
> (e.g. for gitk changes). At least at the time when I contributed it
> was sometimes hard to move things forward if you didn't reach out to
> the right people.

Fwiw, any public-inbox endpoint with Xapian search enabled lets
you request an Atom feed via "x=A" query parameter.

To watch a particular filename, the "dfn:" prefix may be used.
The prefixes supported for a particular instance are documented in
<https://public-inbox.org/git/_/text/help/>, and you
can watch multiple files by combining with "OR".

https://public-inbox.org/git/?q=dfn:cache.h+OR+dfn:git-send-email.perl&x=A

You can also POST to get a gzipped mboxrd file:

curl -d '' \
  'https://public-inbox.org/git/?q=dfn:cache.h+OR+dfn:git-send-email.perl&x=m'

> > of these proposed alternatives involve moving away from something that's
> > a distributed system today (E-Mail infrastructure, local clients), to
> > what's essentially some website run by a centralized entity, in some
> > cases proprietary.
> 
> That's a good point, I admit I haven't thought of that. Probably
> because I also don't care much. So *does* it really matter? What
> exactly concerns you about a "centralized entity"? Is it the technical
> aspect of a single point of failure, or the political / social aspect
> of being dependent on someone you do not want to get influenced by? I
> guess it's a bit of both.

Yes, both for me.  The political/social aspect is the main
reason I'm involved with DVCS (and a large part of why I'm
involved with free software in general).

> While these concerns could probably be addressed somewhat e.g. by
> multiple independently operated Gerrit servers that are kept in sync,
> I was curious and quickly search for more fitting "truly
> decentralized" solutions, and came across radicle [1]. Just FYI.

I don't think any sort of radicle "flag day" or tool mandate is
going to fly.  I seem to recall at least one prominent Linux
kernel hacker doesn't even use git; though I'm not sure if
that's still the case.

Despite being a DVCS user even pre-git, I'm actually
pessimistic about decentralization protocols that either:

1) rely on planet-destroying proof-of-work schemes

2) will need to reinvent the spam filtering techniques
   of email once they hit critical mass

Email is already well-established with a good amount of small
players, and plain-text is relatively inexpensive.  So it seems
best to build off the only halfway-decentralized thing we have
in wide use, rather than trying to start from scratch.

Re: Pain points in Git's patch flow

[Sebastian Schuberth]

On Mon, Apr 19, 2021 at 9:36 PM Eric Wong <e@80x24.org> wrote:

> > Also, I wouldn't say I "gave up in frustration". It was a mostly
> > unemotional decision on which of the many OSS projects I contribute to
> > my rare spare time is spent best.
>
> I guess some things aren't for everybody.  When I started
> git-svn, I never expected git to be the right tool for others.
> I figured most folks could just continue using SVN since they
> seem to like centralized things or at least have some sort of
> "authority" to look to.
>
> I'm largely uninvolved with git nowadays since I'm reasonably
> satisfied with how it works; that and I prefer scripting
> languages rather than ahead-of-time languages.

True, since quite a while I'm also at a point where I'm satisfied with
how Git (for Windows) works, so I also ceased to see the need to
contribute. That's indeed another reason I forgot to mention.

> To watch a particular filename, the "dfn:" prefix may be used.
> The prefixes supported for a particular instance are documented in
> <https://public-inbox.org/git/_/text/help/>, and you
> can watch multiple files by combining with "OR".

Thanks for pointing out these interesting features, I wasn't aware of them.

> I don't think any sort of radicle "flag day" or tool mandate is
> going to fly.  I seem to recall at least one prominent Linux
> kernel hacker doesn't even use git; though I'm not sure if
> that's still the case.

Like you said in the beginning, I guess some things aren't for everybody.

> Email is already well-established with a good amount of small
> players, and plain-text is relatively inexpensive.  So it seems
> best to build off the only halfway-decentralized thing we have
> in wide use, rather than trying to start from scratch.

While I can understand that conservative approach for a community
around a tool as important as Git, I still fear that only ever
sticking to technology that is already in wide use will hinder to look
over the rim of the tea cup.

-- 
Sebastian Schuberth

Re: Pain points in Git's patch flow

[Konstantin Ryabitsev]

On Mon, Apr 19, 2021 at 09:49:46PM +0200, Sebastian Schuberth wrote:
> > To watch a particular filename, the "dfn:" prefix may be used.
> > The prefixes supported for a particular instance are documented in
> > <https://public-inbox.org/git/_/text/help/>, and you
> > can watch multiple files by combining with "OR".
> 
> Thanks for pointing out these interesting features, I wasn't aware of them.

Eric is being modest. There are very cool things brewing in public-inbox, like
ability to create saved searches and follow threads you're interested in.
E.g. you should be able to define something like "whenever someone mentions my
favourite file, function name, or term, copy the entire thread into my inbox
and continuously update it with new messages."

I'm hoping that this will help turn the concept of mailing lists on their head
-- instead of subscribing to a list, folks will instead subscribe to closely
relevant saved searches across any number of remote and local sources.

> > Email is already well-established with a good amount of small
> > players, and plain-text is relatively inexpensive.  So it seems
> > best to build off the only halfway-decentralized thing we have
> > in wide use, rather than trying to start from scratch.
> 
> While I can understand that conservative approach for a community
> around a tool as important as Git, I still fear that only ever
> sticking to technology that is already in wide use will hinder to look
> over the rim of the tea cup.

I view email as merely one way of exchanging RFC2822-formatted messages. 
There are others and RFC2822 is robust enough to serve as a good standard
base that allows both free-form and structured content, including mixed.

-K

Re: Pain points in Git's patch flow

[dwh]

On 19.04.2021 18:00, Konstantin Ryabitsev wrote:
>I view email as merely one way of exchanging RFC2822-formatted messages.
>There are others and RFC2822 is robust enough to serve as a good standard
>base that allows both free-form and structured content, including mixed.

+1 on RFC2822 as universal message format. It is simple, easy to
understand, trivial to manipulate in any programming language and widely
supported. Standard file formats, along with standard protocols, both
without "proprietary extensinos" is the key to maintaining
decentralization and avoiding siloing of data and users.

Cheers!
Dave

Re: Pain points in Git's patch flow

[Konstantin Ryabitsev]

On Mon, Apr 19, 2021 at 07:54:37AM +0200, Sebastian Schuberth wrote:
> > of these proposed alternatives involve moving away from something that's
> > a distributed system today (E-Mail infrastructure, local clients), to
> > what's essentially some website run by a centralized entity, in some
> > cases proprietary.
> 
> That's a good point, I admit I haven't thought of that. Probably
> because I also don't care much. So *does* it really matter? What
> exactly concerns you about a "centralized entity"? Is it the technical
> aspect of a single point of failure, or the political / social aspect
> of being dependent on someone you do not want to get influenced by? I
> guess it's a bit of both.

It's all of the above, and really should not be discounted. Let's take what
Russian government is doing lately as an example. In its effort to control
social dissent, Russian censorship organization RosKomNadzor (RKN) has taken
steps to deliberately break internet operation -- in a very ham-fisted way.
Just a month ago they tried to "slow down" Twitter by blocking DNS queries for
any domains containing the substring "t.co" -- which, hey, broke
gihubusercontent.com among many other sites. There's every reason to believe
that this won't be the only time they do something idiotic like that, so as a
result it is increasingly difficult for Russian contributors to justify
participating in projects that are hosted on GitHub -- one day they may not be
able to reach it reliably (or at all).

(If you think the answer to that would be "just use a VPN", it's one of those
recommendations that are easy to make for someone not worried about their ISP
reporting "sketchy encrypted traffic" to "the authorities.")

Patches sent via email remain immune to this. Even if vger falls over, it's
merely a list service -- there are alternative ways of transmitting RFC2822
messages that don't involve a central host (such as via a NNTP gateway,
publishing a public-inbox "feed", etc). Email remains one of the few protocols
that are designed ground-up to be decentralized and I'm afraid that we are
again finding ourselves in a world where this is increasingly relevant.

> While these concerns could probably be addressed somewhat e.g. by
> multiple independently operated Gerrit servers that are kept in sync,
> I was curious and quickly search for more fitting "truly
> decentralized" solutions, and came across radicle [1]. Just FYI.

I know Radicle folks -- I was on their technical board. A lot of what they
have implemented is very similar to my initial thoughts expressed in
https://people.kernel.org/monsieuricon/patches-carved-into-developer-sigchains

I have high hopes for the project, but it's not ready to take on the world
until they implement code collaboration aspects (issue tracking, change
requests, etc). It's going to be tough and I really hope they succeed.

-K

Re: Pain points in Git's patch flow

[Stephen Smith]

On Monday, April 19, 2021 2:49:21 PM MST Konstantin Ryabitsev wrote:
> On Mon, Apr 19, 2021 at 07:54:37AM +0200, Sebastian Schuberth wrote:
> > That's a good point, I admit I haven't thought of that. Probably
> > because I also don't care much. So *does* it really matter? What
> > exactly concerns you about a "centralized entity"? Is it the technical
> > aspect of a single point of failure, or the political / social aspect
> > of being dependent on someone you do not want to get influenced by? I
> > guess it's a bit of both.
> 
> It's all of the above, and really should not be discounted. Let's take what
> Russian government is doing lately as an example. In its effort to control
> social dissent, Russian censorship organization RosKomNadzor (RKN) has taken
> steps to deliberately break internet operation -- in a very ham-fisted way.

It can be other things too.   

For instance a corporation that for a variety of reasons has an urgent need to 
restrict internet traffic.   Email will usually get through, but web site traffic 
may not.

While Github is unlikely to get taken offline, other sites that hold data may 
not be so lucky.   Think bankrupcy or other issues.   

Email traffic allows for routing around such issues.

Re: Pain points in Git's patch flow

[dwh]

On 19.04.2021 17:49, Konstantin Ryabitsev wrote:
>On Mon, Apr 19, 2021 at 07:54:37AM +0200, Sebastian Schuberth wrote:
>> > of these proposed alternatives involve moving away from something that's
>> > a distributed system today (E-Mail infrastructure, local clients), to
>> > what's essentially some website run by a centralized entity, in some
>> > cases proprietary.
>>
>> That's a good point, I admit I haven't thought of that. Probably
>> because I also don't care much. So *does* it really matter? What
>> exactly concerns you about a "centralized entity"? Is it the technical
>> aspect of a single point of failure, or the political / social aspect
>> of being dependent on someone you do not want to get influenced by? I
>> guess it's a bit of both.
>
>Patches sent via email remain immune to this. Even if vger falls over, it's
>merely a list service -- there are alternative ways of transmitting RFC2822
>messages that don't involve a central host (such as via a NNTP gateway,
>publishing a public-inbox "feed", etc). Email remains one of the few protocols
>that are designed ground-up to be decentralized and I'm afraid that we are
>again finding ourselves in a world where this is increasingly relevant.

I agree with Konstantin on this one. To this day, email is still the
most decentralized and "user sovereign" system on the internet. The
standardization of protocols and file formats is not perfect but it is
"complete" in the sense that it meets all of the requirements for
decentralized software development.

Think about it like this. Right now, I could use an IMAP client to
download all of my emails from GMail, store them in mbox files, then
use the IMAP client to upload the email to Fastmail or SDF.org or some
other email provider. Or better yet, I can install local tools for
working with my email. The fact that email providers/tools are largely
interchangeable and replacable--despite Google/Yahoo/Microsoft's best
efforts--gives maximum power to users.

Like I said, I totally agree with Konstantin and I think the vision he
described in his post on developer sigchains is what I've always wanted
as an open source developer. It is common to hear the argument that
centralized systems are more convient and easier to use and the more
decentralized a sysetem, the harder it gets to use. I suspect that is
only a half-truth because I don't think we've achieved full
decentralization which is what Konstantin touches on in his post too.
Full decentralization will bring automatic maintainence of p2p
connections and synchronization. Things will "just work".

I know I'm veering off topic a bit here but decentralization has been
the focus of all of my learning, research and work for more than a
decade now. Email is critical for maintaining decentralized development
capabilities.

Cheers!
Dave

Re: Pain points in Git's patch flow

[Bagas Sanjaya]

On 08/05/21 09.08, dwh@linuxprogrammer.org wrote:

> Think about it like this. Right now, I could use an IMAP client to
> download all of my emails from GMail, store them in mbox files, then
> use the IMAP client to upload the email to Fastmail or SDF.org or some
> other email provider. Or better yet, I can install local tools for
> working with my email. The fact that email providers/tools are largely
> interchangeable and replacable--despite Google/Yahoo/Microsoft's best
> efforts--gives maximum power to users.

Well, I use Thunderbird because it supports Gmail accounts out of the box.
But I wish I could use Mutt or similar, alas Gmail requires that I need
to enable 2FA and s/<google account password>/<specific app password>/
in order to access Gmail via Mutt. I currently steer clear from 2FA, because
once upon a time in 2018 I screwed up (locked-out) from my older account,
which IMO mission-critical., for I couldn't pass all possible verification
methods for that. So I created new account one in about Día de Kartini
(Kartini's day).

And yes, Google/Yahoo/MS have webmail interface for their mail services
(Gmail/YahooMail/Outlook), but for purposes for sending to vger.kernel.org,
these above are rubbish because they genereated HTML emails, and vger hate
HTML emails.

-- 
An old man doll... just what I always wanted! - Clara

Re: Pain points in Git's patch flow

[Felipe Contreras]

Sebastian Schuberth wrote:
> On Sun, Apr 18, 2021 at 10:54 PM Ævar Arnfjörð Bjarmason
> <avarab@gmail.com> wrote:
> 
> > And thank you for participating in the discussion. I think it's
> > especially valuable to get a viewpoint like yours, i.e. someone who (per
> > this E-Mail below) gave up in frustration with the current development
> > flow.
> 
> To be fair, Git's contribution flow isn't the only reason why I chose
> to stop contributing. Another reason is the very lengthy and tedious
> discussions that too often spark from rather small changes.

I completely agree with this assessment.

In the spectrum from all code is allowed (0) to only perfect code is allowed
(100) I'd say the git project is around 95. It's good in the sense that
user expectation rarely breaks, but on the other hand not much progress
happens.

Personally I would turn the dial of perfectedness down to 90, or even
80.

It's because of this focus on perfection that discussions get tedious,
and thus perfect becomes the enemy of good.

But since the current maintership is never going to change that focus, I
think a fork of git is necessary.

Cheers.

-- 
Felipe Contreras

Re: Pain points in Git's patch flow

[Daniel Axtens]

Hi all,

> I'd like to introduce Raxel (cc-ed), who is starting an internship
> this June with the Git team at Google.
>
> He'll be working on a bit of an experimental project: we want to take
> Patchwork[1], which in principle can be a helpful addition to a
> mailing list centric workflow[2], and improve it to be something that
> people in the Git open source project get day-to-day benefit from.
> Raxel's previous successes in making changes to tools to support a
> better user experience make me excited for the potential for this
> work.

Greetings Raxel! Myself and Stephen F are patchwork maintainers so we'll
be reviewing and merging any proposals you have for patchwork. We try to
be a welcoming place. We're also both extremely busy so (unfortunately)
you will probably need to ping me if I forget to respond.

> Anyway, yesterday[3] Junio, Taylor, and Emily were discussing how to
> encourage more reviews:
>
>  <gitster> this week, i'd be thinking about ways to get topics, that
>            are not reviewed sufficiently, reviewed. I can act as the
>            last-resort fallback reviewer, but that's not sufficient.
>  <ttaylorr> gitster: I share your concern.
>  <nasamuffin> gitster: yep, agree, on both counts
>
> That reminded me that it would be useful preparation to collect
> descriptions of pain points we are having with our existing patch
> flow.  For example:
>
> - As a reviewer, I want to be able to easily find a series that needs
>   review.  Using patchwork, I can see some recent patch series; or
>   using a hierarchical threaded mail reader, I can find a neglected
>   thread or one that seems to be likely to have an interesting
>   discussion going on.  But without reading in detail, there is no
>   easy way to see whether the series has reached a review, whether
>   someone else intends to review it, and what the author believes its
>   status to be.

Patchwork does have the A/R/T/F
(Acked-by:/Reviewed-by:/Tested-by:/Fixes:) column, but this doesn't have
a good way to capture something that falls short of a full
review. There's also the patch states (e.g. Changes Requested), but that
requires either the author or a maintainer to change the status via the
web interface or with an API client.

> - Relatedly, as a patch author or reviewer, I want to be able to
>   easily tell whether a topic has been sufficiently reviewed.  Today,
>   the signals for this are implicit: I have to judge consensus, or to
>   check the Git repository for whether the patch has been merged, or
>   to check the maintainer's latest "What's cooking in git.git"
>   message.
>
> - As a potential reviewer or interested user, I want to be able to
>   follow all relevant discussion for a patch series, while also
>   having the ability to stop following it if the discussion goes on
>   too long and starts overwhelming my email inbox.  Today, I can join
>   the discussion and then (1) it is hit-or-miss whether the patch
>   author ccs me on later iterations of the patch and (2) there is no
>   easy way without aggressive email filtering to stop watching it if
>   I am cc-ed.
>
> - After having diagnosed an issue to be due to a patch, I want to be
>   able to easily find all relevant review discussion.  Today I can
>   use the mailing list archive[4] or patchwork to find review
>   discussion on the latest version of the series that patch was in,
>   but tracing back to previous iterations of that same series can be
>   non-trivial.  Moreover, if I'm interested in a particular puzzling
>   line of code, finding which iteration introduced it can take a long
>   time.
>

Yeah, cross-series linking is something we've been interested in for a
long time. We have a little bit of the infrastructure already but
there's a long way to go.

One of the real challenges for us has been figuring out how to reliably
link iterations. It's one thing if iterations are sent in-reply-to the
early version, but at least for the kernel that's not The Way Things Are
Done. There's lot of common things people do (split series, rename
series, add/drop patches) that makes reliable linking very
challenging. And traditionally Patchwork has tried to Do It Right rather
than go for a probabilistic approach. (We get a lot of email complaints
if we get things wrong.)

(Having said that, I'd certainly be open to considering any attempts to
automatically link series, even if only probabilistic, so long as they
err by missing things rather than err by linking things that are
unrelated.)

One thing that's come up as another possible option is free-form tags. I
think there's some old series on the list from Veronika that attempts
this. That'd allow Someone to 'tag' a patch or a series on
patchwork. For Veronika's use case this was for complex CI - things like
"ready-for-real-hw-tests" that shouldn't happen until someone has cast
an eye over the patches.

One big challenge for patchwork development (which catches us out
regularly) is the odd interaction between patchwork-the-project and
patchwork-the-deployments. There are 2 large deployments:
patchwork.ozlabs.org and patchwork.kernel.org, and they all have
multiple-GB databases. This means that performance and db load matters,
but is really hard to test locally. A good example is writing efficient
migrations - historically we have struggled with this.

Anyway, good luck with the dozens of messages and strong opinions!

Kind regards,
Daniel

> Those four are important in my everyday life.  Questions:
>
>  1. What pain points in the patch flow for git.git are important to
>     you?
>
>  2. What tricks do you use to get by with those existing pain points?
>
>  3. Do you think patchwork goes in a direction that is likely to help
>     with these?
>
>  4. What other tools would you like to see that could help?
>
> Thanks,
> Jonathan
>
> [1] http://jk.ozlabs.org/projects/patchwork/; you can see an instance
> for Git at https://patchwork.kernel.org/project/git/list/
> [2] https://kernel-recipes.org/en/2016/talks/patches-carved-into-stone-tablets/,
> https://www.kernel.org/pub/software/scm/git/docs/gitworkflows.html#_patch_workflow
> [3] https://colabti.org/irclogger/irclogger_log/git-devel?date=2021-04-12#l40
> [4] https://lore.kernel.org/git/
> _______________________________________________
> Patchwork mailing list
> Patchwork@lists.ozlabs.org
> https://lists.ozlabs.org/listinfo/patchwork

Re: Pain points in Git's patch flow

[brian m. carlson]

[-- Attachment #1: Type: text/plain, Size: 5859 bytes --]

On 2021-04-14 at 06:13:26, Jonathan Nieder wrote:
> Those four are important in my everyday life.  Questions:
> 
>  1. What pain points in the patch flow for git.git are important to
>     you?

I realize I'm a bit late here, but I've been thinking about this some
and wanted to chime in.

I have trouble finding all the spots where people have given me review
feedback.  I have patch mails and responses to those mails go to a
particular folder, but I still often find that I'm not quite sure if
I've gotten every piece of feedback in a review.  Sometimes,
embarrassingly, I don't, and then I have to send another reroll.
Regardless, this makes rerolling a series much slower as I have to comb
my mail multiple times.

I find I'm often unsure what to put in the cover letter for a v2 or
subsequent series.  Clearly people don't want the same thing as v1, but
I rarely have useful information other than a summary of changes.

I have tooling to automatically generate the proper range for
range-diffs in cover letters, but that tooling requires some sort of
manual timestamp, which means I need to go search for my previous series
to find the date and generate the range diff, or if I'm in a rush, I
just have to omit it.  This can take some time, having to guess what I
named the cover letter the last time and search for it in a mailbox with
a 6-digit quantity of mails[0].

In general, I have trouble keeping track of the patch mails I've sent.
I do definitely need to refer to them later, but I don't generally keep
them around on my system since they tend to duplicate my repository, so
I end up needing to find them in my mailbox, which as mentioned, is
slow and error prone.

I find that the git-contacts script is often not helpful to find
reviewers.  When I send out a series, it often suggests Peff and Junio.
While both of them are very capable, they are also not capable of
reviewing every series, and in many cases I know full well that one or
the other is not going to be able to give me a good review (for lack of
familiarity with the SHA-256 work, due to having many other things to
review and to do in life, or for other reasons).  It also,
unfortunately, suggests me as a reviewer for many things, which while
flattering, reflects the fact that I've touched a lot of code and not
that I have a deep understanding of most of the codebase, which I do
not.  For areas where I do have relevant insight, such as the signature
code, I'm often not chosen.

I realize a lot of these are not intrinsic to our workflow and can be
solved with tooling, but because I haven't built that tooling, they're
pain points that I experience in our workflow.

>  2. What tricks do you use to get by with those existing pain points?

I've built some tooling around this, including mail filtering, aliases,
and scripting, but it doesn't seem like enough.  I know others have
built really great tooling for themselves, but by the time I notice
these pain points, it's usually the evening and I don't have time
to build tooling and get things sent out as well.  I also don't
especially enjoy building tooling here.

The friction here makes me less likely to send out patches and much
slower to reroll patches than I'd otherwise be.  And I feel like that
means that I practically can only ever send out a series when I have
more time on the weekend, and as a result, I worry that my patches hold
up others in the tree much more often than I'd like.  It also makes
contributing to Git less fun, which is important since overwhelmingly my
patches are sent on my own time.

>  3. Do you think patchwork goes in a direction that is likely to help
>     with these?

I don't think I know enough about it to say.  If it can more clearly
track review feedback and help keep track of patch emails, I think it
would be a major improvement, for me at least.

>  4. What other tools would you like to see that could help?

I think we definitely need a bug tracker.  We extremely frequently lose
bugs and feature requests on the list and people aren't very likely to
search the list.  If we could use the same one as someone else, such as
the kernel, that would be ideal, because it means people are more likely
to already have an account and therefore the friction to report a bug is
lower.  Alternatively, we could use something like debbugs which is
controllable entirely by email and therefore requires no accounts (but
does require someone to occasionally prune reported spam).

I know full well why we don't use a forge-based model and I'm not
recommending that, but I do want to point out that forges solve all of
my pain points, and I do have a much quicker turnaround time on patches
when I'm using a forge.  So ideally we'd have some standard or
recommended tooling, whether built by us or by others (e.g., an open
source project for patch workflows), that addresses these pain points so
that everyone doesn't have to build their own and turnaround time can be
improved.

I have seen replies downthread that some developers really are reticent
to use more common tooling, like web interfaces.  While I do want to
keep our project as accessible as possible to as many people as
possible, I worry that by catering to folks who don't want to adopt this
tooling, we are drastically reducing the number of possible contributors
of all sorts (code authors, documentation writers, bug reporters) by
not doing so and worsening our own experience in many ways.  I do think
we should adopt modern tooling (e.g., web interfaces) provided that it
is usable for people with accessibility needs, even if that makes some
people unhappy.

[0] I don't, as a general rule, delete emails to this list or otherwise.
-- 
brian m. carlson (he/him or they/them)
Houston, Texas, US

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 263 bytes --]

Re: Pain points in Git's patch flow

[Theodore Ts'o]

On Mon, Apr 26, 2021 at 02:04:32AM +0000, brian m. carlson wrote:
> In general, I have trouble keeping track of the patch mails I've sent.
> I do definitely need to refer to them later, but I don't generally keep
> them around on my system since they tend to duplicate my repository, so
> I end up needing to find them in my mailbox, which as mentioned, is
> slow and error prone.

A quick and easy feature request to this (which I have had as well)
would be implementing a sendmail.fcc config, which if set, would
append any mail messages sent by git send-email to the Unix
mbox file specified by sendmail.fcc.

Once you have the message id of any patch mail that you've sent...

> I have trouble finding all the spots where people have given me review
> feedback.  I have patch mails and responses to those mails go to a
> particular folder, but I still often find that I'm not quite sure if
> I've gotten every piece of feedback in a review.  Sometimes,
> embarrassingly, I don't, and then I have to send another reroll.
> Regardless, this makes rerolling a series much slower as I have to comb
> my mail multiple times.

This becomes pretty easy to solve using existing tooling.  For people
who like web interfaces:

    https://lore.kernel.org/r/<message-id>

(This works today because git@vger.kernel.org is archived by the
lore.kernel.org public-inbox archive.)

Or for those who like CLI's and/or text-based mail readers such as
mutt or pine:

   b4 mbox -o /tmp <message-id>

This will dump the full mail thread (given any any message-id in that
mail thread) to /tmp/<messaige-id>.mbx in Unix mbox format, again
relying on lore.kernel.org.  I've found this to be especially handy if
I've been cc'ed part-way through a mail thread, or if I was only cc'ed
on a single patch and I want to see the full patch series for context.

Cheers,

						- Ted

Re: Pain points in Git's patch flow

[Ævar Arnfjörð Bjarmason]

On Mon, Apr 26 2021, brian m. carlson wrote:

> [[PGP Signed Part:Undecided]]
> On 2021-04-14 at 06:13:26, Jonathan Nieder wrote:
>> [...]
>>  4. What other tools would you like to see that could help?
>
> I think we definitely need a bug tracker.  We extremely frequently lose
> bugs and feature requests on the list and people aren't very likely to
> search the list.  If we could use the same one as someone else, such as
> the kernel, that would be ideal, because it means people are more likely
> to already have an account and therefore the friction to report a bug is
> lower.  Alternatively, we could use something like debbugs which is
> controllable entirely by email and therefore requires no accounts (but
> does require someone to occasionally prune reported spam).
>
> I know full well why we don't use a forge-based model and I'm not
> recommending that, but I do want to point out that forges solve all of
> my pain points, and I do have a much quicker turnaround time on patches
> when I'm using a forge.  So ideally we'd have some standard or
> recommended tooling, whether built by us or by others (e.g., an open
> source project for patch workflows), that addresses these pain points so
> that everyone doesn't have to build their own and turnaround time can be
> improved.
>
> I have seen replies downthread that some developers really are reticent
> to use more common tooling, like web interfaces.  While I do want to
> keep our project as accessible as possible to as many people as
> possible, I worry that by catering to folks who don't want to adopt this
> tooling, we are drastically reducing the number of possible contributors
> of all sorts (code authors, documentation writers, bug reporters) by
> not doing so and worsening our own experience in many ways.  I do think
> we should adopt modern tooling (e.g., web interfaces) provided that it
> is usable for people with accessibility needs, even if that makes some
> people unhappy.

I'm not disagreeing, just replying to point out that I think for your
suggestion & others having as much of a split as possible between "what"
and "how" would, I think, be useful in moving things along.

A web interface is a "how", but it's also implicitly a "what" in the way
that most people think about it in this context.

I.e. it's not like we couldn't have a bug tracker now using the ML,
you'd send a patch, Junio would pick up the report and we'd drop it into
bug/some-description.md (with some handwaiving for formatting, merge
conflicts etc.). We'd remember bug reports, feature requests
etc. forever, and patches could atomically change/close/remove those as
they fix/change/implement them.

The point I'm getting at is that the "what" we're also implicitly
discussing is the developer community shouldering the burden of keeping
such a tracker and the information within it up-to-date.

A web-based interface that worked like our mailing list does now would
be one that, say, deleted your bug 30 days of inactivity (or otherwise
made it as "archived" as something in the ML lore).

I couldn't find a reference to it now, but as I recall (and maybe I
wrote some) there's been some prominent defenses of this model of
development in the past.

I.e. it puts the onus on reporters to make sure their issue is being
addressed, if nobody cares to pick it up it probably wasn't that
important, and we shouldn't so lightly assume the fixed cost of adding
that one-off report to an ever-growing list of reports we'd need to
continually look at / curate / keep up to date etc.

But none of that's an argument I'm looking to get into right now, or
really have much of a firm stance on. I just wanted to point out that
it's a clear case where a "what" is being conflated with a "how".

I.e. we're implicitly not only talking about how something gets done,
but a big change in what gets done.

I had a similar comment upthread (or in a side-thread) about how much of
the suggestions of making use of the various reviewer/approver
PR/MR/whatever tools in the wild seem to simply assume a move away from
the long-time model where there's effectively only one approver/merge
master/committer (i.e. Junio). That's an entirely defensible argument,
but another case of making a "how" and "what" argument at the same time.

Maybe the people proposing both a "how" and a "what" will have an easier
time if those are untangled, and we change one thing at a time.

Re: Pain points in Git's patch flow

[Eric Wong]

"brian m. carlson" <sandals@crustytoothpaste.net> wrote:
> I have tooling to automatically generate the proper range for
> range-diffs in cover letters, but that tooling requires some sort of
> manual timestamp, which means I need to go search for my previous series
> to find the date and generate the range diff, or if I'm in a rush, I
> just have to omit it.  This can take some time, having to guess what I
> named the cover letter the last time and search for it in a mailbox with
> a 6-digit quantity of mails[0].
> 
> In general, I have trouble keeping track of the patch mails I've sent.
> I do definitely need to refer to them later, but I don't generally keep
> them around on my system since they tend to duplicate my repository, so
> I end up needing to find them in my mailbox, which as mentioned, is
> slow and error prone.

Along the lines of what Ted said about Fcc, I've always Bcc-ed
myself on every message I send to verify deliverability and
check/train my spam filter.

What search tool do you use?  mairix can handle the 6-digit
quantity of the git list fairly well.  The following finds all
threads with "sandals" in From/To/Cc:

	mairix -t a:sandals d:YYYYMMDD-YYYYMMDD

and dumps it to whatever Maildir/mbox/IMAP "mfolder" you've
configured.  (prefixes in public-inbox such as "a:", "d:" and
"s:" are stolen from mairix; though mairix ranges use "-" and
public-inbox uses ".." due to Xapian).


I've also heard good things about notmuch, but I archive old
mail to gzipped mboxrd right now[1], and that only supports
Maildir...  I learned to use Xapian by reading code in notmuch.



[1] Fwiw, I'm also working on an AGPL Perl5 storage+search CLI
    that scales to 7/8-digit mail collections.  It's not ready
    for prime-time, yet, but getting there...  (Assuming it
    doesn't set my SSD on fire, first :x)

Re: Pain points in Git's patch flow

[brian m. carlson]

[-- Attachment #1: Type: text/plain, Size: 1552 bytes --]

On 2021-04-28 at 07:59:27, Eric Wong wrote:
> "brian m. carlson" <sandals@crustytoothpaste.net> wrote:
> > I have tooling to automatically generate the proper range for
> > range-diffs in cover letters, but that tooling requires some sort of
> > manual timestamp, which means I need to go search for my previous series
> > to find the date and generate the range diff, or if I'm in a rush, I
> > just have to omit it.  This can take some time, having to guess what I
> > named the cover letter the last time and search for it in a mailbox with
> > a 6-digit quantity of mails[0].
> > 
> > In general, I have trouble keeping track of the patch mails I've sent.
> > I do definitely need to refer to them later, but I don't generally keep
> > them around on my system since they tend to duplicate my repository, so
> > I end up needing to find them in my mailbox, which as mentioned, is
> > slow and error prone.
> 
> Along the lines of what Ted said about Fcc, I've always Bcc-ed
> myself on every message I send to verify deliverability and
> check/train my spam filter.
> 
> What search tool do you use?  mairix can handle the 6-digit
> quantity of the git list fairly well.  The following finds all
> threads with "sandals" in From/To/Cc:
> 
> 	mairix -t a:sandals d:YYYYMMDD-YYYYMMDD

I simply use mutt to read my mailbox and search.  Nothing fancy.  I
should point out that it's not local; it's on a Dovecot IMAP server
located on a VPS in New York City.
-- 
brian m. carlson (he/him or they/them)
Houston, Texas, US

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 263 bytes --]

Re: Pain points in Git's patch flow

[Felipe Contreras]

Eric Wong wrote:
> I've also heard good things about notmuch, but I archive old
> mail to gzipped mboxrd right now[1], and that only supports
> Maildir...  I learned to use Xapian by reading code in notmuch.

For what it's worth I've been using notmuch for several years now and
it's much superior to anything I've seen. I can find all kinds of emails
instantly and using my prefered text-editor (vim).

But yeah, the only problem is getting the mail in the right format in my
machine in the first place. But once I have it, it's blazingly fast to
deal with email.

-- 
Felipe Contreras

Re: Pain points in Git's patch flow

[Felipe Contreras]

brian m. carlson wrote:
> I find I'm often unsure what to put in the cover letter for a v2 or
> subsequent series.  Clearly people don't want the same thing as v1, but
> I rarely have useful information other than a summary of changes.
> 
> I have tooling to automatically generate the proper range for
> range-diffs in cover letters, but that tooling requires some sort of
> manual timestamp, which means I need to go search for my previous series
> to find the date and generate the range diff, or if I'm in a rush, I
> just have to omit it.  This can take some time, having to guess what I
> named the cover letter the last time and search for it in a mailbox with
> a 6-digit quantity of mails[0].

I wrote a tool to deal precisely with that: git-send-series[1]. All the
metadata in the patch series is stored in a text file (version, subject, cc,
cover letter, etc.).

So when you want to send the next version of your series you just do
`git send-series` and it deals with all that tedious stuff
automatically. You just need to update the cover letter.

It also keeps track of the previous versions of your series in order to
automatically generate the range-diff.

> In general, I have trouble keeping track of the patch mails I've sent.
> I do definitely need to refer to them later, but I don't generally keep
> them around on my system since they tend to duplicate my repository, so
> I end up needing to find them in my mailbox, which as mentioned, is
> slow and error prone.

I have my mailbox stored in my machine syncronized with isync[2], and
indexed with notmuch[3]. I can view all mails I've ever sent instantly
with a simple search:

  nmm tag:git tag:sent subject:PATCH

> I find that the git-contacts script is often not helpful to find
> reviewers.

git-contacts is a subpar rewrite of my original script: git-related[4].

Using git-contacts on your last merged patch I get this:

  Cornelius Weig <cornelius.weig@tngtech.com>
  Jeff King <peff@peff.net>
  Junio C Hamano <gitster@pobox.com>
  Johannes Schindelin <Johannes.Schindelin@gmx.de>

However, with git-related I get this:

  Junio C Hamano <gitster@pobox.com> (signer: 62%, author: 37%)
  Johannes Schindelin <Johannes.Schindelin@gmx.de> (author: 37%)
  Jeff King <peff@peff.net> (reviewer: 12%, author: 12%)
  Cornelius Weig <cornelius.weig@tngtech.com> (author: 12%)

Which is much more useful.

However, you actually have options to catch more changes:

  % git related --min-percent=5 --since=10-years-ago 75555676ad -1
  Junio C Hamano <gitster@pobox.com> (signer: 80%, author: 20%)
  Jeff King <peff@peff.net> (reviewer: 10%, author: 30%)
  Johannes Schindelin <Johannes.Schindelin@gmx.de> (author: 30%)
  Patrick Steinhardt <ps@pks.im> (author: 10%)
  Cornelius Weig <cornelius.weig@tngtech.com> (author: 10%)

This may not solve your paticular complaint, but it's clearly superior.

Cheers.

[1] https://github.com/felipec/git-send-series
[2] https://isync.sourceforge.io/
[3] https://notmuchmail.org/
[4] https://github.com/felipec/git-related

-- 
Felipe Contreras

RE: Pain points in Git's patch flow

[Felipe Contreras]

Jonathan Nieder wrote:
> He'll be working on a bit of an experimental project: we want to take
> Patchwork[1], which in principle can be a helpful addition to a
> mailing list centric workflow[2], and improve it to be something that
> people in the Git open source project get day-to-day benefit from.
> Raxel's previous successes in making changes to tools to support a
> better user experience make me excited for the potential for this
> work.

Raxel, I would consider looking at nmbug[1]. It is a tool developed by
the notmuch team in order to deal with patches both in command line and
web interfaces.

As you can see from the discussion that ensured from the original mail,
the git community does care about the command line interface a great
deal.

> Those four are important in my everyday life.  Questions:
> 
>  1. What pain points in the patch flow for git.git are important to
>     you?

My only real pain point is that sometimes a patch series is sent as a
subthread of another patch series, which makes it difficult for me to
mentally separate the two.

>  2. What tricks do you use to get by with those existing pain points?

There is nothing I can do. This is a culture thing that hopefully will
change.

>  3. Do you think patchwork goes in a direction that is likely to help
>     with these?

No.

>  4. What other tools would you like to see that could help?

nmbug[1], but again, it depends on how it's actually used.

Cheers.

[1] https://notmuchmail.org/nmbug/

-- 
Felipe Contreras