RFE: use patchwork to submit a patch

RFE: use patchwork to submit a patch

[Konstantin Ryabitsev]

Hi, all:

I would like to propose a new (large) feature to patchwork with the goal 
to make the process of submitting a patch easier for newbies and people 
generally less familiar with patch-based development. This was discussed 
previously on the workflows list: 
https://lore.kernel.org/workflows/20190930202451.GA14403@pure.paranoia.local/

How I envision this would work:

- user creates an account (which requires a mail confirmation)
- they choose a "submit patch" option from the menu
- the patch submission screen has a succession of screens:

  1. a screen with a single field allowing a user to paste a URL to 
     their fork of the git repository. Once submitted, patchwork does a 
     "git ls-remote" to attempt to get a list of refs and to verify that 
     this is indeed a valid git repository

  2. next screen asks the user to select the ref to work from using the 
     list obtained from the remote. Once submitted, patchwork performs a 
     `git clone --reference` to clone the repository locally using a 
     local fork of the same repo to minimize object transfer. This part 
     requires that:
       a. patchwork project is configured with a path to a local fork, 
          if this feature is enabled for a project
       b. that fork is kept current via some mechanism outside of 
          patchwork (e.g. with grokmirror)
       c. there is some sanity-checking during the clone process to 
          avoid abuse (e.g. a sane timeout, a tmpdir with limited size, 
          etc -- other suggestions welcome)

  3. next screen asks the user to pick a starting commit from the log.  
     Once submitted, patchwork generates the patch from the commit 
     provided to the tip of the branch selected by the user earlier,
     using git format-patch.

  4. next screen asks the user to review the patch to make sure this is 
     what they want to submit. Once confirmed, patchwork performs two 
     admin-defined optional hooks:

       a. a hook to generate a list of cc's (e.g. get_maintainer.pl)
       b. a sanity check hook (e.g. checkpatch.pl)

  5. if sanity checking is defined, next screen shows the output of the 
     sanity check hook, asking confirmation to proceed.

  6. next screen shows the user three fields:

       a. title of the patch/series
       b. cover letter for the patch/series
       c. message-id of the previous patch revision (can be picked from 
          the list of previously submitted series by this user -- 
          patchwork should have them already)
       d. number of the revision (can be auto-filled if previous field 
          is provided) and other tags to include in []

  7. next screen shows final review of what would be sent out to the 
     list (and cc's, if the hook to get cc's is defined and returned any 
     results). Once submitted, patchwork sends the patch/series using 
     patchwork's envelope-from and the user's own email in the From: 
     header.

  8. once sent successfully, cleanups are performed (also needs to be 
     done as part of the regular cron job, for any aborted attempts)

I know this is a pretty big RFE, and I would like to hear your thoughts 
about this. If there is general agreement that this is doable/good idea, 
I may be able to come up with funding for this development as part of 
the overall tooling improvement proposal.

Best regards,
-K

Re: RFE: use patchwork to submit a patch

[Mauro Carvalho Chehab]

Em Thu, 10 Oct 2019 10:41:50 -0400
Konstantin Ryabitsev <konstantin@linuxfoundation.org> escreveu:

> Hi, all:
> 
> I would like to propose a new (large) feature to patchwork with the goal 
> to make the process of submitting a patch easier for newbies and people 
> generally less familiar with patch-based development. This was discussed 
> previously on the workflows list: 
> https://lore.kernel.org/workflows/20190930202451.GA14403@pure.paranoia.local/
> 
> How I envision this would work:
> 
> - user creates an account (which requires a mail confirmation)
> - they choose a "submit patch" option from the menu
> - the patch submission screen has a succession of screens:
> 
>   1. a screen with a single field allowing a user to paste a URL to 
>      their fork of the git repository. 

This will raise the bar, as it will force all developers to have a
public site to host the tree. I guess only a fraction of the 4k kernel
devs have it... In special, the ones that just want to send us a patch
fixing a bug may have serious troubles implementing that.

> Once submitted, patchwork does a 
>      "git ls-remote" to attempt to get a list of refs and to verify that 
>      this is indeed a valid git repository
> 
>   2. next screen asks the user to select the ref to work from using the 
>      list obtained from the remote. Once submitted, patchwork performs a 
>      `git clone --reference` to clone the repository locally using a 
>      local fork of the same repo to minimize object transfer. This part 
>      requires that:
>        a. patchwork project is configured with a path to a local fork, 
>           if this feature is enabled for a project
>        b. that fork is kept current via some mechanism outside of 
>           patchwork (e.g. with grokmirror)
>        c. there is some sanity-checking during the clone process to 
>           avoid abuse (e.g. a sane timeout, a tmpdir with limited size, 
>           etc -- other suggestions welcome)

That would require a high bandwidth at the machine with as patchwork.

Also, doesn't sound a good idea to me, as the server may end by having
tons of open sections, most waiting for tens of minutes, in order to
complete git clone.

> 
>   3. next screen asks the user to pick a starting commit from the log.  
>      Once submitted, patchwork generates the patch from the commit 
>      provided to the tip of the branch selected by the user earlier,
>      using git format-patch.
> 
>   4. next screen asks the user to review the patch to make sure this is 
>      what they want to submit. Once confirmed, patchwork performs two 
>      admin-defined optional hooks:
> 
>        a. a hook to generate a list of cc's (e.g. get_maintainer.pl)
>        b. a sanity check hook (e.g. checkpatch.pl)
> 
>   5. if sanity checking is defined, next screen shows the output of the 
>      sanity check hook, asking confirmation to proceed.
> 
>   6. next screen shows the user three fields:
> 
>        a. title of the patch/series
>        b. cover letter for the patch/series
>        c. message-id of the previous patch revision (can be picked from 
>           the list of previously submitted series by this user -- 
>           patchwork should have them already)
>        d. number of the revision (can be auto-filled if previous field 
>           is provided) and other tags to include in []
> 
>   7. next screen shows final review of what would be sent out to the 
>      list (and cc's, if the hook to get cc's is defined and returned any 
>      results). Once submitted, patchwork sends the patch/series using 
>      patchwork's envelope-from and the user's own email in the From: 
>      header.
> 
>   8. once sent successfully, cleanups are performed (also needs to be 
>      done as part of the regular cron job, for any aborted attempts)
> 
> I know this is a pretty big RFE, and I would like to hear your thoughts 
> about this. If there is general agreement that this is doable/good idea, 
> I may be able to come up with funding for this development as part of 
> the overall tooling improvement proposal.

The procedure itself is not bad, but, if implemented, IMHO, it should,
instead, be something that will run at the machine of the one submitting
the patch. For instance, this could be a perl or python script inside 
Kernel's ./script directory that would handle everything locally, and
then submit the patch via patchwork's REST API.

By using the REST API, it would avoid the need of having to do special
e-mail setups for the casual developers.

Thanks,
Mauro

Re: RFE: use patchwork to submit a patch

[Steven Rostedt]

On Thu, 10 Oct 2019 15:07:29 -0300
Mauro Carvalho Chehab <mchehab+samsung@kernel.org> wrote:

> Em Thu, 10 Oct 2019 10:41:50 -0400
> Konstantin Ryabitsev <konstantin@linuxfoundation.org> escreveu:
> 
> > Hi, all:
> > 
> > I would like to propose a new (large) feature to patchwork with the goal 
> > to make the process of submitting a patch easier for newbies and people 
> > generally less familiar with patch-based development. This was discussed 
> > previously on the workflows list: 
> > https://lore.kernel.org/workflows/20190930202451.GA14403@pure.paranoia.local/
> > 
> > How I envision this would work:
> > 
> > - user creates an account (which requires a mail confirmation)
> > - they choose a "submit patch" option from the menu
> > - the patch submission screen has a succession of screens:
> > 
> >   1. a screen with a single field allowing a user to paste a URL to 
> >      their fork of the git repository.   
> 
> This will raise the bar, as it will force all developers to have a
> public site to host the tree. I guess only a fraction of the 4k kernel
> devs have it... In special, the ones that just want to send us a patch
> fixing a bug may have serious troubles implementing that.

A lot of people are going the way of gitlab/github. I don't think this
raises the bar very high. Anyone can trivially get a public git
repository somewhere.

We can still have email coming in for those die hards, as long as we
have someone to proxy it in. Note, I'm closer to one of the die hards,
but I see where the world is heading.

> 
> > Once submitted, patchwork does a 
> >      "git ls-remote" to attempt to get a list of refs and to verify that 
> >      this is indeed a valid git repository
> > 
> >   2. next screen asks the user to select the ref to work from using the 
> >      list obtained from the remote. Once submitted, patchwork performs a 
> >      `git clone --reference` to clone the repository locally using a 
> >      local fork of the same repo to minimize object transfer. This part 
> >      requires that:
> >        a. patchwork project is configured with a path to a local fork, 
> >           if this feature is enabled for a project
> >        b. that fork is kept current via some mechanism outside of 
> >           patchwork (e.g. with grokmirror)
> >        c. there is some sanity-checking during the clone process to 
> >           avoid abuse (e.g. a sane timeout, a tmpdir with limited size, 
> >           etc -- other suggestions welcome)  
> 
> That would require a high bandwidth at the machine with as patchwork.

Which most solutions may not be able to avoid this.

> 
> Also, doesn't sound a good idea to me, as the server may end by having
> tons of open sections, most waiting for tens of minutes, in order to
> complete git clone.
> 
> > 
> >   3. next screen asks the user to pick a starting commit from the log.  
> >      Once submitted, patchwork generates the patch from the commit 
> >      provided to the tip of the branch selected by the user earlier,
> >      using git format-patch.
> > 
> >   4. next screen asks the user to review the patch to make sure this is 
> >      what they want to submit. Once confirmed, patchwork performs two 
> >      admin-defined optional hooks:
> > 
> >        a. a hook to generate a list of cc's (e.g. get_maintainer.pl)
> >        b. a sanity check hook (e.g. checkpatch.pl)
> > 
> >   5. if sanity checking is defined, next screen shows the output of the 
> >      sanity check hook, asking confirmation to proceed.
> > 
> >   6. next screen shows the user three fields:
> > 
> >        a. title of the patch/series
> >        b. cover letter for the patch/series
> >        c. message-id of the previous patch revision (can be picked from 
> >           the list of previously submitted series by this user -- 
> >           patchwork should have them already)
> >        d. number of the revision (can be auto-filled if previous field 
> >           is provided) and other tags to include in []
> > 
> >   7. next screen shows final review of what would be sent out to the 
> >      list (and cc's, if the hook to get cc's is defined and returned any 
> >      results). Once submitted, patchwork sends the patch/series using 
> >      patchwork's envelope-from and the user's own email in the From: 
> >      header.
> > 
> >   8. once sent successfully, cleanups are performed (also needs to be 
> >      done as part of the regular cron job, for any aborted attempts)
> > 
> > I know this is a pretty big RFE, and I would like to hear your thoughts 
> > about this. If there is general agreement that this is doable/good idea, 
> > I may be able to come up with funding for this development as part of 
> > the overall tooling improvement proposal.  
> 
> The procedure itself is not bad, but, if implemented, IMHO, it should,
> instead, be something that will run at the machine of the one submitting
> the patch. For instance, this could be a perl or python script inside 
> Kernel's ./script directory that would handle everything locally, and
> then submit the patch via patchwork's REST API.
> 
> By using the REST API, it would avoid the need of having to do special
> e-mail setups for the casual developers.

I agree that having the most work done on the submitter's side is
probably the best. But we need to be very careful about security in any
way we do this.

-- Steve

Re: RFE: use patchwork to submit a patch

[Konstantin Ryabitsev]

On Thu, Oct 10, 2019 at 03:07:29PM -0300, Mauro Carvalho Chehab wrote:
>> - the patch submission screen has a succession of screens:
>>
>>   1. a screen with a single field allowing a user to paste a URL to
>>      their fork of the git repository.
>
>This will raise the bar, as it will force all developers to have a
>public site to host the tree. I guess only a fraction of the 4k kernel
>devs have it... In special, the ones that just want to send us a patch
>fixing a bug may have serious troubles implementing that.

I don't think this will raise the bar, as Github/Gitlab allow for very 
easy forking of https://github.com/torvalds/linux. This is also not at 
all aimed at "all developers" -- only those that don't want to use the 
current CLI workflow and are more comfortable with web tools like 
Github.

>>   2. next screen asks the user to select the ref to work from using 
>>      the
>>      list obtained from the remote. Once submitted, patchwork performs a
>>      `git clone --reference` to clone the repository locally using a
>>      local fork of the same repo to minimize object transfer. This part
>>      requires that:
>>        a. patchwork project is configured with a path to a local fork,
>>           if this feature is enabled for a project
>>        b. that fork is kept current via some mechanism outside of
>>           patchwork (e.g. with grokmirror)
>>        c. there is some sanity-checking during the clone process to
>>           avoid abuse (e.g. a sane timeout, a tmpdir with limited size,
>>           etc -- other suggestions welcome)
>
>That would require a high bandwidth at the machine with as patchwork.
>Also, doesn't sound a good idea to me, as the server may end by having
>tons of open sections, most waiting for tens of minutes, in order to
>complete git clone.

This is actually really fast if you already have a local copy of the 
repository with most objects. Try this yourself if you have 
torvalds/linux.git locally:

git clone --bare -s torvalds/linux.git test
cd test
git remote add arm-soc https://git.kernel.org/pub/scm/linux/kernel/git/arm/arm-soc
git fetch arm-soc for-next

The whole process takes a second or so and the resulting repo is 328K in 
size.

Of course, this assumes that the remote repository isn't trying to do 
something nasty, which is why I suggest anti-abuse precautions.

>> I know this is a pretty big RFE, and I would like to hear your 
>> thoughts
>> about this. If there is general agreement that this is doable/good idea,
>> I may be able to come up with funding for this development as part of
>> the overall tooling improvement proposal.
>
>The procedure itself is not bad, but, if implemented, IMHO, it should,
>instead, be something that will run at the machine of the one submitting
>the patch. For instance, this could be a perl or python script inside
>Kernel's ./script directory that would handle everything locally, and
>then submit the patch via patchwork's REST API.

I think I didn't make clear that this isn't supposed to go straight to 
patchwork. Patchwork is merely a convenient tool where this happens -- 
the resulting patch gets mailed out to the mailing list just as the user 
would have done.

-K

Re: RFE: use patchwork to submit a patch

[Eric Wong]

Konstantin Ryabitsev <konstantin@linuxfoundation.org> wrote:
> This is actually really fast if you already have a local copy of the
> repository with most objects. Try this yourself if you have
> torvalds/linux.git locally:
> 
> git clone --bare -s torvalds/linux.git test

Yep, -s (--shared) makes cloning really cheap.  One of my goals is to get

	git clone -s https://example.com/torvalds/linux.git

and

	git clone -s https://example.com/torvalds/linux.git/clone.bundle

working.  That would make it easier for new contributors to
setup lightweight clones and pull in history on an as-needed
basis w/o hacks like shallow cloning.

Re: RFE: use patchwork to submit a patch

[Jonathan Nieder]

(+cc: git)
Hi,

Konstantin Ryabitsev wrote[1]:

> How I envision this would work:
>
> - user creates an account (which requires a mail confirmation)
> - they choose a "submit patch" option from the menu
> - the patch submission screen has a succession of screens:

Interesting!  This reminds me a bit of https://gitgitgadget.github.io
(except using patchwork instead of github pull requests as substrate).

That leads me to wonder: should these kinds of tools share some code?
Are there any subtleties that web-to-mail patch submission interfaces
have in common or can learn from each other?

Thanks,
Jonathan

[1] https://lore.kernel.org/workflows/20191010144150.hqiosvwolm3lmzp5@chatter.i7.local/

Re: RFE: use patchwork to submit a patch

[Jonathan Nieder]

Eric Wong wrote:
> Konstantin Ryabitsev <konstantin@linuxfoundation.org> wrote:

>> This is actually really fast if you already have a local copy of the
>> repository with most objects. Try this yourself if you have
>> torvalds/linux.git locally:
>>
>> git clone --bare -s torvalds/linux.git test
>
> Yep, -s (--shared) makes cloning really cheap.  One of my goals is to get
>
> 	git clone -s https://example.com/torvalds/linux.git
>
> and
>
> 	git clone -s https://example.com/torvalds/linux.git/clone.bundle
>
> working.  That would make it easier for new contributors to
> setup lightweight clones and pull in history on an as-needed
> basis w/o hacks like shallow cloning.

Does "git clone --filter=blob:none" do what you're looking for?

Thanks,
Jonathan

Re: RFE: use patchwork to submit a patch

[Eric Wong]

Jonathan Nieder <jrnieder@gmail.com> wrote:
> Eric Wong wrote:
> > Konstantin Ryabitsev <konstantin@linuxfoundation.org> wrote:
> 
> >> This is actually really fast if you already have a local copy of the
> >> repository with most objects. Try this yourself if you have
> >> torvalds/linux.git locally:
> >>
> >> git clone --bare -s torvalds/linux.git test
> >
> > Yep, -s (--shared) makes cloning really cheap.  One of my goals is to get
> >
> > 	git clone -s https://example.com/torvalds/linux.git
> >
> > and
> >
> > 	git clone -s https://example.com/torvalds/linux.git/clone.bundle
> >
> > working.  That would make it easier for new contributors to
> > setup lightweight clones and pull in history on an as-needed
> > basis w/o hacks like shallow cloning.
> 
> Does "git clone --filter=blob:none" do what you're looking for?

Oops, haven't seen that new feature :x  And haven't tried, pu @
8d9027fa59b943db96a8a9090ec31d7f0f935596 is broken due to
conflicts with hashmap (probably won't have time for a bit to
look at it).

What I'm hoping to do with "git clone -s" would be client-only
and compatible with existing HTTP servers that run
"git update-server-info".

Re: RFE: use patchwork to submit a patch

[Daniel Axtens]

Hi Konstantin,

tl;dr: I think a git-to-email bridge is a good step. I'm not sure why
patchwork would be the thing to build it on top of, and I'm worried that
it would slow us both down. I'm very open to being convinced though.

> I would like to propose a new (large) feature to patchwork with the goal 
> to make the process of submitting a patch easier for newbies and people 
> generally less familiar with patch-based development. This was discussed 
> previously on the workflows list: 
> https://lore.kernel.org/workflows/20190930202451.GA14403@pure.paranoia.local/
>
> How I envision this would work:
>
If I understand correctly, you're using Patchwork as:

> - user creates an account (which requires a mail confirmation)

 a) an identity provider, and

> - they choose a "submit patch" option from the menu
> - the patch submission screen has a succession of screens:
>
>   1. a screen with a single field allowing a user to paste a URL to 
>      their fork of the git repository. Once submitted, patchwork does a 
>      "git ls-remote" to attempt to get a list of refs and to verify that 
>      this is indeed a valid git repository
>
>   2. next screen asks the user to select the ref to work from using the 
>      list obtained from the remote. Once submitted, patchwork performs a 
>      `git clone --reference` to clone the repository locally using a 
>      local fork of the same repo to minimize object transfer. This part 
>      requires that:
>        a. patchwork project is configured with a path to a local fork, 
>           if this feature is enabled for a project

 b) a way to integrate with existing concepts of a project and keep
    metadata about them in 1 place

>        b. that fork is kept current via some mechanism outside of 
>           patchwork (e.g. with grokmirror)
>        c. there is some sanity-checking during the clone process to 
>           avoid abuse (e.g. a sane timeout, a tmpdir with limited size, 
>           etc -- other suggestions welcome)
>
>   3. next screen asks the user to pick a starting commit from the log.  
>      Once submitted, patchwork generates the patch from the commit 
>      provided to the tip of the branch selected by the user earlier,
>      using git format-patch.
>
>   4. next screen asks the user to review the patch to make sure this is 
>      what they want to submit. Once confirmed, patchwork performs two 
>      admin-defined optional hooks:
>
>        a. a hook to generate a list of cc's (e.g. get_maintainer.pl)
>        b. a sanity check hook (e.g. checkpatch.pl)
>
>   5. if sanity checking is defined, next screen shows the output of the 
>      sanity check hook, asking confirmation to proceed.
>
>   6. next screen shows the user three fields:
>
>        a. title of the patch/series
>        b. cover letter for the patch/series
>        c. message-id of the previous patch revision (can be picked from 
>           the list of previously submitted series by this user -- 
>           patchwork should have them already)

 c) a handy tool for getting previous series by a given user.

>        d. number of the revision (can be auto-filled if previous field 
>           is provided) and other tags to include in []
>
>   7. next screen shows final review of what would be sent out to the 
>      list (and cc's, if the hook to get cc's is defined and returned any 
>      results). Once submitted, patchwork sends the patch/series using 
>      patchwork's envelope-from and the user's own email in the From: 
>      header.

 d) a 'trusted' source of email.

Is that right? I just ask because this idea seems a long way from what
Patchwork traditionally does. That's not necessarily bad, I just want to
make sure I understand, and that if you get funding you're not tying
yourself to a platform that doesn't suit your needs.

I'm particularly curious about Patchwork as (a) an identity
provider. You wrote:

> - user creates an account (which requires a mail confirmation)

This seems like "optional centralising" on Patchwork - it becomes a
central identity provider but it's optional in that you can just send
email directly if you prefer.

If you're going to do 'lightweight' centralisation like that, why not do
it on a platform that already understands git? It's really easy to
extract the information you describe in (c) just by querying the
patchwork API. You don't need to actually integrate into patchwork, or
be logged in, to do that. You lose the ability to load any git remote,
but if you have a git remote that isn't github or gitlab, you probably
already have a good email flow (e.g. if you repo is on kernel.org).

If you really want to use Patchwork as an identity provider, rather than
a forge, could we just teach Patchwork how to be an identity broker, and
then build things separately, authenticating through Patchwork to
confirm a user's identity? That means you could build in whatever
language you like and, critically, run on whatever deployment schedule
you want. You could also get much better isolation that way, which would
be good - I don't want an RCE in the git library to allow someone to
wipe out all patchwork data, for example.

> I know this is a pretty big RFE, and I would like to hear your thoughts 
> about this. If there is general agreement that this is doable/good idea, 
> I may be able to come up with funding for this development as part of 
> the overall tooling improvement proposal.

As I said up top, I'm not opposed to this per se. I think a git-to-email
bridge is a good step. I'm just confused as to why patchwork would be
the thing to build it on top of, and I'm worried about how you'd deploy
and update this extended Patchwork. I'm very open to being convinced
though.

Regards,
Daniel

Re: RFE: use patchwork to submit a patch

[Konstantin Ryabitsev]

On Fri, Oct 11, 2019 at 08:38:49AM +1100, Daniel Axtens wrote:
>Hi Konstantin,
>
>tl;dr: I think a git-to-email bridge is a good step. I'm not sure why
>patchwork would be the thing to build it on top of, and I'm worried that
>it would slow us both down. I'm very open to being convinced though.

In very broad terms, I chose patchwork because:

1. in people's minds patchwork.kernel.org already deals with patches and 
   mailing lists, so it seems to be a logical place for such tool to 
   live
2. it's already built on top of a powerful web system (django), and we 
   already have it up and running, so this wouldn't require setting up 
   Yet Another Web Framework Service


I will readily admit that both of these assertions are pretty tenuous.

>If I understand correctly, you're using Patchwork as:
>
>> - user creates an account (which requires a mail confirmation)
>
> a) an identity provider, and

Well, rather as a tool that already has account management which 
includes email confirmation at some point.

> b) a way to integrate with existing concepts of a project and keep
>    metadata about them in 1 place

Yes.

> c) a handy tool for getting previous series by a given user.

Yes, it's convenient to already have that user's previously submitted 
series readily available.

> d) a 'trusted' source of email.

Well, this part isn't really that important. Rather, it's a tool where, 
to have an account, one must confirm email delivery.

>Is that right? I just ask because this idea seems a long way from what
>Patchwork traditionally does. That's not necessarily bad, I just want to
>make sure I understand, and that if you get funding you're not tying
>yourself to a platform that doesn't suit your needs.

Well, in my mind patchwork:

1. already deals with patches and series, including knowing how to do 
   diff highlights and all the fancy stuff
2. will hopefully gain ability to do interdiffs in the future, so if 
   someone submits a series revision, they can see what actually changed 
   before their previous submission and their new attempt
3. already has a lot of knowledge around git, mboxes, formats, etc.

This, of course, is not to say that patchwork is where this *must* 
happen, but I think it would be *nice* if this is where this happens. :)

>I'm particularly curious about Patchwork as (a) an identity
>provider. You wrote:
>
>> - user creates an account (which requires a mail confirmation)
>
>This seems like "optional centralising" on Patchwork - it becomes a
>central identity provider but it's optional in that you can just send
>email directly if you prefer.

Right, this is a tool to help people allergic to CLI (or who do this 
infrequently enough that they can never remember all the steps, and oh 
my god, why isn't there a web tool to hold my hand?).

>If you're going to do 'lightweight' centralisation like that, why not do
>it on a platform that already understands git? It's really easy to
>extract the information you describe in (c) just by querying the
>patchwork API. You don't need to actually integrate into patchwork, or
>be logged in, to do that. You lose the ability to load any git remote,
>but if you have a git remote that isn't github or gitlab, you probably
>already have a good email flow (e.g. if you repo is on kernel.org).
>
>If you really want to use Patchwork as an identity provider, rather than
>a forge, could we just teach Patchwork how to be an identity broker, and
>then build things separately, authenticating through Patchwork to
>confirm a user's identity? That means you could build in whatever
>language you like and, critically, run on whatever deployment schedule
>you want. You could also get much better isolation that way, which would
>be good - I don't want an RCE in the git library to allow someone to
>wipe out all patchwork data, for example.

Well, ve hawe vays of prewenting that (e.g. by transitioning git calls 
into their own selinux domain which cannot talk to databases). 

>> I know this is a pretty big RFE, and I would like to hear your thoughts
>> about this. If there is general agreement that this is doable/good idea,
>> I may be able to come up with funding for this development as part of
>> the overall tooling improvement proposal.
>
>As I said up top, I'm not opposed to this per se. I think a git-to-email
>bridge is a good step. I'm just confused as to why patchwork would be
>the thing to build it on top of, and I'm worried about how you'd deploy
>and update this extended Patchwork. I'm very open to being convinced
>though.

Generally, I think patchwork, as a web application that already deals 
with patches and series is a convenient place for this tool to live, 
that's basically the extent of my thinking. For sure, it can exist as a 
separate tool, but then I'd have to set up and maintain that separate 
tool in addition to patchwork, as opposed to just patchwork.

Best,
-K

Re: RFE: use patchwork to submit a patch

[Greg KH]

On Thu, Oct 10, 2019 at 10:41:50AM -0400, Konstantin Ryabitsev wrote:
> Hi, all:
> 
> I would like to propose a new (large) feature to patchwork with the goal to
> make the process of submitting a patch easier for newbies and people
> generally less familiar with patch-based development. This was discussed
> previously on the workflows list:
> https://lore.kernel.org/workflows/20190930202451.GA14403@pure.paranoia.local/
> 
> How I envision this would work:
> 
> - user creates an account (which requires a mail confirmation)
> - they choose a "submit patch" option from the menu
> - the patch submission screen has a succession of screens:
> 
>  1. a screen with a single field allowing a user to paste a URL to     their
> fork of the git repository. Once submitted, patchwork does a     "git
> ls-remote" to attempt to get a list of refs and to verify that     this is
> indeed a valid git repository

s/valid git repository/valid git repository based on the kernel git tree/

Otherwise you might be sending out lots of emails for other projects :)

> 
>  2. next screen asks the user to select the ref to work from using the
> list obtained from the remote. Once submitted, patchwork performs a     `git
> clone --reference` to clone the repository locally using a     local fork of
> the same repo to minimize object transfer. This part     requires that:
>       a. patchwork project is configured with a path to a local fork,
> if this feature is enabled for a project
>       b. that fork is kept current via some mechanism outside of
> patchwork (e.g. with grokmirror)
>       c. there is some sanity-checking during the clone process to
> avoid abuse (e.g. a sane timeout, a tmpdir with limited size,          etc
> -- other suggestions welcome)
> 
>  3. next screen asks the user to pick a starting commit from the log.
> Once submitted, patchwork generates the patch from the commit     provided
> to the tip of the branch selected by the user earlier,
>     using git format-patch.
> 
>  4. next screen asks the user to review the patch to make sure this is
> what they want to submit. Once confirmed, patchwork performs two
> admin-defined optional hooks:
> 
>       a. a hook to generate a list of cc's (e.g. get_maintainer.pl)
>       b. a sanity check hook (e.g. checkpatch.pl)

I will note that many "first patch" submissions are checkpatch.pl
cleanups for staging.  When doing that, I require that they do "one
logical change per patch", which means that many of the individual
patches themselves will not be checkpatch.pl clean, because many lines
have multiple issues with them (tabs, spaces, format, length, etc.)

So other than that minor thing, sounds interesting.  It's hard to
determine just how difficult the whole "set up git and send a patch out"
process is for people these days given the _huge_ numbers of new
contributions we keep getting, and the numerous good tutorials we have
created that spell out exactly how to do this.

So you might be "solving" a problem that we don't really have.  It's
hard to tell :(

> I know this is a pretty big RFE, and I would like to hear your thoughts
> about this. If there is general agreement that this is doable/good idea, I
> may be able to come up with funding for this development as part of the
> overall tooling improvement proposal.

The workflow seems sane, and matches what most people do today, with the
exception that it "solves" the git send-email issue, right?  Is that our
biggest barrier?

I would recommend interviewing some of the recent kernel mentor project
and outreachy applicants first, to try to determine exactly what their
problems, if any, were with our development process.  If they say that
this type of tool/workflow would have saved them hours of time and
energy, then that's a great indication that we should try to do this.

thanks,

greg k-h

Re: RFE: use patchwork to submit a patch

[Shuah Khan]

On 10/11/19 2:57 AM, Greg KH wrote:
> On Thu, Oct 10, 2019 at 10:41:50AM -0400, Konstantin Ryabitsev wrote:
>> Hi, all:
>>
>> I would like to propose a new (large) feature to patchwork with the goal to
>> make the process of submitting a patch easier for newbies and people
>> generally less familiar with patch-based development. This was discussed
>> previously on the workflows list:
>> https://lore.kernel.org/workflows/20190930202451.GA14403@pure.paranoia.local/
>>
>> How I envision this would work:
>>
>> - user creates an account (which requires a mail confirmation) >> - they choose a "submit patch" option from the menu
>> - the patch submission screen has a succession of screens:
>>
>>   1. a screen with a single field allowing a user to paste a URL to     their
>> fork of the git repository. Once submitted, patchwork does a     "git
>> ls-remote" to attempt to get a list of refs and to verify that     this is
>> indeed a valid git repository
> 
> s/valid git repository/valid git repository based on the kernel git tree/
> 
> Otherwise you might be sending out lots of emails for other projects :)
> 
>>
>>   2. next screen asks the user to select the ref to work from using the
>> list obtained from the remote. Once submitted, patchwork performs a     `git
>> clone --reference` to clone the repository locally using a     local fork of
>> the same repo to minimize object transfer. This part     requires that:
>>        a. patchwork project is configured with a path to a local fork,
>> if this feature is enabled for a project
>>        b. that fork is kept current via some mechanism outside of
>> patchwork (e.g. with grokmirror)
>>        c. there is some sanity-checking during the clone process to
>> avoid abuse (e.g. a sane timeout, a tmpdir with limited size,          etc
>> -- other suggestions welcome)
>>
>>   3. next screen asks the user to pick a starting commit from the log.
>> Once submitted, patchwork generates the patch from the commit     provided
>> to the tip of the branch selected by the user earlier,
>>      using git format-patch.
>>
>>   4. next screen asks the user to review the patch to make sure this is
>> what they want to submit. Once confirmed, patchwork performs two
>> admin-defined optional hooks:
>>
>>        a. a hook to generate a list of cc's (e.g. get_maintainer.pl)
>>        b. a sanity check hook (e.g. checkpatch.pl)
> 
> I will note that many "first patch" submissions are checkpatch.pl
> cleanups for staging.  When doing that, I require that they do "one
> logical change per patch", which means that many of the individual
> patches themselves will not be checkpatch.pl clean, because many lines
> have multiple issues with them (tabs, spaces, format, length, etc.)
> 
> So other than that minor thing, sounds interesting.  It's hard to
> determine just how difficult the whole "set up git and send a patch out"
> process is for people these days given the _huge_ numbers of new
> contributions we keep getting, and the numerous good tutorials we have
> created that spell out exactly how to do this.
> 
> So you might be "solving" a problem that we don't really have.  It's
> hard to tell :(
> 

I agree with this. I don't think this a problem that is worth solving.
When a new developer wants to send a patch, they don't need to create
any accounts. They setup their email client and send patch.

We have several resources that walk them through setting up email
clients and sending patches. checkpatch.pl can be automated with
git hooks.

>> I know this is a pretty big RFE, and I would like to hear your thoughts
>> about this. If there is general agreement that this is doable/good idea, I
>> may be able to come up with funding for this development as part of the
>> overall tooling improvement proposal.
> 
> The workflow seems sane, and matches what most people do today, with the
> exception that it "solves" the git send-email issue, right?  Is that our
> biggest barrier?
> 
> I would recommend interviewing some of the recent kernel mentor project
> and outreachy applicants first, to try to determine exactly what their
> problems, if any, were with our development process.  If they say that
> this type of tool/workflow would have saved them hours of time and
> energy, then that's a great indication that we should try to do this.
> 

I would say considering the number of applicants to mentorship program
and new developers it will be lot overhead to require them to create
patchwork accounts, and it might even be hard overtime. A lot of them
start out and drop out in the middle. With the current setup, nothing
to cleanup.

Setting up email clients and git hooks is one time task. It is the
easiest of the learning curve for many new developers. New developers
struggle with getting the change logs right, coding styles right, and
responding to review comments and acting on them.

These aren't something that can be automated and they just have to
learn through experience of sending patches.

My opinion based on contact with new developers as well running the
mentorship program, I would sat this isn't something that needs
solving.

thanks,
-- Shuah

Re: RFE: use patchwork to submit a patch

[Mauro Carvalho Chehab]

Em Fri, 11 Oct 2019 11:20:17 -0600
Shuah Khan <skhan@linuxfoundation.org> escreveu:

> On 10/11/19 2:57 AM, Greg KH wrote:
> > On Thu, Oct 10, 2019 at 10:41:50AM -0400, Konstantin Ryabitsev wrote:  
> >> Hi, all:
> >>
> >> I would like to propose a new (large) feature to patchwork with the goal to
> >> make the process of submitting a patch easier for newbies and people
> >> generally less familiar with patch-based development. This was discussed
> >> previously on the workflows list:
> >> https://lore.kernel.org/workflows/20190930202451.GA14403@pure.paranoia.local/
> >>
> >> How I envision this would work:
> >>
> >> - user creates an account (which requires a mail confirmation) >> - they choose a "submit patch" option from the menu
> >> - the patch submission screen has a succession of screens:
> >>
> >>   1. a screen with a single field allowing a user to paste a URL to     their
> >> fork of the git repository. Once submitted, patchwork does a     "git
> >> ls-remote" to attempt to get a list of refs and to verify that     this is
> >> indeed a valid git repository  
> > 
> > s/valid git repository/valid git repository based on the kernel git tree/
> > 
> > Otherwise you might be sending out lots of emails for other projects :)
> >   
> >>
> >>   2. next screen asks the user to select the ref to work from using the
> >> list obtained from the remote. Once submitted, patchwork performs a     `git
> >> clone --reference` to clone the repository locally using a     local fork of
> >> the same repo to minimize object transfer. This part     requires that:
> >>        a. patchwork project is configured with a path to a local fork,
> >> if this feature is enabled for a project
> >>        b. that fork is kept current via some mechanism outside of
> >> patchwork (e.g. with grokmirror)
> >>        c. there is some sanity-checking during the clone process to
> >> avoid abuse (e.g. a sane timeout, a tmpdir with limited size,          etc
> >> -- other suggestions welcome)
> >>
> >>   3. next screen asks the user to pick a starting commit from the log.
> >> Once submitted, patchwork generates the patch from the commit     provided
> >> to the tip of the branch selected by the user earlier,
> >>      using git format-patch.
> >>
> >>   4. next screen asks the user to review the patch to make sure this is
> >> what they want to submit. Once confirmed, patchwork performs two
> >> admin-defined optional hooks:
> >>
> >>        a. a hook to generate a list of cc's (e.g. get_maintainer.pl)
> >>        b. a sanity check hook (e.g. checkpatch.pl)  
> > 
> > I will note that many "first patch" submissions are checkpatch.pl
> > cleanups for staging.  When doing that, I require that they do "one
> > logical change per patch", which means that many of the individual
> > patches themselves will not be checkpatch.pl clean, because many lines
> > have multiple issues with them (tabs, spaces, format, length, etc.)
> > 
> > So other than that minor thing, sounds interesting.  It's hard to
> > determine just how difficult the whole "set up git and send a patch out"
> > process is for people these days given the _huge_ numbers of new
> > contributions we keep getting, and the numerous good tutorials we have
> > created that spell out exactly how to do this.
> > 
> > So you might be "solving" a problem that we don't really have.  It's
> > hard to tell :(
> >   
> 
> I agree with this. I don't think this a problem that is worth solving.
> When a new developer wants to send a patch, they don't need to create
> any accounts. They setup their email client and send patch.
> 
> We have several resources that walk them through setting up email
> clients and sending patches. checkpatch.pl can be automated with
> git hooks.
> 
> >> I know this is a pretty big RFE, and I would like to hear your thoughts
> >> about this. If there is general agreement that this is doable/good idea, I
> >> may be able to come up with funding for this development as part of the
> >> overall tooling improvement proposal.  
> > 
> > The workflow seems sane, and matches what most people do today, with the
> > exception that it "solves" the git send-email issue, right?  Is that our
> > biggest barrier?
> > 
> > I would recommend interviewing some of the recent kernel mentor project
> > and outreachy applicants first, to try to determine exactly what their
> > problems, if any, were with our development process.  If they say that
> > this type of tool/workflow would have saved them hours of time and
> > energy, then that's a great indication that we should try to do this.
> >   
> 
> I would say considering the number of applicants to mentorship program
> and new developers it will be lot overhead to require them to create
> patchwork accounts, and it might even be hard overtime. A lot of them
> start out and drop out in the middle. With the current setup, nothing
> to cleanup.
> 
> Setting up email clients and git hooks is one time task. It is the
> easiest of the learning curve for many new developers. New developers
> struggle with getting the change logs right, coding styles right, and
> responding to review comments and acting on them.
> 
> These aren't something that can be automated and they just have to
> learn through experience of sending patches.
> 
> My opinion based on contact with new developers as well running the
> mentorship program, I would sat this isn't something that needs
> solving.

I agree with Shuah. From time to time, we have one time only patch
submission from people that just want to fix something that affects
his machine.

The learning curve for such submissions is usually fast. It only takes one
or two e-mails to be exchanged with the person wanting to submit his
stuff in order to teach them how to do it.

Ok, there's always space to optimize it. 

For example, having something like:

	./scripts/submit_patches

That would automatically do whatever needed to check if patches are ok and
to submit them from origin/master to upstream would be great.

Such script should of course verify if the subject and descriptions appear
to be coherent, it should run checkpatch, run make with some options like
W=1 and check if the number of patches is below a certain default 
(let's say, 10 patches) - in order to prevent newbies to spam our mailing
lists. 

Another thing is to add a web interface for patch submission and requiring
people to open an account at github/gitlab (or on some other git hosting
service). That sounds a lot more complex than what we already have today.

Thanks,
Mauro

Re: RFE: use patchwork to submit a patch

[Steven Rostedt]

On Fri, 11 Oct 2019 14:37:52 -0300
Mauro Carvalho Chehab <mchehab@kernel.org> wrote:

> I agree with Shuah. From time to time, we have one time only patch
> submission from people that just want to fix something that affects
> his machine.

I like to stress this point. There's been projects that I have written
a patch for to fix or update a feature for that project for my own
personal machine. Then when looking at upstreaming it, if the
procedure is more than simply sending a patch to a mailing list, I
usually stop right there, and just maintain the fix locally. (Note,
this is for simple updates for my personal machine, if this is for
something for work, I spend the time to get it upstream).

Thus, if we want people to send us their fixes, we better keep just an
email with a patch the lowest bar for entry.

-- Steve

Re: RFE: use patchwork to submit a patch

[Mauro Carvalho Chehab]

Em Thu, 10 Oct 2019 15:53:35 -0400
Konstantin Ryabitsev <konstantin@linuxfoundation.org> escreveu:

> On Thu, Oct 10, 2019 at 03:07:29PM -0300, Mauro Carvalho Chehab wrote:
> >> - the patch submission screen has a succession of screens:
> >>
> >>   1. a screen with a single field allowing a user to paste a URL to
> >>      their fork of the git repository.  
> >
> >This will raise the bar, as it will force all developers to have a
> >public site to host the tree. I guess only a fraction of the 4k kernel
> >devs have it... In special, the ones that just want to send us a patch
> >fixing a bug may have serious troubles implementing that.  
> 
> I don't think this will raise the bar, as Github/Gitlab allow for very 
> easy forking of https://github.com/torvalds/linux. This is also not at 
> all aimed at "all developers" -- only those that don't want to use the 
> current CLI workflow and are more comfortable with web tools like 
> Github.

I guess people have different views about what's the goal.

If the goal is to attract more developers, the focus should be on
making something that would be simpler than what we current have.

What we currently have here is that just adding this to .git/config:

	[sendemail]
	  smtpserver = smtp.gmail.com
	  smtpserverport = 587
	  smtpencryption = tls
	  smtpuser = <gmail email address>
	  from = <email address for From: field>

Is usually enough for the vast majority of the newcomers.

Using github/gitlab for Kernel development takes a lot more time
and a lot more steps than writing the above at .git/config, even
if the user already have an account there.

Also, instead of just doing:

	git send-email origin..

Your proposal will require to do:

	1) git push github_clone
	2) open the browser
	3) fill the forms, pointing to "github_clone" URL
	4) click at the submission button

That is adding a lot additional complexity. I fail to see any gain
with that.

> >>   2. next screen asks the user to select the ref to work from using 
> >>      the
> >>      list obtained from the remote. Once submitted, patchwork performs a
> >>      `git clone --reference` to clone the repository locally using a
> >>      local fork of the same repo to minimize object transfer. This part
> >>      requires that:
> >>        a. patchwork project is configured with a path to a local fork,
> >>           if this feature is enabled for a project
> >>        b. that fork is kept current via some mechanism outside of
> >>           patchwork (e.g. with grokmirror)
> >>        c. there is some sanity-checking during the clone process to
> >>           avoid abuse (e.g. a sane timeout, a tmpdir with limited size,
> >>           etc -- other suggestions welcome)  
> >
> >That would require a high bandwidth at the machine with as patchwork.
> >Also, doesn't sound a good idea to me, as the server may end by having
> >tons of open sections, most waiting for tens of minutes, in order to
> >complete git clone.  
> 
> This is actually really fast if you already have a local copy of the 
> repository with most objects. Try this yourself if you have 
> torvalds/linux.git locally:
> 
> git clone --bare -s torvalds/linux.git test
> cd test
> git remote add arm-soc https://git.kernel.org/pub/scm/linux/kernel/git/arm/arm-soc
> git fetch arm-soc for-next
> 
> The whole process takes a second or so and the resulting repo is 328K in 
> size.
> 
> Of course, this assumes that the remote repository isn't trying to do 
> something nasty, which is why I suggest anti-abuse precautions.

Well, one could, for example, send a pull request, let's say, from
a DRM development-based tree, into media (or vice versa), with would
require to sync the "alien" patches, just to get the ones that are
useful.

It can even be worse (still valid): the tree to be pulled could be
based on linux-next.

Here, I receive bull requests that are sometimes based on non-media
trees: it may take a few mins to get the patches.

-

Except if the idea is to have this only at kernel.org, and add an
alternates for every single other tree, even a non-nasty PR would
take a while, as not all kernel trees are hosted there.

Also, as you said, one could do something really nasty, like
sending a PR from a huge non-kernel repository into the Kernel.

Not sure how easy/hard would be to avoid that.

This could even happen by mistake, as, at least on some places
(like linuxtv.org) non-kernel trees are also hosted. Btw, on media,
our patchwork instance is also used by VDR, whose project is even 
hosted elsewhere.

> 
> >> I know this is a pretty big RFE, and I would like to hear your 
> >> thoughts
> >> about this. If there is general agreement that this is doable/good idea,
> >> I may be able to come up with funding for this development as part of
> >> the overall tooling improvement proposal.  
> >
> >The procedure itself is not bad, but, if implemented, IMHO, it should,
> >instead, be something that will run at the machine of the one submitting
> >the patch. For instance, this could be a perl or python script inside
> >Kernel's ./script directory that would handle everything locally, and
> >then submit the patch via patchwork's REST API.  
> 
> I think I didn't make clear that this isn't supposed to go straight to 
> patchwork. Patchwork is merely a convenient tool where this happens -- 
> the resulting patch gets mailed out to the mailing list just as the user 
> would have done.

IMHO, doing things like "git clone" for web-based patch submission seems
a very bad idea.

Ok, we might have something that would locally run "git format-email",
and pick the formatted patches, but that's exactly the type of
cross-site scripting that most ad-blocks prevent.

So, IMHO, the best is to let the user to run a local command to
generate/validate the patchset, using his own resources to do the
task. Once everything is ok, then it could use a web-based app to
upload the patches from his local disk.

Or, just run a command-line program that would do that.

Thanks,
Mauro

Re: RFE: use patchwork to submit a patch

[David Miller]

From: Steven Rostedt <rostedt@goodmis.org>
Date: Fri, 11 Oct 2019 14:01:08 -0400

> Thus, if we want people to send us their fixes, we better keep just
> an email with a patch the lowest bar for entry.

I argue that for people coming into the software engineering world
today, a PR is the lowest bar for entry.  And email is the exact
opposite, _especially_ our way of doing email.

Because it IS NOT just an email with a patch.

It is an email with a specific Subject line format, with various
fields.  Some field are optional and some are mandatory, and it also
depends upon which tree you are targetting.

Then there is the commit message which has content and formatting
requirements.

And then there are the tags, of various types and flavors, and the
context (which is sometimes subtle) determines which of those tags are
relevant and how they should be filled in.

The various other standard email fields like TO: and CC: have to
be set a certain way otherwise the patch won't even be looked at.

If you think all of this stuff is a low barrier to entry you are
frankly kidding yourself.

It is ONLY a low barrier to entry for an experienced Linux kernel
developer.  Full stop.

You are not helping casual contributors with this "simple" email based
submission method.  It is understood and easy us, but nobody else.

Re: RFE: use patchwork to submit a patch

[Steven Rostedt]

On Fri, 11 Oct 2019 11:32:54 -0700 (PDT)
David Miller <davem@davemloft.net> wrote:

> From: Steven Rostedt <rostedt@goodmis.org>
> Date: Fri, 11 Oct 2019 14:01:08 -0400
> 
> > Thus, if we want people to send us their fixes, we better keep just
> > an email with a patch the lowest bar for entry.  
> 
> I argue that for people coming into the software engineering world
> today, a PR is the lowest bar for entry.  And email is the exact
> opposite, _especially_ our way of doing email.
> 
> Because it IS NOT just an email with a patch.

That is if the maintainer wants to be anal about the submission. I've
taken patches where I only asked the person to give me a signed off by.
I also ask:

"Hi, do you intend on being a contributor, or do you only want to get
this fix upstream? If the latter, I will handle the change log and
other formatting for you, all I need is the signed-off-by. Otherwise, I
will help you submit a proper patch."

Again, this is for one offs, where someone found something like an off
by one error or other trivial bug to fix. I just want the fix in, but
will let the submitter decide how strict I will be to get it in. I've
had people say "I don't care, just get it fixed", and I do the
formatting and all the grudge work, but still give the submitter the
credit.

Not to mention, there's several times I get a patch where the solution
is totally wrong, and I need to make the fix anyway. A simple
Reported-by is what the submitter gets. Also, perfectly done by email.

> 
> You are not helping casual contributors with this "simple" email based
> submission method.  It is understood and easy us, but nobody else.

I'm not saying that email is the only way, this entire thread is about
getting another tool to help. But I will scream very loudly if we
eliminate email totally.

-- Steve

Re: RFE: use patchwork to submit a patch

[Mauro Carvalho Chehab]

Em Fri, 11 Oct 2019 14:44:01 -0400
Steven Rostedt <rostedt@goodmis.org> escreveu:

> On Fri, 11 Oct 2019 11:32:54 -0700 (PDT)
> David Miller <davem@davemloft.net> wrote:
> 
> > From: Steven Rostedt <rostedt@goodmis.org>
> > Date: Fri, 11 Oct 2019 14:01:08 -0400
> >   
> > > Thus, if we want people to send us their fixes, we better keep just
> > > an email with a patch the lowest bar for entry.    
> > 
> > I argue that for people coming into the software engineering world
> > today, a PR is the lowest bar for entry.  And email is the exact
> > opposite, _especially_ our way of doing email.
> > 
> > Because it IS NOT just an email with a patch.  
> 
> That is if the maintainer wants to be anal about the submission. I've
> taken patches where I only asked the person to give me a signed off by.
> I also ask:
> 
> "Hi, do you intend on being a contributor, or do you only want to get
> this fix upstream? If the latter, I will handle the change log and
> other formatting for you, all I need is the signed-off-by. Otherwise, I
> will help you submit a proper patch."
> 
> Again, this is for one offs, where someone found something like an off
> by one error or other trivial bug to fix. I just want the fix in, but
> will let the submitter decide how strict I will be to get it in. I've
> had people say "I don't care, just get it fixed", and I do the
> formatting and all the grudge work, but still give the submitter the
> credit.
> 
> Not to mention, there's several times I get a patch where the solution
> is totally wrong, and I need to make the fix anyway. A simple
> Reported-by is what the submitter gets. Also, perfectly done by email.
> 

About the same here: if needed, I change patch descriptions, and
for trivial patches submitted by newbies, I even correct bad whitespacing,
if needed.

All it takes for a patch to be handled is to send it in a way that
patchwork will recognize as a patch - e. g. it should carry a properly
formatted diff.

> > 
> > You are not helping casual contributors with this "simple" email based
> > submission method.  It is understood and easy us, but nobody else.  
> 
> I'm not saying that email is the only way, this entire thread is about
> getting another tool to help. But I will scream very loudly if we
> eliminate email totally.

Yes, e-mail should stay.

Thanks,
Mauro

Re: RFE: use patchwork to submit a patch

[Mauro Carvalho Chehab]

Em Fri, 11 Oct 2019 11:32:54 -0700 (PDT)
David Miller <davem@davemloft.net> escreveu:

> From: Steven Rostedt <rostedt@goodmis.org>
> Date: Fri, 11 Oct 2019 14:01:08 -0400
> 
> > Thus, if we want people to send us their fixes, we better keep just
> > an email with a patch the lowest bar for entry.  
> 
> I argue that for people coming into the software engineering world
> today, a PR is the lowest bar for entry.  And email is the exact
> opposite, _especially_ our way of doing email.
> 
> Because it IS NOT just an email with a patch.
> 
> It is an email with a specific Subject line format, with various
> fields.  Some field are optional and some are mandatory, and it also
> depends upon which tree you are targetting.
> 
> Then there is the commit message which has content and formatting
> requirements.
> 
> And then there are the tags, of various types and flavors, and the
> context (which is sometimes subtle) determines which of those tags are
> relevant and how they should be filled in.
> 
> The various other standard email fields like TO: and CC: have to
> be set a certain way otherwise the patch won't even be looked at.

I can't see how just adding a web interface like github/gitlab
would solve any of the above. I mean, the submitter will still
need to write a proper subject/description before submitting the
patch to the web interface, add the tags, etc. 

The only thing that would be different if we write a a Kernel-specific
web GUI would be to automate some sanity checks on it, like starting a CI
build for the patch series, run checkpatch and get_maintainers, but that 
could also happen if we write a script/submit_patches tool, that would
internally call checkpatch, get_maintainers and git send e-mail
(or use some other interface like patchwork REST to submit patches).

Thanks,
Mauro

Re: RFE: use patchwork to submit a patch

[Drew DeVault]

I happen to have been working on something like this today:

https://sr.ht/x71F.png

This doesn't require any JavaScript.

I've also been working on the other side, but it's less shiny - I'm
teaching pygit2 about pluggable object db backends so that I can teach
my mailing lists to talk to my git repos, and do things like offer
expandable diffs, check mergability of patches, submit CI jobs, etc.

Some more screenshots from my work today:

https://sr.ht/3grl.png
https://sr.ht/rcsA.png
https://sr.ht/CLHm.png

Re: RFE: use patchwork to submit a patch

[David Miller]

From: Mauro Carvalho Chehab <mchehab@kernel.org>
Date: Fri, 11 Oct 2019 15:59:49 -0300

> I can't see how just adding a web interface like github/gitlab
> would solve any of the above. I mean, the submitter will still
> need to write a proper subject/description before submitting the
> patch to the web interface, add the tags, etc. 

You can actually codify the various fields into a web form with
help texts etc.

Things that are mandatory can even be marked required and people
can't even submit a PR without Signed-off-by: for example.

Re: RFE: use patchwork to submit a patch

[Konstantin Ryabitsev]

On Fri, Oct 11, 2019 at 10:57:02AM +0200, Greg KH wrote:
>So other than that minor thing, sounds interesting.  It's hard to
>determine just how difficult the whole "set up git and send a patch out"
>process is for people these days given the _huge_ numbers of new
>contributions we keep getting, and the numerous good tutorials we have
>created that spell out exactly how to do this.
>
>So you might be "solving" a problem that we don't really have.  It's
>hard to tell :(

It is interesting that there are split views on this. The main reason 
why I was thinking about it was because the topic came up a few times 
already. For example, in a conversation last year on ksummit-discuss:

https://lore.kernel.org/ksummit-discuss/ECADFF3FD767C149AD96A924E7EA6EAF7C1EAA24@USCULXMSG01.am.sony.com/

Tim Bird mentioned that Sony developers couldn't send/receive patches 
because their corporate mail server rewrote all links to go through some 
kind of security appliance verification. If you read that thread, what 
we are discussing now is what I suggested we did then -- a web tool that 
could take corporate SMTP servers out of the equation.

(This is the same reason I generally disagree with Eric Wong about 
preserving SMTP as the primary transmission protocol -- I've heard lots 
of complaints both from kernel developers and especially from people 
trying to contribute to CAF about corporate policies actually making it 
impossible to submit patches -- and no, using a different mail server is 
not a possibility for them because it can be a firing offense under 
their IT AUP rules.)

>> I know this is a pretty big RFE, and I would like to hear your thoughts
>> about this. If there is general agreement that this is doable/good idea, I
>> may be able to come up with funding for this development as part of the
>> overall tooling improvement proposal.
>
>The workflow seems sane, and matches what most people do today, with the
>exception that it "solves" the git send-email issue, right?  Is that our
>biggest barrier?

Well, I can't really speak from my extensive experience as a kernel 
developer, but I *have* submitted patches to Documentation/* before.  
This happens infrequently enough that I basically have to relearn the 
whole process from scratch, and it *is* a lot of steps. I can't fault 
people who are only familiar with the GitHub way of doing things when
they complain that this process is a challenge for them.

Not everyone submitting changes to the kernel are going to be highly 
skilled and comfortable with the terminal and command-line tools. They 
may be submitting a documentation fix, or it can be a driver developer 
who never leaves Visual Studio submitting a small bugfix so their driver 
works better in Linux.

>I would recommend interviewing some of the recent kernel mentor project
>and outreachy applicants first, to try to determine exactly what their
>problems, if any, were with our development process.  If they say that
>this type of tool/workflow would have saved them hours of time and
>energy, then that's a great indication that we should try to do this.

I don't disagree and Shuah's comments are very valuable here. However, I 
would argue that these folks don't necessarily represent the target 
audience for this tool. They may be newbies, but they join these 
initiatives with the goal of spending significant time with the kernel 
and its code, so they don't mind the effort of learning the proper way 
of submitting patches.

I'm thinking of someone who needs to submit an occasional contribution 
once every six months and to whom this document is both long and 
daunting: 
https://www.kernel.org/doc/html/v4.17/process/submitting-patches.html.

-K

Re: RFE: use patchwork to submit a patch

[Stephen Hemminger]

On Fri, 11 Oct 2019 12:11:53 -0700 (PDT)
David Miller <davem@davemloft.net> wrote:

> From: Mauro Carvalho Chehab <mchehab@kernel.org>
> Date: Fri, 11 Oct 2019 15:59:49 -0300
> 
> > I can't see how just adding a web interface like github/gitlab
> > would solve any of the above. I mean, the submitter will still
> > need to write a proper subject/description before submitting the
> > patch to the web interface, add the tags, etc.   
> 
> You can actually codify the various fields into a web form with
> help texts etc.
> 
> Things that are mandatory can even be marked required and people
> can't even submit a PR without Signed-off-by: for example.

As someone who has to deal with a corporate busted mail server.
I agree mailing patches is a barrier to entry for many people today.

Re: RFE: use patchwork to submit a patch

[Eric Wong]

Konstantin Ryabitsev <konstantin@linuxfoundation.org> wrote:
> On Fri, Oct 11, 2019 at 10:57:02AM +0200, Greg KH wrote:
> > So other than that minor thing, sounds interesting.  It's hard to
> > determine just how difficult the whole "set up git and send a patch out"
> > process is for people these days given the _huge_ numbers of new
> > contributions we keep getting, and the numerous good tutorials we have
> > created that spell out exactly how to do this.
> > 
> > So you might be "solving" a problem that we don't really have.  It's
> > hard to tell :(
> 
> It is interesting that there are split views on this. The main reason why I
> was thinking about it was because the topic came up a few times already. For
> example, in a conversation last year on ksummit-discuss:
> 
> https://lore.kernel.org/ksummit-discuss/ECADFF3FD767C149AD96A924E7EA6EAF7C1EAA24@USCULXMSG01.am.sony.com/
> 
> Tim Bird mentioned that Sony developers couldn't send/receive patches
> because their corporate mail server rewrote all links to go through some
> kind of security appliance verification. If you read that thread, what we
> are discussing now is what I suggested we did then -- a web tool that could
> take corporate SMTP servers out of the equation.
> 
> (This is the same reason I generally disagree with Eric Wong about
> preserving SMTP as the primary transmission protocol -- I've heard lots of
> complaints both from kernel developers and especially from people trying to
> contribute to CAF about corporate policies actually making it impossible to
> submit patches -- and no, using a different mail server is not a possibility
> for them because it can be a firing offense under their IT AUP rules.)

I'm not opposed to a webmail interface tailored to kernel hacking
which does stuff like checkpatch.pl and get_maintainer.pl before
sending (similar to your patchwork proposal and
gitgadgetgadget).  That would get around security appliances
but SMTP would still be used in the background.

Or offer full-blown HTTPS webmail + IMAP + SMTP access like any
other webmail provider + checkpatch + get_maintainer helpers.

Re: RFE: use patchwork to submit a patch

[Konstantin Ryabitsev]

On Fri, Oct 11, 2019 at 09:23:08PM +0000, Eric Wong wrote:
>> (This is the same reason I generally disagree with Eric Wong about
>> preserving SMTP as the primary transmission protocol -- I've heard lots of
>> complaints both from kernel developers and especially from people trying to
>> contribute to CAF about corporate policies actually making it impossible to
>> submit patches -- and no, using a different mail server is not a possibility
>> for them because it can be a firing offense under their IT AUP rules.)
>
>I'm not opposed to a webmail interface tailored to kernel hacking
>which does stuff like checkpatch.pl and get_maintainer.pl before
>sending (similar to your patchwork proposal and
>gitgadgetgadget).  That would get around security appliances
>but SMTP would still be used in the background.
>
>Or offer full-blown HTTPS webmail + IMAP + SMTP access like any
>other webmail provider + checkpatch + get_maintainer helpers.

Well, this is the bit where I say that it may not be allowed by 
corporate rules. I see this all the time in CAF/Android world where 
companies *require* that all email goes through their SMTP server so 
that it can be properly logged (often for legal reasons). And it is 
often equally required that any code submissions come from 
person@corporate.com and not person@free-email-provider.com for 
License/CLA reasons, so setting up a webmail server is not a solution 
either.

This is basically why SMTP sucks in my view -- and it's worthless trying 
to pick fights with IT departments, because they are told to do so by 
lawyers. So, I want to take SMTP out of the equation:

1. provide a way for someone to submit a patch using a web interface 
   (but still in a way that From: is their corporate ID)
2. use individual git feeds as a way to send out patches instead of 
   always being secondary to SMTP

-K

Re: RFE: use patchwork to submit a patch

[Steven Rostedt]

On Fri, 11 Oct 2019 14:19:09 -0700
Stephen Hemminger <stephen@networkplumber.org> wrote:

> As someone who has to deal with a corporate busted mail server.
> I agree mailing patches is a barrier to entry for many people today.

Hmm, and I wonder what company fosters so many issues with doing that
today? :-/

/me notices that Stephen uses his personal email account ;-)

-- Steve

Re: RFE: use patchwork to submit a patch

[Dave Airlie]

On Sat, 12 Oct 2019 at 08:34, Steven Rostedt <rostedt@goodmis.org> wrote:
>
> On Fri, 11 Oct 2019 14:19:09 -0700
> Stephen Hemminger <stephen@networkplumber.org> wrote:
>
> > As someone who has to deal with a corporate busted mail server.
> > I agree mailing patches is a barrier to entry for many people today.
>
> Hmm, and I wonder what company fosters so many issues with doing that
> today? :-/
>
> /me notices that Stephen uses his personal email account ;-)

Your current employer also defaults to a crappy link replacer on most emails :-)

Dave.

Re: RFE: use patchwork to submit a patch

[Dave Airlie]

> I don't disagree and Shuah's comments are very valuable here. However, I
> would argue that these folks don't necessarily represent the target
> audience for this tool. They may be newbies, but they join these
> initiatives with the goal of spending significant time with the kernel
> and its code, so they don't mind the effort of learning the proper way
> of submitting patches.
>

It's two different group,

a) people who aren't currently employed to work on Linux and want to
be will jump through a lot more hoops with a mentor, they also are
unlikely to been working at a company with horrible emails.
b) people who are employed to work on Linux or at least Linux
ecosystem tasks and want to send one or two patches upstream they
found while developing their products, but can't use personal email
and corp email sucks, and who use github for most of their lives
elsewhere.

I'm guessing we've got a lot of (b) we don't really see represented in
mentorship programmes.

Dave.

Re: RFE: use patchwork to submit a patch

[Steven Rostedt]

On Sat, 12 Oct 2019 08:54:22 +1000
Dave Airlie <airlied@gmail.com> wrote:

> On Sat, 12 Oct 2019 at 08:34, Steven Rostedt <rostedt@goodmis.org> wrote:
> >
> > On Fri, 11 Oct 2019 14:19:09 -0700
> > Stephen Hemminger <stephen@networkplumber.org> wrote:
> >  
> > > As someone who has to deal with a corporate busted mail server.
> > > I agree mailing patches is a barrier to entry for many people today.  
> >
> > Hmm, and I wonder what company fosters so many issues with doing that
> > today? :-/
> >
> > /me notices that Stephen uses his personal email account ;-)  
> 
> Your current employer also defaults to a crappy link replacer on most emails :-)

Yes, because they use Stephen's employer's crappy email servers.

-- Steve

Re: RFE: use patchwork to submit a patch

[Stephen Hemminger]

On Fri, 11 Oct 2019 19:00:09 -0400
Steven Rostedt <rostedt@goodmis.org> wrote:

> > > > As someone who has to deal with a corporate busted mail server.
> > > > I agree mailing patches is a barrier to entry for many people today.    
> > >
> > > Hmm, and I wonder what company fosters so many issues with doing that
> > > today? :-/
> > >
> > > /me notices that Stephen uses his personal email account ;-)    
> > 
> > Your current employer also defaults to a crappy link replacer on most emails :-)  
> 
> Yes, because they use Stephen's employer's crappy email servers.

Yes, my life is too short to fight with corporate stuff.
There is a security first mindset which is good but gets in the way.
The current generation of Internet standard mail protocols does not have enough security to be allowed.

It does bring up that any new workflow has to have security protocol
and threat model as part of its design.

Re: RFE: use patchwork to submit a patch

[Steven Rostedt]

On Fri, 11 Oct 2019 17:08:39 -0700
Stephen Hemminger <stephen@networkplumber.org> wrote:

> Yes, my life is too short to fight with corporate stuff.

Heh, me too.

> There is a security first mindset which is good but gets in the way.
> The current generation of Internet standard mail protocols does not have enough security to be allowed.

Wish we could come up with a "secured email" RFC.

> 
> It does bring up that any new workflow has to have security protocol
> and threat model as part of its design.

True.

-- Steve

Re: RFE: use patchwork to submit a patch

[Greg KH]

On Fri, Oct 11, 2019 at 05:35:53PM -0400, Konstantin Ryabitsev wrote:
> On Fri, Oct 11, 2019 at 09:23:08PM +0000, Eric Wong wrote:
> > > (This is the same reason I generally disagree with Eric Wong about
> > > preserving SMTP as the primary transmission protocol -- I've heard lots of
> > > complaints both from kernel developers and especially from people trying to
> > > contribute to CAF about corporate policies actually making it impossible to
> > > submit patches -- and no, using a different mail server is not a possibility
> > > for them because it can be a firing offense under their IT AUP rules.)
> > 
> > I'm not opposed to a webmail interface tailored to kernel hacking
> > which does stuff like checkpatch.pl and get_maintainer.pl before
> > sending (similar to your patchwork proposal and
> > gitgadgetgadget).  That would get around security appliances
> > but SMTP would still be used in the background.
> > 
> > Or offer full-blown HTTPS webmail + IMAP + SMTP access like any
> > other webmail provider + checkpatch + get_maintainer helpers.
> 
> Well, this is the bit where I say that it may not be allowed by corporate
> rules. I see this all the time in CAF/Android world where companies
> *require* that all email goes through their SMTP server so that it can be
> properly logged (often for legal reasons). And it is often equally required
> that any code submissions come from person@corporate.com and not
> person@free-email-provider.com for License/CLA reasons, so setting up a
> webmail server is not a solution either.
> 
> This is basically why SMTP sucks in my view -- and it's worthless trying to
> pick fights with IT departments, because they are told to do so by lawyers.
> So, I want to take SMTP out of the equation:
> 
> 1. provide a way for someone to submit a patch using a web interface   (but
> still in a way that From: is their corporate ID)

If you do this, what happens when a maintainer/reviewer responds to that
patch and says "looks good, but can you change X and resend it?"

How will they get that message if it didn't go through their email
system?  How will they be able to respond to it?

> 2. use individual git feeds as a way to send out patches instead of   always
> being secondary to SMTP

Sending patches that way is one thing, the interaction based on those
patches is another.

Everyone needs to remember that only 1/3 of the patches submitted are
applied.  The "normal" path of development is at least a review/resend
cycle for submissions (2/3 of patches).  So that 2/3 can't be ignored as
the "new/drive-by submissions" are probably more in that category than
not.

thanks,

greg k-h

Re: RFE: use patchwork to submit a patch

[Greg KH]

On Fri, Oct 11, 2019 at 04:02:28PM -0400, Konstantin Ryabitsev wrote:
> On Fri, Oct 11, 2019 at 10:57:02AM +0200, Greg KH wrote:
> > So other than that minor thing, sounds interesting.  It's hard to
> > determine just how difficult the whole "set up git and send a patch out"
> > process is for people these days given the _huge_ numbers of new
> > contributions we keep getting, and the numerous good tutorials we have
> > created that spell out exactly how to do this.
> > 
> > So you might be "solving" a problem that we don't really have.  It's
> > hard to tell :(
> 
> It is interesting that there are split views on this. The main reason why I
> was thinking about it was because the topic came up a few times already. For
> example, in a conversation last year on ksummit-discuss:
> 
> https://lore.kernel.org/ksummit-discuss/ECADFF3FD767C149AD96A924E7EA6EAF7C1EAA24@USCULXMSG01.am.sony.com/
> 
> Tim Bird mentioned that Sony developers couldn't send/receive patches
> because their corporate mail server rewrote all links to go through some
> kind of security appliance verification. If you read that thread, what we
> are discussing now is what I suggested we did then -- a web tool that could
> take corporate SMTP servers out of the equation.

Yes, compamy SMTP servers are a huge problem, which is why we have so
many contributors from gmail accounts :)

But will your "send a patch" tool work for someone behind a corporate
firewall?  If we are needing/wanting corporate contributions that bad,
by providing an end-run around their infrastructure is not going to be
seen as a "good" thing, once those corporate security teams realize what
is going on...

And again, how will the feedback loop look like here?  If joe@sony sends
me a patch, how can they respond to my response of "please fix the
change up to do X instead"?

I don't want this to be a "fire and forget" type of system where
companies submit patches through this interface and never respond.  We
already have enough examples of companies that do that today.  My last
example was a company that send out a 4 patch series, the response was
"great work, but can you resend just the last 2 so we can take them now,
and do X and Y on patches 1 and 2 so we can take them?"  That was a year
ago, with no response, when internally the developer said "I sent it
upstream, but it was rejected!".

</rant>

> (This is the same reason I generally disagree with Eric Wong about
> preserving SMTP as the primary transmission protocol -- I've heard lots of
> complaints both from kernel developers and especially from people trying to
> contribute to CAF about corporate policies actually making it impossible to
> submit patches -- and no, using a different mail server is not a possibility
> for them because it can be a firing offense under their IT AUP rules.)

So if we create yet-another-patch-submission-path, how will that
circumvent those rules?  CAF is crazy, I've been in the middle of that
many times.  I doubt the "well I didn't send it in an email, I used this
tool over here instead." will play well with their lawyers/managers to
keep them from being fired.

It's not our job to solve the "my managers are not letting me send
patches out" problem here.  To solve that requires working with the
company at a totally different level.

> > > I know this is a pretty big RFE, and I would like to hear your thoughts
> > > about this. If there is general agreement that this is doable/good idea, I
> > > may be able to come up with funding for this development as part of the
> > > overall tooling improvement proposal.
> > 
> > The workflow seems sane, and matches what most people do today, with the
> > exception that it "solves" the git send-email issue, right?  Is that our
> > biggest barrier?
> 
> Well, I can't really speak from my extensive experience as a kernel
> developer, but I *have* submitted patches to Documentation/* before.  This
> happens infrequently enough that I basically have to relearn the whole
> process from scratch, and it *is* a lot of steps. I can't fault people who
> are only familiar with the GitHub way of doing things when
> they complain that this process is a challenge for them.
> 
> Not everyone submitting changes to the kernel are going to be highly skilled
> and comfortable with the terminal and command-line tools. They may be
> submitting a documentation fix, or it can be a driver developer who never
> leaves Visual Studio submitting a small bugfix so their driver works better
> in Linux.

Simple drive-by patches for this is fine, if this type of tool will
help, as long as the feedback loop is there.

> > I would recommend interviewing some of the recent kernel mentor project
> > and outreachy applicants first, to try to determine exactly what their
> > problems, if any, were with our development process.  If they say that
> > this type of tool/workflow would have saved them hours of time and
> > energy, then that's a great indication that we should try to do this.
> 
> I don't disagree and Shuah's comments are very valuable here. However, I
> would argue that these folks don't necessarily represent the target audience
> for this tool. They may be newbies, but they join these initiatives with the
> goal of spending significant time with the kernel and its code, so they
> don't mind the effort of learning the proper way of submitting patches.

I still recommend at least talking to these developers as we _know_ they
are new to our process, so their feedback for a tool to make it easy for
people like themselves (or themselves 2 months ago), will be valuable.

thanks,

greg k-h

Re: RFE: use patchwork to submit a patch

[Stephen Finucane]

On Thu, 2019-10-10 at 10:41 -0400, Konstantin Ryabitsev wrote:
> Hi, all:
> 
> I would like to propose a new (large) feature to patchwork with the goal 
> to make the process of submitting a patch easier for newbies and people 
> generally less familiar with patch-based development. This was discussed 
> previously on the workflows list: 
> https://lore.kernel.org/workflows/20190930202451.GA14403@pure.paranoia.local/

I'll echo Daniel's sentiments that I like the idea of a git-to-email
bridge, but that I'm not sure if that should belong in Patchwork core.
I too am open to being convinced but before we get to anything like a
solution though, I'd like to identify the problem(s) you're trying to
solve. From reading this thread, it seems there are three separate
issues issues intertwined here.

 * Corporate email is often broken, meaning people have to jump through
   hoops to simply submit a patch, if it's even possible. [1]
 * Encoding metadata in emails has to be done in an ad-hoc, freeform
   fashion, through special headers, tags in the subject or specific
   annotations in the body. This requires reading large contributors
   guides before sending anything.
 * Using CLI tools in general is hard for newbies (?)

Asssuming those are correct, I'd like to challenge some of them :)
There isn't a whole lot that can be done about broken email, at least
until JMAP becomes a thing (it's done over HTTP so that could help. Or
not. idk), but I'm not sure about the other two. I don't know what kind
of metadata would be needed to submit a patch to a random subtree of
the kernel, but I assume the metadata exists for a reason? If so, is
this actually something we can tackle via a UI or CLI without producing
custom workflow tools for every single project and if not, can we
actually solve this particular issue? Personal, I would consider
reading the contributor guide a minimum barrier to entry, as without
this you're simply transferring work from contributors to (often
overworked) maintainers but I do acknowledge that it's easy for me to
say that since I know how to do this stuff already. The same idea
applies to the idea of not using CLIs. Is there an statistically
significant amount of people that would be able to submit useful
changes but can't use a CLI tool? I know you can get away without using
a terminal for traditional Windows development or web development, but
surely terminal knowledge is a prerequisite for almost everything else?

What I'm trying to get at here is figure out if (a) this is something
that can really benefit from living in Patchwork, (b) this is something
that needs a UI, (c) this is something's that necessary full stop (vs.
just waiting for projects to switch to GitLab or Gerrit or whatever new
cool ends up being). If we can figure that out, we can look into how we
can go about implementing stuff.

Stephen

[1] I've multiple personal examples of this, from having to ask IT
during my Intel days to remove automatic legal signatures to
outlook.com refusing to respect message-id's, resulting in broken
threading at a minimum. Thankfully I don't have to jump through those
hoops at Red Hat but yeah, eew.

Re: RFE: use patchwork to submit a patch

[Stephen Finucane]

On Sat, 2019-10-12 at 14:16 +0100, Stephen Finucane wrote:
> On Thu, 2019-10-10 at 10:41 -0400, Konstantin Ryabitsev wrote:
> > Hi, all:
> > 
> > I would like to propose a new (large) feature to patchwork with the goal 
> > to make the process of submitting a patch easier for newbies and people 
> > generally less familiar with patch-based development. This was discussed 
> > previously on the workflows list: 
> > https://lore.kernel.org/workflows/20190930202451.GA14403@pure.paranoia.local/

[top posting, sort of]

After writing this, I reopened my email client and noticed a lot of
replies that hadn't been there before. Stupid client/email provider
(I'm not sure who's to blame). It _seems_ like I'm not the only one
questioning the need for a point and click patch submission format, but
that doesn't mean there isn't some work we can do here. Previously,
Patchwork had the stated goal of aquiring the ability to reply to
patches via the web UI a lá Google Groups. If email is really an issue,
what about if we added that functionality along with the ability to
POST patches via the REST API? That would mean people would be able to
submit patches via e.g. 'git pw series submit' and then reply on the
web UI. I haven't really thought it through fully, but this _should_ do
most of what you're looking for, right?

Stephen

> I'll echo Daniel's sentiments that I like the idea of a git-to-email
> bridge, but that I'm not sure if that should belong in Patchwork core.
> I too am open to being convinced but before we get to anything like a
> solution though, I'd like to identify the problem(s) you're trying to
> solve. From reading this thread, it seems there are three separate
> issues issues intertwined here.
> 
>  * Corporate email is often broken, meaning people have to jump through
>    hoops to simply submit a patch, if it's even possible. [1]
>  * Encoding metadata in emails has to be done in an ad-hoc, freeform
>    fashion, through special headers, tags in the subject or specific
>    annotations in the body. This requires reading large contributors
>    guides before sending anything.
>  * Using CLI tools in general is hard for newbies (?)
> 
> Asssuming those are correct, I'd like to challenge some of them :)
> There isn't a whole lot that can be done about broken email, at least
> until JMAP becomes a thing (it's done over HTTP so that could help. Or
> not. idk), but I'm not sure about the other two. I don't know what kind
> of metadata would be needed to submit a patch to a random subtree of
> the kernel, but I assume the metadata exists for a reason? If so, is
> this actually something we can tackle via a UI or CLI without producing
> custom workflow tools for every single project and if not, can we
> actually solve this particular issue? Personal, I would consider
> reading the contributor guide a minimum barrier to entry, as without
> this you're simply transferring work from contributors to (often
> overworked) maintainers but I do acknowledge that it's easy for me to
> say that since I know how to do this stuff already. The same idea
> applies to the idea of not using CLIs. Is there an statistically
> significant amount of people that would be able to submit useful
> changes but can't use a CLI tool? I know you can get away without using
> a terminal for traditional Windows development or web development, but
> surely terminal knowledge is a prerequisite for almost everything else?
> 
> What I'm trying to get at here is figure out if (a) this is something
> that can really benefit from living in Patchwork, (b) this is something
> that needs a UI, (c) this is something's that necessary full stop (vs.
> just waiting for projects to switch to GitLab or Gerrit or whatever new
> cool ends up being). If we can figure that out, we can look into how we
> can go about implementing stuff.
> 
> Stephen
> 
> [1] I've multiple personal examples of this, from having to ask IT
> during my Intel days to remove automatic legal signatures to
> outlook.com refusing to respect message-id's, resulting in broken
> threading at a minimum. Thankfully I don't have to jump through those
> hoops at Red Hat but yeah, eew.
> 
> _______________________________________________
> Patchwork mailing list
> Patchwork@lists.ozlabs.org
> https://lists.ozlabs.org/listinfo/patchwork

Re: RFE: use patchwork to submit a patch

[Daniel Axtens]

> It does bring up that any new workflow has to have security protocol
> and threat model as part of its design.

This is actually something that worries me about the patchwork
workflow. Maintainers seem to trust the patchwork version of a patch
without much (or any) verification that it matches what was sent to the
list.

Say Alice emails a patch that says something like:

 if ((permissions == allowed) && other_stuff) {
    do_things();
 }
 do_more_stuff(permissions);

List members read that email, and many review it in the email client.

However, say that the version in Patchwork actually reads like this:

 if ((permission = allowed) && other_stuff) {
    do_things();
 }
 do_more_stuff(permissions);

(You could get this with a malicious/compromised patchwork admin, or a
sufficiently advanced network adversary - patchwork takes the first mail
for a given message-id + project and later ones are discarded, so
there's a race you can win.)

If the maintainer or someone else happens to apply the patch from
patchwork and then review it, or if tests happen to catch the relevant
case, we'll catch this. Otherwise...

It's not the easiest or lowest risk way to get malicious code into the
kernel, but nonetheless, I worry about it.

I can't think of a sensible way to fix this, unless we want to move to a
world where patch submissions are GPG signed, and teach patchwork to
verify sigs.

Regards,
Daniel

Re: RFE: use patchwork to submit a patch

[Eric Wong]

Konstantin Ryabitsev <konstantin@linuxfoundation.org> wrote:
> On Fri, Oct 11, 2019 at 09:23:08PM +0000, Eric Wong wrote:
> > > (This is the same reason I generally disagree with Eric Wong about
> > > preserving SMTP as the primary transmission protocol -- I've heard lots of
> > > complaints both from kernel developers and especially from people trying to
> > > contribute to CAF about corporate policies actually making it impossible to
> > > submit patches -- and no, using a different mail server is not a possibility
> > > for them because it can be a firing offense under their IT AUP rules.)
> > 
> > I'm not opposed to a webmail interface tailored to kernel hacking
> > which does stuff like checkpatch.pl and get_maintainer.pl before
> > sending (similar to your patchwork proposal and
> > gitgadgetgadget).  That would get around security appliances
> > but SMTP would still be used in the background.
> > 
> > Or offer full-blown HTTPS webmail + IMAP + SMTP access like any
> > other webmail provider + checkpatch + get_maintainer helpers.
> 
> Well, this is the bit where I say that it may not be allowed by corporate
> rules. I see this all the time in CAF/Android world where companies
> *require* that all email goes through their SMTP server so that it can be
> properly logged (often for legal reasons). And it is often equally required
> that any code submissions come from person@corporate.com and not
> person@free-email-provider.com for License/CLA reasons, so setting up a
> webmail server is not a solution either.

Aren't they still allowed to submit stuff via forges the same way
they'd use a potential hacker-oriented webmail/SMTP/IMAP solution?

Sometimes I see @username_COMPANY-type names on forges, but
AFAIK it's not very common.

> This is basically why SMTP sucks in my view -- and it's worthless trying to
> pick fights with IT departments, because they are told to do so by lawyers.
> So, I want to take SMTP out of the equation:

If the open source community can fight to get GPL accepted, I
don't see why we can't fight or subvert dumb corporate policies.

> 1. provide a way for someone to submit a patch using a web interface   (but
> still in a way that From: is their corporate ID)
> 2. use individual git feeds as a way to send out patches instead of   always
> being secondary to SMTP

username-COMPANY@users.kernel.org could probably work if they're
required to use @username_COMPANY on forges.

We can also find creative ways to subvert corporate policies:
For example; if their policy specifically prevents outgoing SMTP,
"git imap-send" could be used.

If their policy forbids using external "email" services, we'd
name it "Kernel Hackers' Messaging System" or something of that
sort and say we use an email bridge :>

Re: RFE: use patchwork to submit a patch

[Geert Uytterhoeven]

Hi Eric,

On Mon, Oct 14, 2019 at 2:12 AM Eric Wong <e@80x24.org> wrote:
> Konstantin Ryabitsev <konstantin@linuxfoundation.org> wrote:
> > On Fri, Oct 11, 2019 at 09:23:08PM +0000, Eric Wong wrote:
> > > > (This is the same reason I generally disagree with Eric Wong about
> > > > preserving SMTP as the primary transmission protocol -- I've heard lots of
> > > > complaints both from kernel developers and especially from people trying to
> > > > contribute to CAF about corporate policies actually making it impossible to
> > > > submit patches -- and no, using a different mail server is not a possibility
> > > > for them because it can be a firing offense under their IT AUP rules.)
> > >
> > > I'm not opposed to a webmail interface tailored to kernel hacking
> > > which does stuff like checkpatch.pl and get_maintainer.pl before
> > > sending (similar to your patchwork proposal and
> > > gitgadgetgadget).  That would get around security appliances
> > > but SMTP would still be used in the background.
> > >
> > > Or offer full-blown HTTPS webmail + IMAP + SMTP access like any
> > > other webmail provider + checkpatch + get_maintainer helpers.
> >
> > Well, this is the bit where I say that it may not be allowed by corporate
> > rules. I see this all the time in CAF/Android world where companies
> > *require* that all email goes through their SMTP server so that it can be
> > properly logged (often for legal reasons). And it is often equally required
> > that any code submissions come from person@corporate.com and not
> > person@free-email-provider.com for License/CLA reasons, so setting up a
> > webmail server is not a solution either.
>
> Aren't they still allowed to submit stuff via forges the same way
> they'd use a potential hacker-oriented webmail/SMTP/IMAP solution?
>
> Sometimes I see @username_COMPANY-type names on forges, but
> AFAIK it's not very common.
>
> > This is basically why SMTP sucks in my view -- and it's worthless trying to
> > pick fights with IT departments, because they are told to do so by lawyers.
> > So, I want to take SMTP out of the equation:
>
> If the open source community can fight to get GPL accepted, I
> don't see why we can't fight or subvert dumb corporate policies.
>
> > 1. provide a way for someone to submit a patch using a web interface   (but
> > still in a way that From: is their corporate ID)
> > 2. use individual git feeds as a way to send out patches instead of   always
> > being secondary to SMTP
>
> username-COMPANY@users.kernel.org could probably work if they're
> required to use @username_COMPANY on forges.

username+foo is the standard way.

> We can also find creative ways to subvert corporate policies:
> For example; if their policy specifically prevents outgoing SMTP,
> "git imap-send" could be used.

IMAP may be blocked, too?

Bascially the only thing you can rely on is HTTP(S), through a proxy,
possibly with HTTPS inspection through a company-specific trusted
certificate that allows MITM.

> If their policy forbids using external "email" services, we'd
> name it "Kernel Hackers' Messaging System" or something of that
> sort and say we use an email bridge :>

Anything named "Hacker" may be blocked, too ;-)

Gr{oetje,eeting}s,

                        Geert

-- 
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org

In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
                                -- Linus Torvalds

Re: RFE: use patchwork to submit a patch

[Toke Høiland-Jørgensen]

Daniel Axtens <dja@axtens.net> writes:

>> It does bring up that any new workflow has to have security protocol
>> and threat model as part of its design.
>
> This is actually something that worries me about the patchwork
> workflow. Maintainers seem to trust the patchwork version of a patch
> without much (or any) verification that it matches what was sent to the
> list.
>
> Say Alice emails a patch that says something like:
>
>  if ((permissions == allowed) && other_stuff) {
>     do_things();
>  }
>  do_more_stuff(permissions);
>
> List members read that email, and many review it in the email client.
>
> However, say that the version in Patchwork actually reads like this:
>
>  if ((permission = allowed) && other_stuff) {
>     do_things();
>  }
>  do_more_stuff(permissions);
>
> (You could get this with a malicious/compromised patchwork admin, or a
> sufficiently advanced network adversary - patchwork takes the first mail
> for a given message-id + project and later ones are discarded, so
> there's a race you can win.)
>
> If the maintainer or someone else happens to apply the patch from
> patchwork and then review it, or if tests happen to catch the relevant
> case, we'll catch this. Otherwise...
>
> It's not the easiest or lowest risk way to get malicious code into the
> kernel, but nonetheless, I worry about it.
>
> I can't think of a sensible way to fix this, unless we want to move to a
> world where patch submissions are GPG signed, and teach patchwork to
> verify sigs.

It should be detectable, though, right?

Say you have two independently administered patchwork instances (or even
better, two different software packages entirely) that both subscribe to
the mailing lists, and compare patch content with each other. They
should at least be able to detect mismatches. Especially if you add a
sanity check before discarding duplicate message-ids.

This way you'd need to compromise multiple machines to achieve the kind
of compromise you're worried about. And you can add more independent
machines until you're satisfied that the risk is low enough :)

-Toke

Re: RFE: use patchwork to submit a patch

[Mark Brown]

[-- Attachment #1: Type: text/plain, Size: 952 bytes --]

On Sat, Oct 12, 2019 at 09:19:11AM +0200, Greg KH wrote:
> On Fri, Oct 11, 2019 at 05:35:53PM -0400, Konstantin Ryabitsev wrote:

> > 1. provide a way for someone to submit a patch using a web interface   (but
> > still in a way that From: is their corporate ID)

> If you do this, what happens when a maintainer/reviewer responds to that
> patch and says "looks good, but can you change X and resend it?"

> How will they get that message if it didn't go through their email
> system?  How will they be able to respond to it?

If they're using their work e-mail address like Konstantin says they'll
get the reply there.  Probably any response from them will be some top
posted thing with HTML but this happens often enough currently - people
often use git send-email or something to send but continue using a mail
setup that's not really what's expected for kernel work.  Usually it's
not the end of the world even if people get a bit grumpy at them.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: RFE: use patchwork to submit a patch

[Theodore Y. Ts'o]

On Mon, Oct 14, 2019 at 12:42:36PM +0200, Toke Høiland-Jørgensen wrote:
> It should be detectable, though, right?
> 
> Say you have two independently administered patchwork instances (or even
> better, two different software packages entirely) that both subscribe to
> the mailing lists, and compare patch content with each other. They
> should at least be able to detect mismatches. Especially if you add a
> sanity check before discarding duplicate message-ids.

They don't even need to compare against each other; patchwork is about
to add a feature where you can look up patches via message-id, right?
That means it's easy enough to write a program which fetches patches
from patchwork, and compares it to the patches found in
lore.kernel.org.  If they don't match, then an alarm can be sounded.

Individuals who are reviewing patches can also compare the copy in
their inbox with the copy from lore or some other public inbox.  And
maintainers can compare copies from lore.kernel.org and patchwork
before they apply a patch.  (99% of the time, I actually use the patch
from my inbox, anyway.)

> This way you'd need to compromise multiple machines to achieve the kind
> of compromise you're worried about. And you can add more independent
> machines until you're satisfied that the risk is low enough :)

Yep, exactly.  This is basically the theory behind Certificate
Transparency[1], applied to patches.  For example, here's the
certificate transparency report for kernel.org:

   https://transparencyreport.google.com/https/certificates?cert_search=domain:kernel.org

					- Ted

[1] http://www.certificate-transparency.org/what-is-ct

Re: RFE: use patchwork to submit a patch

[Daniel Axtens]

Toke Høiland-Jørgensen <toke@redhat.com> writes:

> Daniel Axtens <dja@axtens.net> writes:
>
>>> It does bring up that any new workflow has to have security protocol
>>> and threat model as part of its design.
>>
>> This is actually something that worries me about the patchwork
>> workflow. Maintainers seem to trust the patchwork version of a patch
>> without much (or any) verification that it matches what was sent to the
>> list.
>>
>> Say Alice emails a patch that says something like:
>>
>>  if ((permissions == allowed) && other_stuff) {
>>     do_things();
>>  }
>>  do_more_stuff(permissions);
>>
>> List members read that email, and many review it in the email client.
>>
>> However, say that the version in Patchwork actually reads like this:
>>
>>  if ((permission = allowed) && other_stuff) {
>>     do_things();
>>  }
>>  do_more_stuff(permissions);
>>
>> (You could get this with a malicious/compromised patchwork admin, or a
>> sufficiently advanced network adversary - patchwork takes the first mail
>> for a given message-id + project and later ones are discarded, so
>> there's a race you can win.)
>>
>> If the maintainer or someone else happens to apply the patch from
>> patchwork and then review it, or if tests happen to catch the relevant
>> case, we'll catch this. Otherwise...
>>
>> It's not the easiest or lowest risk way to get malicious code into the
>> kernel, but nonetheless, I worry about it.
>>
>> I can't think of a sensible way to fix this, unless we want to move to a
>> world where patch submissions are GPG signed, and teach patchwork to
>> verify sigs.
>
> It should be detectable, though, right?
>
> Say you have two independently administered patchwork instances (or even
> better, two different software packages entirely) that both subscribe to
> the mailing lists, and compare patch content with each other. They
> should at least be able to detect mismatches. Especially if you add a
> sanity check before discarding duplicate message-ids.
>
> This way you'd need to compromise multiple machines to achieve the kind
> of compromise you're worried about. And you can add more independent
> machines until you're satisfied that the risk is low enough :)

Yeah, it's detectable, even more simply if someone who subscribes to the
list checks patches they receive against a public patchwork. But my
concern isn't that it's undetectable, it's that it unlikely to be
detected in practice unless someone goes to the effort of writing code
to do it.

Regards,
Daniel

>
> -Toke

Re: RFE: use patchwork to submit a patch

[Toke Høiland-Jørgensen]

"Theodore Y. Ts'o" <tytso@mit.edu> writes:

> On Mon, Oct 14, 2019 at 12:42:36PM +0200, Toke Høiland-Jørgensen wrote:
>> It should be detectable, though, right?
>> 
>> Say you have two independently administered patchwork instances (or even
>> better, two different software packages entirely) that both subscribe to
>> the mailing lists, and compare patch content with each other. They
>> should at least be able to detect mismatches. Especially if you add a
>> sanity check before discarding duplicate message-ids.
>
> They don't even need to compare against each other; patchwork is about
> to add a feature where you can look up patches via message-id, right?
> That means it's easy enough to write a program which fetches patches
> from patchwork, and compares it to the patches found in
> lore.kernel.org.  If they don't match, then an alarm can be sounded.

Yeah. I guess what is needed is to go from "can be" to "will be" (as
Daniel pointed out in his simultaneous reply).

>> This way you'd need to compromise multiple machines to achieve the kind
>> of compromise you're worried about. And you can add more independent
>> machines until you're satisfied that the risk is low enough :)
>
> Yep, exactly.  This is basically the theory behind Certificate
> Transparency[1], applied to patches.

Indeed I'm familiar with certificate transparency, so this was certainly
not an idea conceived in a vacuum ;)

-Toke

Re: RFE: use patchwork to submit a patch

[Steven Rostedt]

On Mon, 14 Oct 2019 10:38:51 +1100
Daniel Axtens <dja@axtens.net> wrote:

> > It does bring up that any new workflow has to have security protocol
> > and threat model as part of its design.  
> 
> This is actually something that worries me about the patchwork
> workflow. Maintainers seem to trust the patchwork version of a patch
> without much (or any) verification that it matches what was sent to the
> list.

I tend to download the patch from patchwork, and review that. When I
find something to comment on, I go back to my INBOX and reply to the
original patch where I want to comment on.

Thus, what's in patchwork is what I have verified anyway.

-- Steve

Re: RFE: use patchwork to submit a patch

[Mauro Carvalho Chehab]

Em Mon, 14 Oct 2019 08:26:37 -0400
"Theodore Y. Ts'o" <tytso@mit.edu> escreveu:

> On Mon, Oct 14, 2019 at 12:42:36PM +0200, Toke Høiland-Jørgensen wrote:
> > It should be detectable, though, right?
> > 
> > Say you have two independently administered patchwork instances (or even
> > better, two different software packages entirely) that both subscribe to
> > the mailing lists, and compare patch content with each other. They
> > should at least be able to detect mismatches. Especially if you add a
> > sanity check before discarding duplicate message-ids.  
> 
> They don't even need to compare against each other; patchwork is about
> to add a feature where you can look up patches via message-id, right?
> That means it's easy enough to write a program which fetches patches
> from patchwork, and compares it to the patches found in
> lore.kernel.org.  If they don't match, then an alarm can be sounded.
> 
> Individuals who are reviewing patches can also compare the copy in
> their inbox with the copy from lore or some other public inbox.  And
> maintainers can compare copies from lore.kernel.org and patchwork
> before they apply a patch.  (99% of the time, I actually use the patch
> from my inbox, anyway.)

It can still have man-in-the-middle (MITM) attacks between the sender and
vger.kernel.org. Please notice that using https and adding the patch
via a web interface can also be subject to MITM, as companies and even some
Countries with strong policy enforcement may have some gateway on their
infra that will prevent end-to-end encryption[1], blocking direct
client-server https tunnels.

[1] They add an internal certificate to the browsers, so that the client
will see the connection as trustful, but the infra will actually do two
separate HTTPS encryption:

	client  ---> Gateway
	Gateway ---> Server

While unlikely, nothing prevents that the patch would be maliciously 
altered at the Gateway.

From security PoV, the only way to ensure that the patch was not
altered is to have it signed by the one who wrote it.

> 
> > This way you'd need to compromise multiple machines to achieve the kind
> > of compromise you're worried about. And you can add more independent
> > machines until you're satisfied that the risk is low enough :)  
> 
> Yep, exactly.  This is basically the theory behind Certificate
> Transparency[1], applied to patches.  For example, here's the
> certificate transparency report for kernel.org:
> 
>    https://transparencyreport.google.com/https/certificates?cert_search=domain:kernel.org
> 
> 					- Ted
> 
> [1] http://www.certificate-transparency.org/what-is-ct

That won't prevent companies/governments to require the manual
installation of the gateway's certificate on their browers, in order
to be able to navigate using https.

Thanks,
Mauro

Re: RFE: use patchwork to submit a patch

[Theodore Y. Ts'o]

On Mon, Oct 14, 2019 at 10:41:32AM -0300, Mauro Carvalho Chehab wrote:
> It can still have man-in-the-middle (MITM) attacks between the sender and
> vger.kernel.org. Please notice that using https and adding the patch
> via a web interface can also be subject to MITM, as companies and even some
> Countries with strong policy enforcement may have some gateway on their
> infra that will prevent end-to-end encryption[1], blocking direct
> client-server https tunnels.
> 
> [1] They add an internal certificate to the browsers, so that the client
> will see the connection as trustful, but the infra will actually do two
> separate HTTPS encryption:
> 
> 	client  ---> Gateway
> 	Gateway ---> Server
> 
> While unlikely, nothing prevents that the patch would be maliciously 
> altered at the Gateway.
> 
> From security PoV, the only way to ensure that the patch was not
> altered is to have it signed by the one who wrote it.

Well, sure, but the maintainer should be reviewing the patch looking
for problems anyway.  There is the risk that people might slap a
"Reviewed-by:" tag on a patch without sufficiently careful review if
it's from a prominent kernel contributor, but we've always had that
problem.  And so nothing, not even a digitally signed patch from a
reviewer should absolve the maintainer from doing their own review.

Now, one might argue that if there is a forged patch from "famous
kernel developer A", followed up with a forged patch from "famous
kernel developer B", that might cause a maintainer to happily take the
patch without doing their own, independent review, for scaling
reasons.  But that's a "vulernability" we've lived with for a long
time, since today neither patches or "Reviewed-by" messages are
usually signed.

And at least (as far as we know) no one has managed to sneak a
malicious patch with a zero-day hidden with malice aforethought.  And
perhaps that shouldn't be surprising.  We seem to be quite capable of
introducing our own security vulererabilities without "help", so
perhaps most malicious attackers wouldn't want to do something which
could be so easily detected, when they can just pay money to a black
hat hacker.

	       	    	     	       - Ted

Re: RFE: use patchwork to submit a patch

[Mauro Carvalho Chehab]

Em Mon, 14 Oct 2019 09:53:58 -0400
"Theodore Y. Ts'o" <tytso@mit.edu> escreveu:

> On Mon, Oct 14, 2019 at 10:41:32AM -0300, Mauro Carvalho Chehab wrote:
> > It can still have man-in-the-middle (MITM) attacks between the sender and
> > vger.kernel.org. Please notice that using https and adding the patch
> > via a web interface can also be subject to MITM, as companies and even some
> > Countries with strong policy enforcement may have some gateway on their
> > infra that will prevent end-to-end encryption[1], blocking direct
> > client-server https tunnels.
> > 
> > [1] They add an internal certificate to the browsers, so that the client
> > will see the connection as trustful, but the infra will actually do two
> > separate HTTPS encryption:
> > 
> > 	client  ---> Gateway
> > 	Gateway ---> Server
> > 
> > While unlikely, nothing prevents that the patch would be maliciously 
> > altered at the Gateway.
> > 
> > From security PoV, the only way to ensure that the patch was not
> > altered is to have it signed by the one who wrote it.  
> 
> Well, sure, but the maintainer should be reviewing the patch looking
> for problems anyway.  There is the risk that people might slap a
> "Reviewed-by:" tag on a patch without sufficiently careful review if
> it's from a prominent kernel contributor, but we've always had that
> problem.  And so nothing, not even a digitally signed patch from a
> reviewer should absolve the maintainer from doing their own review.

Yeah, our current security model is based at the maintainer for
him to do his duties, properly reviewing the patch.

Yet, at the example that Daniel gave:

Instead of:

	if ((permissions == allowed) && other_stuff) {
		do_things();
	}
	do_more_stuff(permissions);

Patch was maliciously modified to:

	if ((permission == allowed) && other_stuff) {
		do_things();
	}
	do_more_stuff(permissions);

I suspect that a change like that might sleep though the maintainer's
review.

> 
> Now, one might argue that if there is a forged patch from "famous
> kernel developer A", followed up with a forged patch from "famous
> kernel developer B", that might cause a maintainer to happily take the
> patch without doing their own, independent review, for scaling
> reasons.  But that's a "vulernability" we've lived with for a long
> time, since today neither patches or "Reviewed-by" messages are
> usually signed.
> 
> And at least (as far as we know) no one has managed to sneak a
> malicious patch with a zero-day hidden with malice aforethought.  And
> perhaps that shouldn't be surprising.  We seem to be quite capable of
> introducing our own security vulererabilities without "help", so
> perhaps most malicious attackers wouldn't want to do something which
> could be so easily detected, when they can just pay money to a black
> hat hacker.

True, but, once we discover a patch introduced with a malicious code,
some action should be taken to eliminate the source of the bad code.

I mean, one thing is if "famous kernel developer A" maliciously
wrote a patch to violate security. The other thing is that the
infra used by "famous kernel developer A" is not safe, and has
some hidden back hat hacker in the middle of it.

The only way to identify that is by using signed patches/PRs.


Thanks,
Mauro

Re: RFE: use patchwork to submit a patch

[Dmitry Vyukov]

On Fri, Oct 11, 2019 at 7:20 PM Shuah Khan <skhan@linuxfoundation.org> wrote:
>
> On 10/11/19 2:57 AM, Greg KH wrote:
> > On Thu, Oct 10, 2019 at 10:41:50AM -0400, Konstantin Ryabitsev wrote:
> >> Hi, all:
> >>
> >> I would like to propose a new (large) feature to patchwork with the goal to
> >> make the process of submitting a patch easier for newbies and people
> >> generally less familiar with patch-based development. This was discussed
> >> previously on the workflows list:
> >> https://lore.kernel.org/workflows/20190930202451.GA14403@pure.paranoia.local/
> >>
> >> How I envision this would work:
> >>
> >> - user creates an account (which requires a mail confirmation) >> - they choose a "submit patch" option from the menu
> >> - the patch submission screen has a succession of screens:
> >>
> >>   1. a screen with a single field allowing a user to paste a URL to     their
> >> fork of the git repository. Once submitted, patchwork does a     "git
> >> ls-remote" to attempt to get a list of refs and to verify that     this is
> >> indeed a valid git repository
> >
> > s/valid git repository/valid git repository based on the kernel git tree/
> >
> > Otherwise you might be sending out lots of emails for other projects :)
> >
> >>
> >>   2. next screen asks the user to select the ref to work from using the
> >> list obtained from the remote. Once submitted, patchwork performs a     `git
> >> clone --reference` to clone the repository locally using a     local fork of
> >> the same repo to minimize object transfer. This part     requires that:
> >>        a. patchwork project is configured with a path to a local fork,
> >> if this feature is enabled for a project
> >>        b. that fork is kept current via some mechanism outside of
> >> patchwork (e.g. with grokmirror)
> >>        c. there is some sanity-checking during the clone process to
> >> avoid abuse (e.g. a sane timeout, a tmpdir with limited size,          etc
> >> -- other suggestions welcome)
> >>
> >>   3. next screen asks the user to pick a starting commit from the log.
> >> Once submitted, patchwork generates the patch from the commit     provided
> >> to the tip of the branch selected by the user earlier,
> >>      using git format-patch.
> >>
> >>   4. next screen asks the user to review the patch to make sure this is
> >> what they want to submit. Once confirmed, patchwork performs two
> >> admin-defined optional hooks:
> >>
> >>        a. a hook to generate a list of cc's (e.g. get_maintainer.pl)
> >>        b. a sanity check hook (e.g. checkpatch.pl)
> >
> > I will note that many "first patch" submissions are checkpatch.pl
> > cleanups for staging.  When doing that, I require that they do "one
> > logical change per patch", which means that many of the individual
> > patches themselves will not be checkpatch.pl clean, because many lines
> > have multiple issues with them (tabs, spaces, format, length, etc.)
> >
> > So other than that minor thing, sounds interesting.  It's hard to
> > determine just how difficult the whole "set up git and send a patch out"
> > process is for people these days given the _huge_ numbers of new
> > contributions we keep getting, and the numerous good tutorials we have
> > created that spell out exactly how to do this.
> >
> > So you might be "solving" a problem that we don't really have.  It's
> > hard to tell :(
> >
>
> I agree with this. I don't think this a problem that is worth solving.
> When a new developer wants to send a patch, they don't need to create
> any accounts. They setup their email client and send patch.
>
> We have several resources that walk them through setting up email
> clients and sending patches. checkpatch.pl can be automated with
> git hooks.
>
> >> I know this is a pretty big RFE, and I would like to hear your thoughts
> >> about this. If there is general agreement that this is doable/good idea, I
> >> may be able to come up with funding for this development as part of the
> >> overall tooling improvement proposal.
> >
> > The workflow seems sane, and matches what most people do today, with the
> > exception that it "solves" the git send-email issue, right?  Is that our
> > biggest barrier?
> >
> > I would recommend interviewing some of the recent kernel mentor project
> > and outreachy applicants first, to try to determine exactly what their
> > problems, if any, were with our development process.  If they say that
> > this type of tool/workflow would have saved them hours of time and
> > energy, then that's a great indication that we should try to do this.
> >
>
> I would say considering the number of applicants to mentorship program
> and new developers it will be lot overhead to require them to create
> patchwork accounts, and it might even be hard overtime. A lot of them
> start out and drop out in the middle. With the current setup, nothing
> to cleanup.
>
> Setting up email clients and git hooks is one time task. It is the
> easiest of the learning curve for many new developers. New developers
> struggle with getting the change logs right, coding styles right, and
> responding to review comments and acting on them.
>
> These aren't something that can be automated and they just have to
> learn through experience of sending patches.
>
> My opinion based on contact with new developers as well running the
> mentorship program, I would sat this isn't something that needs
> solving.
>
> thanks,
> -- Shuah


As one data point, I cannot send emails with git send-email anymore.
It used to work, then broke and I don't know how to fix it. Now it says:

5.7.8 Username and Password not accepted. Learn more at
5.7.8  https://support.google.com/mail/?p=BadCredentials
s10sm8376885wrr.5 - gsmtp

I suspect it has something to do with two factor auth.
So that's it: it cannot contribute to kernel right now.
I will not consider time spent fixing it as useful time investment.

Any kernel documentation that I can find for gmail, mentions config
that I am already using and that is not working:
https://www.kernel.org/doc/html/latest/search.html?q=gmail&check_keywords=yes&area=default#
https://www.kernel.org/doc/html/latest/process/email-clients.html?highlight=gmail

As another data point, I spoke to KP Singh at the Plumbers. He is a
"returning" kernel developer (so already did this before), he said it
took him 3 days and 52 configurations changes (all were committed to
git, so was possible to count exactly) to setup mail client properly.
And he is "staffed" to do kernel work, I would expect that most people
who don't _have_ to do kernel contributions will turn away half-way.

As another data point, several people told me that they are afraid of
sending kernel patches b/c there is so much "on you" to do right.

I would say that we need to aim at  a process that does not require a
friendly experienced person to answer any of your questions in the
common case. Lots of people will simply not ask any questions.

Re: RFE: use patchwork to submit a patch

[Laurent Pinchart]

Hi Dmitry,

On Mon, Oct 14, 2019 at 04:58:17PM +0200, Dmitry Vyukov wrote:
> On Fri, Oct 11, 2019 at 7:20 PM Shuah Khan wrote:
> > On 10/11/19 2:57 AM, Greg KH wrote:
> >> On Thu, Oct 10, 2019 at 10:41:50AM -0400, Konstantin Ryabitsev wrote:
> >>> Hi, all:
> >>>
> >>> I would like to propose a new (large) feature to patchwork with the goal to
> >>> make the process of submitting a patch easier for newbies and people
> >>> generally less familiar with patch-based development. This was discussed
> >>> previously on the workflows list:
> >>> https://lore.kernel.org/workflows/20190930202451.GA14403@pure.paranoia.local/
> >>>
> >>> How I envision this would work:
> >>>
> >>> - user creates an account (which requires a mail confirmation) >> - they choose a "submit patch" option from the menu
> >>> - the patch submission screen has a succession of screens:
> >>>
> >>>   1. a screen with a single field allowing a user to paste a URL to     their
> >>> fork of the git repository. Once submitted, patchwork does a     "git
> >>> ls-remote" to attempt to get a list of refs and to verify that     this is
> >>> indeed a valid git repository
> >>
> >> s/valid git repository/valid git repository based on the kernel git tree/
> >>
> >> Otherwise you might be sending out lots of emails for other projects :)
> >>
> >>>   2. next screen asks the user to select the ref to work from using the
> >>> list obtained from the remote. Once submitted, patchwork performs a     `git
> >>> clone --reference` to clone the repository locally using a     local fork of
> >>> the same repo to minimize object transfer. This part     requires that:
> >>>        a. patchwork project is configured with a path to a local fork,
> >>> if this feature is enabled for a project
> >>>        b. that fork is kept current via some mechanism outside of
> >>> patchwork (e.g. with grokmirror)
> >>>        c. there is some sanity-checking during the clone process to
> >>> avoid abuse (e.g. a sane timeout, a tmpdir with limited size,          etc
> >>> -- other suggestions welcome)
> >>>
> >>>   3. next screen asks the user to pick a starting commit from the log.
> >>> Once submitted, patchwork generates the patch from the commit     provided
> >>> to the tip of the branch selected by the user earlier,
> >>>      using git format-patch.
> >>>
> >>>   4. next screen asks the user to review the patch to make sure this is
> >>> what they want to submit. Once confirmed, patchwork performs two
> >>> admin-defined optional hooks:
> >>>
> >>>        a. a hook to generate a list of cc's (e.g. get_maintainer.pl)
> >>>        b. a sanity check hook (e.g. checkpatch.pl)
> >>
> >> I will note that many "first patch" submissions are checkpatch.pl
> >> cleanups for staging.  When doing that, I require that they do "one
> >> logical change per patch", which means that many of the individual
> >> patches themselves will not be checkpatch.pl clean, because many lines
> >> have multiple issues with them (tabs, spaces, format, length, etc.)
> >>
> >> So other than that minor thing, sounds interesting.  It's hard to
> >> determine just how difficult the whole "set up git and send a patch out"
> >> process is for people these days given the _huge_ numbers of new
> >> contributions we keep getting, and the numerous good tutorials we have
> >> created that spell out exactly how to do this.
> >>
> >> So you might be "solving" a problem that we don't really have.  It's
> >> hard to tell :(
> >
> > I agree with this. I don't think this a problem that is worth solving.
> > When a new developer wants to send a patch, they don't need to create
> > any accounts. They setup their email client and send patch.
> >
> > We have several resources that walk them through setting up email
> > clients and sending patches. checkpatch.pl can be automated with
> > git hooks.
> >
> >>> I know this is a pretty big RFE, and I would like to hear your thoughts
> >>> about this. If there is general agreement that this is doable/good idea, I
> >>> may be able to come up with funding for this development as part of the
> >>> overall tooling improvement proposal.
> >>
> >> The workflow seems sane, and matches what most people do today, with the
> >> exception that it "solves" the git send-email issue, right?  Is that our
> >> biggest barrier?
> >>
> >> I would recommend interviewing some of the recent kernel mentor project
> >> and outreachy applicants first, to try to determine exactly what their
> >> problems, if any, were with our development process.  If they say that
> >> this type of tool/workflow would have saved them hours of time and
> >> energy, then that's a great indication that we should try to do this.
> >
> > I would say considering the number of applicants to mentorship program
> > and new developers it will be lot overhead to require them to create
> > patchwork accounts, and it might even be hard overtime. A lot of them
> > start out and drop out in the middle. With the current setup, nothing
> > to cleanup.
> >
> > Setting up email clients and git hooks is one time task. It is the
> > easiest of the learning curve for many new developers. New developers
> > struggle with getting the change logs right, coding styles right, and
> > responding to review comments and acting on them.
> >
> > These aren't something that can be automated and they just have to
> > learn through experience of sending patches.
> >
> > My opinion based on contact with new developers as well running the
> > mentorship program, I would sat this isn't something that needs
> > solving.
> 
> As one data point, I cannot send emails with git send-email anymore.
> It used to work, then broke and I don't know how to fix it. Now it says:
> 
> 5.7.8 Username and Password not accepted. Learn more at
> 5.7.8  https://support.google.com/mail/?p=BadCredentials
> s10sm8376885wrr.5 - gsmtp
> 
> I suspect it has something to do with two factor auth.
> So that's it: it cannot contribute to kernel right now.
> I will not consider time spent fixing it as useful time investment.

Starting from an estalished working process, a change on your e-mail
provider side broke your workflow. The exact same problem could happen
regardless of how changes get submitted, a corporate HTTP proxy or
firewall could also break HTTP-based submissions.

Now, gmail being one of the largest e-mail providers, I think it's fair
to consider that the kernel community should provide clear and easy to
follow instructions on how to use git-send-email with gmail. In
particular, with two-factor authentication being widespread, how to set
it up with git-send-email should likely be described in
https://www.kernel.org/doc/html/latest/process/email-clients.html.
Blaming it solely on the SMTP protocol is a bit of a shortcut.

> Any kernel documentation that I can find for gmail, mentions config
> that I am already using and that is not working:
> https://www.kernel.org/doc/html/latest/search.html?q=gmail&check_keywords=yes&area=default#
> https://www.kernel.org/doc/html/latest/process/email-clients.html?highlight=gmail
> 
> As another data point, I spoke to KP Singh at the Plumbers. He is a
> "returning" kernel developer (so already did this before), he said it
> took him 3 days and 52 configurations changes (all were committed to
> git, so was possible to count exactly) to setup mail client properly.
> And he is "staffed" to do kernel work, I would expect that most people
> who don't _have_ to do kernel contributions will turn away half-way.

That's very interesting information, is there any way that more details
about the 52 steps could be shared ?

> As another data point, several people told me that they are afraid of
> sending kernel patches b/c there is so much "on you" to do right.

Is that related to the submission mechanism, or to all the other things
you need to get right ?

> I would say that we need to aim at  a process that does not require a
> friendly experienced person to answer any of your questions in the
> common case. Lots of people will simply not ask any questions.

I fully agree with that.

-- 
Regards,

Laurent Pinchart

Re: RFE: use patchwork to submit a patch

[Greg KH]

On Mon, Oct 14, 2019 at 04:58:17PM +0200, Dmitry Vyukov wrote:
> On Fri, Oct 11, 2019 at 7:20 PM Shuah Khan <skhan@linuxfoundation.org> wrote:
> >
> > On 10/11/19 2:57 AM, Greg KH wrote:
> > > On Thu, Oct 10, 2019 at 10:41:50AM -0400, Konstantin Ryabitsev wrote:
> > >> Hi, all:
> > >>
> > >> I would like to propose a new (large) feature to patchwork with the goal to
> > >> make the process of submitting a patch easier for newbies and people
> > >> generally less familiar with patch-based development. This was discussed
> > >> previously on the workflows list:
> > >> https://lore.kernel.org/workflows/20190930202451.GA14403@pure.paranoia.local/
> > >>
> > >> How I envision this would work:
> > >>
> > >> - user creates an account (which requires a mail confirmation) >> - they choose a "submit patch" option from the menu
> > >> - the patch submission screen has a succession of screens:
> > >>
> > >>   1. a screen with a single field allowing a user to paste a URL to     their
> > >> fork of the git repository. Once submitted, patchwork does a     "git
> > >> ls-remote" to attempt to get a list of refs and to verify that     this is
> > >> indeed a valid git repository
> > >
> > > s/valid git repository/valid git repository based on the kernel git tree/
> > >
> > > Otherwise you might be sending out lots of emails for other projects :)
> > >
> > >>
> > >>   2. next screen asks the user to select the ref to work from using the
> > >> list obtained from the remote. Once submitted, patchwork performs a     `git
> > >> clone --reference` to clone the repository locally using a     local fork of
> > >> the same repo to minimize object transfer. This part     requires that:
> > >>        a. patchwork project is configured with a path to a local fork,
> > >> if this feature is enabled for a project
> > >>        b. that fork is kept current via some mechanism outside of
> > >> patchwork (e.g. with grokmirror)
> > >>        c. there is some sanity-checking during the clone process to
> > >> avoid abuse (e.g. a sane timeout, a tmpdir with limited size,          etc
> > >> -- other suggestions welcome)
> > >>
> > >>   3. next screen asks the user to pick a starting commit from the log.
> > >> Once submitted, patchwork generates the patch from the commit     provided
> > >> to the tip of the branch selected by the user earlier,
> > >>      using git format-patch.
> > >>
> > >>   4. next screen asks the user to review the patch to make sure this is
> > >> what they want to submit. Once confirmed, patchwork performs two
> > >> admin-defined optional hooks:
> > >>
> > >>        a. a hook to generate a list of cc's (e.g. get_maintainer.pl)
> > >>        b. a sanity check hook (e.g. checkpatch.pl)
> > >
> > > I will note that many "first patch" submissions are checkpatch.pl
> > > cleanups for staging.  When doing that, I require that they do "one
> > > logical change per patch", which means that many of the individual
> > > patches themselves will not be checkpatch.pl clean, because many lines
> > > have multiple issues with them (tabs, spaces, format, length, etc.)
> > >
> > > So other than that minor thing, sounds interesting.  It's hard to
> > > determine just how difficult the whole "set up git and send a patch out"
> > > process is for people these days given the _huge_ numbers of new
> > > contributions we keep getting, and the numerous good tutorials we have
> > > created that spell out exactly how to do this.
> > >
> > > So you might be "solving" a problem that we don't really have.  It's
> > > hard to tell :(
> > >
> >
> > I agree with this. I don't think this a problem that is worth solving.
> > When a new developer wants to send a patch, they don't need to create
> > any accounts. They setup their email client and send patch.
> >
> > We have several resources that walk them through setting up email
> > clients and sending patches. checkpatch.pl can be automated with
> > git hooks.
> >
> > >> I know this is a pretty big RFE, and I would like to hear your thoughts
> > >> about this. If there is general agreement that this is doable/good idea, I
> > >> may be able to come up with funding for this development as part of the
> > >> overall tooling improvement proposal.
> > >
> > > The workflow seems sane, and matches what most people do today, with the
> > > exception that it "solves" the git send-email issue, right?  Is that our
> > > biggest barrier?
> > >
> > > I would recommend interviewing some of the recent kernel mentor project
> > > and outreachy applicants first, to try to determine exactly what their
> > > problems, if any, were with our development process.  If they say that
> > > this type of tool/workflow would have saved them hours of time and
> > > energy, then that's a great indication that we should try to do this.
> > >
> >
> > I would say considering the number of applicants to mentorship program
> > and new developers it will be lot overhead to require them to create
> > patchwork accounts, and it might even be hard overtime. A lot of them
> > start out and drop out in the middle. With the current setup, nothing
> > to cleanup.
> >
> > Setting up email clients and git hooks is one time task. It is the
> > easiest of the learning curve for many new developers. New developers
> > struggle with getting the change logs right, coding styles right, and
> > responding to review comments and acting on them.
> >
> > These aren't something that can be automated and they just have to
> > learn through experience of sending patches.
> >
> > My opinion based on contact with new developers as well running the
> > mentorship program, I would sat this isn't something that needs
> > solving.
> >
> > thanks,
> > -- Shuah
> 
> 
> As one data point, I cannot send emails with git send-email anymore.
> It used to work, then broke and I don't know how to fix it. Now it says:
> 
> 5.7.8 Username and Password not accepted. Learn more at
> 5.7.8  https://support.google.com/mail/?p=BadCredentials
> s10sm8376885wrr.5 - gsmtp
> 
> I suspect it has something to do with two factor auth.
> So that's it: it cannot contribute to kernel right now.

That is because your employer changed how it manages imap.  So yes, this
configuration is now broken, you can not contribute to the kernel this
way.  They know about it, and there's an "opt-out" list you can sign up
for if you want to fix it.  Nothing the community can do about something
crazy like this.

> As another data point, I spoke to KP Singh at the Plumbers. He is a
> "returning" kernel developer (so already did this before), he said it
> took him 3 days and 52 configurations changes (all were committed to
> git, so was possible to count exactly) to setup mail client properly.
> And he is "staffed" to do kernel work, I would expect that most people
> who don't _have_ to do kernel contributions will turn away half-way.
> 
> As another data point, several people told me that they are afraid of
> sending kernel patches b/c there is so much "on you" to do right.
> 
> I would say that we need to aim at  a process that does not require a
> friendly experienced person to answer any of your questions in the
> common case. Lots of people will simply not ask any questions.

Again, interview the outreachy and mentorship applicants and see what
they say about this.

All corporate email systems do crazy things with email to help prevent
them from participating in Linux kernel development.  We have known this
for decades.  Is it the community's job to fix that, or is it the
individual company's job to do that for when they want to have people
participate?

There's a good reason almost all Linux groups at companies have a Linux
email server in the corner from which to send out patches from.  This
started 2 decades ago with IBM, and continues to this day with many many
many other companies.  That's proof that if a company does want to
participate, it will do the needed work to do so.

But that's not the group of people we are trying to help here, are we?
I can't tell, there seems to be complaints from both sides (newbies and
companies...)

thanks,

greg k-h

Re: RFE: use patchwork to submit a patch

[Konstantin Ryabitsev]

On Mon, Oct 14, 2019 at 11:28:59AM -0300, Mauro Carvalho Chehab wrote:
>Yeah, our current security model is based at the maintainer for
>him to do his duties, properly reviewing the patch.
>
>Yet, at the example that Daniel gave:
>
>Instead of:
>
>	if ((permissions == allowed) && other_stuff) {
>		do_things();
>	}
>	do_more_stuff(permissions);
>
>Patch was maliciously modified to:
>
>	if ((permission == allowed) && other_stuff) {
>		do_things();
>	}
>	do_more_stuff(permissions);
>
>I suspect that a change like that might sleep though the maintainer's
>review.

I submitted a proposal to the git list that would address this, but it 
saw little uptake in the discussion:

https://public-inbox.org/git/20190910121324.GA6867@pure.paranoia.local/t/#u

To summarize it in brief, I suggested using minisign signatures and 
git-tracked TOFU (trust on first use) databases. TOFU is not perfect, 
but it's better than nothing at all, which is what we have right now.

-K

Re: RFE: use patchwork to submit a patch

[Laurent Pinchart]

Hi Greg,

On Mon, Oct 14, 2019 at 05:17:11PM +0200, Greg KH wrote:
> On Mon, Oct 14, 2019 at 04:58:17PM +0200, Dmitry Vyukov wrote:
> > On Fri, Oct 11, 2019 at 7:20 PM Shuah Khan wrote:
> > > On 10/11/19 2:57 AM, Greg KH wrote:
> > > > On Thu, Oct 10, 2019 at 10:41:50AM -0400, Konstantin Ryabitsev wrote:
> > > >> Hi, all:
> > > >>
> > > >> I would like to propose a new (large) feature to patchwork with the goal to
> > > >> make the process of submitting a patch easier for newbies and people
> > > >> generally less familiar with patch-based development. This was discussed
> > > >> previously on the workflows list:
> > > >> https://lore.kernel.org/workflows/20190930202451.GA14403@pure.paranoia.local/
> > > >>
> > > >> How I envision this would work:
> > > >>
> > > >> - user creates an account (which requires a mail confirmation) >> - they choose a "submit patch" option from the menu
> > > >> - the patch submission screen has a succession of screens:
> > > >>
> > > >>   1. a screen with a single field allowing a user to paste a URL to     their
> > > >> fork of the git repository. Once submitted, patchwork does a     "git
> > > >> ls-remote" to attempt to get a list of refs and to verify that     this is
> > > >> indeed a valid git repository
> > > >
> > > > s/valid git repository/valid git repository based on the kernel git tree/
> > > >
> > > > Otherwise you might be sending out lots of emails for other projects :)
> > > >
> > > >>
> > > >>   2. next screen asks the user to select the ref to work from using the
> > > >> list obtained from the remote. Once submitted, patchwork performs a     `git
> > > >> clone --reference` to clone the repository locally using a     local fork of
> > > >> the same repo to minimize object transfer. This part     requires that:
> > > >>        a. patchwork project is configured with a path to a local fork,
> > > >> if this feature is enabled for a project
> > > >>        b. that fork is kept current via some mechanism outside of
> > > >> patchwork (e.g. with grokmirror)
> > > >>        c. there is some sanity-checking during the clone process to
> > > >> avoid abuse (e.g. a sane timeout, a tmpdir with limited size,          etc
> > > >> -- other suggestions welcome)
> > > >>
> > > >>   3. next screen asks the user to pick a starting commit from the log.
> > > >> Once submitted, patchwork generates the patch from the commit     provided
> > > >> to the tip of the branch selected by the user earlier,
> > > >>      using git format-patch.
> > > >>
> > > >>   4. next screen asks the user to review the patch to make sure this is
> > > >> what they want to submit. Once confirmed, patchwork performs two
> > > >> admin-defined optional hooks:
> > > >>
> > > >>        a. a hook to generate a list of cc's (e.g. get_maintainer.pl)
> > > >>        b. a sanity check hook (e.g. checkpatch.pl)
> > > >
> > > > I will note that many "first patch" submissions are checkpatch.pl
> > > > cleanups for staging.  When doing that, I require that they do "one
> > > > logical change per patch", which means that many of the individual
> > > > patches themselves will not be checkpatch.pl clean, because many lines
> > > > have multiple issues with them (tabs, spaces, format, length, etc.)
> > > >
> > > > So other than that minor thing, sounds interesting.  It's hard to
> > > > determine just how difficult the whole "set up git and send a patch out"
> > > > process is for people these days given the _huge_ numbers of new
> > > > contributions we keep getting, and the numerous good tutorials we have
> > > > created that spell out exactly how to do this.
> > > >
> > > > So you might be "solving" a problem that we don't really have.  It's
> > > > hard to tell :(
> > > >
> > >
> > > I agree with this. I don't think this a problem that is worth solving.
> > > When a new developer wants to send a patch, they don't need to create
> > > any accounts. They setup their email client and send patch.
> > >
> > > We have several resources that walk them through setting up email
> > > clients and sending patches. checkpatch.pl can be automated with
> > > git hooks.
> > >
> > > >> I know this is a pretty big RFE, and I would like to hear your thoughts
> > > >> about this. If there is general agreement that this is doable/good idea, I
> > > >> may be able to come up with funding for this development as part of the
> > > >> overall tooling improvement proposal.
> > > >
> > > > The workflow seems sane, and matches what most people do today, with the
> > > > exception that it "solves" the git send-email issue, right?  Is that our
> > > > biggest barrier?
> > > >
> > > > I would recommend interviewing some of the recent kernel mentor project
> > > > and outreachy applicants first, to try to determine exactly what their
> > > > problems, if any, were with our development process.  If they say that
> > > > this type of tool/workflow would have saved them hours of time and
> > > > energy, then that's a great indication that we should try to do this.
> > > >
> > >
> > > I would say considering the number of applicants to mentorship program
> > > and new developers it will be lot overhead to require them to create
> > > patchwork accounts, and it might even be hard overtime. A lot of them
> > > start out and drop out in the middle. With the current setup, nothing
> > > to cleanup.
> > >
> > > Setting up email clients and git hooks is one time task. It is the
> > > easiest of the learning curve for many new developers. New developers
> > > struggle with getting the change logs right, coding styles right, and
> > > responding to review comments and acting on them.
> > >
> > > These aren't something that can be automated and they just have to
> > > learn through experience of sending patches.
> > >
> > > My opinion based on contact with new developers as well running the
> > > mentorship program, I would sat this isn't something that needs
> > > solving.
> > 
> > As one data point, I cannot send emails with git send-email anymore.
> > It used to work, then broke and I don't know how to fix it. Now it says:
> > 
> > 5.7.8 Username and Password not accepted. Learn more at
> > 5.7.8  https://support.google.com/mail/?p=BadCredentials
> > s10sm8376885wrr.5 - gsmtp
> > 
> > I suspect it has something to do with two factor auth.
> > So that's it: it cannot contribute to kernel right now.
> 
> That is because your employer changed how it manages imap.  So yes, this
> configuration is now broken, you can not contribute to the kernel this
> way.  They know about it, and there's an "opt-out" list you can sign up
> for if you want to fix it.  Nothing the community can do about something
> crazy like this.
> 
> > As another data point, I spoke to KP Singh at the Plumbers. He is a
> > "returning" kernel developer (so already did this before), he said it
> > took him 3 days and 52 configurations changes (all were committed to
> > git, so was possible to count exactly) to setup mail client properly.
> > And he is "staffed" to do kernel work, I would expect that most people
> > who don't _have_ to do kernel contributions will turn away half-way.
> > 
> > As another data point, several people told me that they are afraid of
> > sending kernel patches b/c there is so much "on you" to do right.
> > 
> > I would say that we need to aim at  a process that does not require a
> > friendly experienced person to answer any of your questions in the
> > common case. Lots of people will simply not ask any questions.
> 
> Again, interview the outreachy and mentorship applicants and see what
> they say about this.
> 
> All corporate email systems do crazy things with email to help prevent
> them from participating in Linux kernel development.  We have known this
> for decades.  Is it the community's job to fix that, or is it the
> individual company's job to do that for when they want to have people
> participate?

As I have replied separately, I don't think it's our duty to fix
corporate e-mail servers, but when we know about common issues with
large e-mail providers, and especially when we know of possible
workarounds, I think it would be useful to centralise that information
in a place that newcomers can easily find.

> There's a good reason almost all Linux groups at companies have a Linux
> email server in the corner from which to send out patches from.  This
> started 2 decades ago with IBM, and continues to this day with many many
> many other companies.  That's proof that if a company does want to
> participate, it will do the needed work to do so.
> 
> But that's not the group of people we are trying to help here, are we?
> I can't tell, there seems to be complaints from both sides (newbies and
> companies...)

-- 
Regards,

Laurent Pinchart

Re: RFE: use patchwork to submit a patch

[Laurent Pinchart]

Hi Konstantin,

On Fri, Oct 11, 2019 at 05:35:53PM -0400, Konstantin Ryabitsev wrote:
> On Fri, Oct 11, 2019 at 09:23:08PM +0000, Eric Wong wrote:
> >> (This is the same reason I generally disagree with Eric Wong about
> >> preserving SMTP as the primary transmission protocol -- I've heard lots of
> >> complaints both from kernel developers and especially from people trying to
> >> contribute to CAF about corporate policies actually making it impossible to
> >> submit patches -- and no, using a different mail server is not a possibility
> >> for them because it can be a firing offense under their IT AUP rules.)
> >
> > I'm not opposed to a webmail interface tailored to kernel hacking
> > which does stuff like checkpatch.pl and get_maintainer.pl before
> > sending (similar to your patchwork proposal and
> > gitgadgetgadget).  That would get around security appliances
> > but SMTP would still be used in the background.
> >
> > Or offer full-blown HTTPS webmail + IMAP + SMTP access like any
> > other webmail provider + checkpatch + get_maintainer helpers.
> 
> Well, this is the bit where I say that it may not be allowed by 
> corporate rules. I see this all the time in CAF/Android world where 
> companies *require* that all email goes through their SMTP server so 
> that it can be properly logged (often for legal reasons). And it is 
> often equally required that any code submissions come from 
> person@corporate.com and not person@free-email-provider.com for 
> License/CLA reasons, so setting up a webmail server is not a solution 
> either.
> 
> This is basically why SMTP sucks in my view -- and it's worthless trying 
> to pick fights with IT departments, because they are told to do so by 
> lawyers. So, I want to take SMTP out of the equation:
> 
> 1. provide a way for someone to submit a patch using a web interface 
>    (but still in a way that From: is their corporate ID)
> 2. use individual git feeds as a way to send out patches instead of 
>    always being secondary to SMTP

If the goal is to work around SMTP-related technical issues, is a web UI
really the best way to go ? Wouldn't it be better to do the same through
a git push ? We could setup a git server that requires authentication,
and implement a push-to-email bridge. The information that would need to
be entered in a web UI could be put in a tag message, and we could have
a CLI to create the tag from a list of questions.

-- 
Regards,

Laurent Pinchart

Re: RFE: use patchwork to submit a patch

[Theodore Y. Ts'o]

On Mon, Oct 14, 2019 at 04:58:17PM +0200, Dmitry Vyukov wrote:
> As one data point, I cannot send emails with git send-email anymore.
> It used to work, then broke and I don't know how to fix it. Now it says:
> 
> 5.7.8 Username and Password not accepted. Learn more at
> 5.7.8  https://support.google.com/mail/?p=BadCredentials
> s10sm8376885wrr.5 - gsmtp
> 
> I suspect it has something to do with two factor auth.
> So that's it: it cannot contribute to kernel right now.
> I will not consider time spent fixing it as useful time investment.

Note that this is a Googler-specific problem, as it's due to
corp-security requirements that other GMail users and most GSuites
customers won't have.  (For example, you could use another SMTP server
and bypass the corp. mail submission server, but for google.com e-mail
addresses, the DMARC settings foreclose that particular workaround.)

So it's an extreme case, and is an example of corp e-mail insanity.
The good news is that for companies that are really serious about
working with upstream, they will roll out solutions for their own
customers.  For example, with IBM, since Lotus notes was maximally
unfriendly to OSS development practices, they created a separate mail
system with ldap@linux.ibm.com addresses be cause it was simply
impossible to engage with upstream development using ibm.com
addresses.  So there are work arounds, and I encourage you to reach
out to other Google kernel developers --- and if you are unhappy,
please file internal bugs against the corp infrastructure.

Personally, my solution to this problem is to use a non-google address
(tytso@mit.edu) for upstream work, because life is too short to fight
with GMail.  (Although it is doable; as many people can attest.)

> I would say that we need to aim at  a process that does not require a
> friendly experienced person to answer any of your questions in the
> common case. Lots of people will simply not ask any questions.

If you work inside a corporation with "unique" e-mail settings, you
may need to ask a friendly experienced person inside your company, or
look for the "new employee" (Noogler) web documentation for working
with the kernel.

						- Ted

Re: RFE: use patchwork to submit a patch

[Eric Wong]

Geert Uytterhoeven <geert@linux-m68k.org> wrote:
> On Mon, Oct 14, 2019 at 2:12 AM Eric Wong <e@80x24.org> wrote:
> > We can also find creative ways to subvert corporate policies:
> > For example; if their policy specifically prevents outgoing SMTP,
> > "git imap-send" could be used.
> 
> IMAP may be blocked, too?

Yes, was just pointing to "git imap-send" which already exists;
but anything can go over HTTPS POST + Transfer-Encoding:chunked.

> Bascially the only thing you can rely on is HTTP(S), through a proxy,
> possibly with HTTPS inspection through a company-specific trusted
> certificate that allows MITM.

Right, I was tunneling arbitrary data over HTTP/1.1 via
Transfer-Encoding:chunked on both requests/responses over a
decade ago.  Probably won't work with nginx because of input
buffering, but public-inbox-httpd can be made to support it
w/o buffering, too (it already does HTTPS + chunk parsing).

I got something working on the server-side for git:// using Ruby
back in 2009:
  https://public-inbox.org/git/20090702085440.GC11119@dcvr.yhbt.net/

Client-side needed some work, though...

> > If their policy forbids using external "email" services, we'd
> > name it "Kernel Hackers' Messaging System" or something of that
> > sort and say we use an email bridge :>
> 
> Anything named "Hacker" may be blocked, too ;-)

Unpxref, then :>

Re: RFE: use patchwork to submit a patch

[Dmitry Vyukov]

On Mon, Oct 14, 2019 at 10:57 PM Theodore Y. Ts'o <tytso@mit.edu> wrote:
>
> On Mon, Oct 14, 2019 at 04:58:17PM +0200, Dmitry Vyukov wrote:
> > As one data point, I cannot send emails with git send-email anymore.
> > It used to work, then broke and I don't know how to fix it. Now it says:
> >
> > 5.7.8 Username and Password not accepted. Learn more at
> > 5.7.8  https://support.google.com/mail/?p=BadCredentials
> > s10sm8376885wrr.5 - gsmtp
> >
> > I suspect it has something to do with two factor auth.
> > So that's it: it cannot contribute to kernel right now.
> > I will not consider time spent fixing it as useful time investment.
>
> Note that this is a Googler-specific problem, as it's due to
> corp-security requirements that other GMail users and most GSuites
> customers won't have.  (For example, you could use another SMTP server
> and bypass the corp. mail submission server, but for google.com e-mail
> addresses, the DMARC settings foreclose that particular workaround.)
>
> So it's an extreme case, and is an example of corp e-mail insanity.
> The good news is that for companies that are really serious about
> working with upstream, they will roll out solutions for their own
> customers.  For example, with IBM, since Lotus notes was maximally
> unfriendly to OSS development practices, they created a separate mail
> system with ldap@linux.ibm.com addresses be cause it was simply
> impossible to engage with upstream development using ibm.com
> addresses.  So there are work arounds, and I encourage you to reach
> out to other Google kernel developers --- and if you are unhappy,
> please file internal bugs against the corp infrastructure.
>
> Personally, my solution to this problem is to use a non-google address
> (tytso@mit.edu) for upstream work, because life is too short to fight
> with GMail.  (Although it is doable; as many people can attest.)
>
> > I would say that we need to aim at  a process that does not require a
> > friendly experienced person to answer any of your questions in the
> > common case. Lots of people will simply not ask any questions.
>
> If you work inside a corporation with "unique" e-mail settings, you
> may need to ask a friendly experienced person inside your company, or
> look for the "new employee" (Noogler) web documentation for working
> with the kernel.

I used my private gmail account to send emails. My private email
account stopped working.

Re: RFE: use patchwork to submit a patch

[Dmitry Vyukov]

On Mon, Oct 14, 2019 at 5:17 PM Greg KH <gregkh@linuxfoundation.org> wrote:
>
> On Mon, Oct 14, 2019 at 04:58:17PM +0200, Dmitry Vyukov wrote:
> > On Fri, Oct 11, 2019 at 7:20 PM Shuah Khan <skhan@linuxfoundation.org> wrote:
> > >
> > > On 10/11/19 2:57 AM, Greg KH wrote:
> > > > On Thu, Oct 10, 2019 at 10:41:50AM -0400, Konstantin Ryabitsev wrote:
> > > >> Hi, all:
> > > >>
> > > >> I would like to propose a new (large) feature to patchwork with the goal to
> > > >> make the process of submitting a patch easier for newbies and people
> > > >> generally less familiar with patch-based development. This was discussed
> > > >> previously on the workflows list:
> > > >> https://lore.kernel.org/workflows/20190930202451.GA14403@pure.paranoia.local/
> > > >>
> > > >> How I envision this would work:
> > > >>
> > > >> - user creates an account (which requires a mail confirmation) >> - they choose a "submit patch" option from the menu
> > > >> - the patch submission screen has a succession of screens:
> > > >>
> > > >>   1. a screen with a single field allowing a user to paste a URL to     their
> > > >> fork of the git repository. Once submitted, patchwork does a     "git
> > > >> ls-remote" to attempt to get a list of refs and to verify that     this is
> > > >> indeed a valid git repository
> > > >
> > > > s/valid git repository/valid git repository based on the kernel git tree/
> > > >
> > > > Otherwise you might be sending out lots of emails for other projects :)
> > > >
> > > >>
> > > >>   2. next screen asks the user to select the ref to work from using the
> > > >> list obtained from the remote. Once submitted, patchwork performs a     `git
> > > >> clone --reference` to clone the repository locally using a     local fork of
> > > >> the same repo to minimize object transfer. This part     requires that:
> > > >>        a. patchwork project is configured with a path to a local fork,
> > > >> if this feature is enabled for a project
> > > >>        b. that fork is kept current via some mechanism outside of
> > > >> patchwork (e.g. with grokmirror)
> > > >>        c. there is some sanity-checking during the clone process to
> > > >> avoid abuse (e.g. a sane timeout, a tmpdir with limited size,          etc
> > > >> -- other suggestions welcome)
> > > >>
> > > >>   3. next screen asks the user to pick a starting commit from the log.
> > > >> Once submitted, patchwork generates the patch from the commit     provided
> > > >> to the tip of the branch selected by the user earlier,
> > > >>      using git format-patch.
> > > >>
> > > >>   4. next screen asks the user to review the patch to make sure this is
> > > >> what they want to submit. Once confirmed, patchwork performs two
> > > >> admin-defined optional hooks:
> > > >>
> > > >>        a. a hook to generate a list of cc's (e.g. get_maintainer.pl)
> > > >>        b. a sanity check hook (e.g. checkpatch.pl)
> > > >
> > > > I will note that many "first patch" submissions are checkpatch.pl
> > > > cleanups for staging.  When doing that, I require that they do "one
> > > > logical change per patch", which means that many of the individual
> > > > patches themselves will not be checkpatch.pl clean, because many lines
> > > > have multiple issues with them (tabs, spaces, format, length, etc.)
> > > >
> > > > So other than that minor thing, sounds interesting.  It's hard to
> > > > determine just how difficult the whole "set up git and send a patch out"
> > > > process is for people these days given the _huge_ numbers of new
> > > > contributions we keep getting, and the numerous good tutorials we have
> > > > created that spell out exactly how to do this.
> > > >
> > > > So you might be "solving" a problem that we don't really have.  It's
> > > > hard to tell :(
> > > >
> > >
> > > I agree with this. I don't think this a problem that is worth solving.
> > > When a new developer wants to send a patch, they don't need to create
> > > any accounts. They setup their email client and send patch.
> > >
> > > We have several resources that walk them through setting up email
> > > clients and sending patches. checkpatch.pl can be automated with
> > > git hooks.
> > >
> > > >> I know this is a pretty big RFE, and I would like to hear your thoughts
> > > >> about this. If there is general agreement that this is doable/good idea, I
> > > >> may be able to come up with funding for this development as part of the
> > > >> overall tooling improvement proposal.
> > > >
> > > > The workflow seems sane, and matches what most people do today, with the
> > > > exception that it "solves" the git send-email issue, right?  Is that our
> > > > biggest barrier?
> > > >
> > > > I would recommend interviewing some of the recent kernel mentor project
> > > > and outreachy applicants first, to try to determine exactly what their
> > > > problems, if any, were with our development process.  If they say that
> > > > this type of tool/workflow would have saved them hours of time and
> > > > energy, then that's a great indication that we should try to do this.
> > > >
> > >
> > > I would say considering the number of applicants to mentorship program
> > > and new developers it will be lot overhead to require them to create
> > > patchwork accounts, and it might even be hard overtime. A lot of them
> > > start out and drop out in the middle. With the current setup, nothing
> > > to cleanup.
> > >
> > > Setting up email clients and git hooks is one time task. It is the
> > > easiest of the learning curve for many new developers. New developers
> > > struggle with getting the change logs right, coding styles right, and
> > > responding to review comments and acting on them.
> > >
> > > These aren't something that can be automated and they just have to
> > > learn through experience of sending patches.
> > >
> > > My opinion based on contact with new developers as well running the
> > > mentorship program, I would sat this isn't something that needs
> > > solving.
> > >
> > > thanks,
> > > -- Shuah
> >
> >
> > As one data point, I cannot send emails with git send-email anymore.
> > It used to work, then broke and I don't know how to fix it. Now it says:
> >
> > 5.7.8 Username and Password not accepted. Learn more at
> > 5.7.8  https://support.google.com/mail/?p=BadCredentials
> > s10sm8376885wrr.5 - gsmtp
> >
> > I suspect it has something to do with two factor auth.
> > So that's it: it cannot contribute to kernel right now.
>
> That is because your employer changed how it manages imap.  So yes, this
> configuration is now broken, you can not contribute to the kernel this
> way.  They know about it, and there's an "opt-out" list you can sign up
> for if you want to fix it.  Nothing the community can do about something
> crazy like this.

I used my private gmail account to send emails. My private email
account stopped working.


> > As another data point, I spoke to KP Singh at the Plumbers. He is a
> > "returning" kernel developer (so already did this before), he said it
> > took him 3 days and 52 configurations changes (all were committed to
> > git, so was possible to count exactly) to setup mail client properly.
> > And he is "staffed" to do kernel work, I would expect that most people
> > who don't _have_ to do kernel contributions will turn away half-way.
> >
> > As another data point, several people told me that they are afraid of
> > sending kernel patches b/c there is so much "on you" to do right.
> >
> > I would say that we need to aim at  a process that does not require a
> > friendly experienced person to answer any of your questions in the
> > common case. Lots of people will simply not ask any questions.
>
> Again, interview the outreachy and mentorship applicants and see what
> they say about this.
>
> All corporate email systems do crazy things with email to help prevent
> them from participating in Linux kernel development.  We have known this
> for decades.  Is it the community's job to fix that, or is it the
> individual company's job to do that for when they want to have people
> participate?
>
> There's a good reason almost all Linux groups at companies have a Linux
> email server in the corner from which to send out patches from.  This
> started 2 decades ago with IBM, and continues to this day with many many
> many other companies.  That's proof that if a company does want to
> participate, it will do the needed work to do so.
>
> But that's not the group of people we are trying to help here, are we?
> I can't tell, there seems to be complaints from both sides (newbies and
> companies...)
>
> thanks,
>
> greg k-h

Re: RFE: use patchwork to submit a patch

[Dmitry Vyukov]

On Mon, Oct 14, 2019 at 5:12 PM Laurent Pinchart
<laurent.pinchart@ideasonboard.com> wrote:
>
> Hi Dmitry,
>
> On Mon, Oct 14, 2019 at 04:58:17PM +0200, Dmitry Vyukov wrote:
> > On Fri, Oct 11, 2019 at 7:20 PM Shuah Khan wrote:
> > > On 10/11/19 2:57 AM, Greg KH wrote:
> > >> On Thu, Oct 10, 2019 at 10:41:50AM -0400, Konstantin Ryabitsev wrote:
> > >>> Hi, all:
> > >>>
> > >>> I would like to propose a new (large) feature to patchwork with the goal to
> > >>> make the process of submitting a patch easier for newbies and people
> > >>> generally less familiar with patch-based development. This was discussed
> > >>> previously on the workflows list:
> > >>> https://lore.kernel.org/workflows/20190930202451.GA14403@pure.paranoia.local/
> > >>>
> > >>> How I envision this would work:
> > >>>
> > >>> - user creates an account (which requires a mail confirmation) >> - they choose a "submit patch" option from the menu
> > >>> - the patch submission screen has a succession of screens:
> > >>>
> > >>>   1. a screen with a single field allowing a user to paste a URL to     their
> > >>> fork of the git repository. Once submitted, patchwork does a     "git
> > >>> ls-remote" to attempt to get a list of refs and to verify that     this is
> > >>> indeed a valid git repository
> > >>
> > >> s/valid git repository/valid git repository based on the kernel git tree/
> > >>
> > >> Otherwise you might be sending out lots of emails for other projects :)
> > >>
> > >>>   2. next screen asks the user to select the ref to work from using the
> > >>> list obtained from the remote. Once submitted, patchwork performs a     `git
> > >>> clone --reference` to clone the repository locally using a     local fork of
> > >>> the same repo to minimize object transfer. This part     requires that:
> > >>>        a. patchwork project is configured with a path to a local fork,
> > >>> if this feature is enabled for a project
> > >>>        b. that fork is kept current via some mechanism outside of
> > >>> patchwork (e.g. with grokmirror)
> > >>>        c. there is some sanity-checking during the clone process to
> > >>> avoid abuse (e.g. a sane timeout, a tmpdir with limited size,          etc
> > >>> -- other suggestions welcome)
> > >>>
> > >>>   3. next screen asks the user to pick a starting commit from the log.
> > >>> Once submitted, patchwork generates the patch from the commit     provided
> > >>> to the tip of the branch selected by the user earlier,
> > >>>      using git format-patch.
> > >>>
> > >>>   4. next screen asks the user to review the patch to make sure this is
> > >>> what they want to submit. Once confirmed, patchwork performs two
> > >>> admin-defined optional hooks:
> > >>>
> > >>>        a. a hook to generate a list of cc's (e.g. get_maintainer.pl)
> > >>>        b. a sanity check hook (e.g. checkpatch.pl)
> > >>
> > >> I will note that many "first patch" submissions are checkpatch.pl
> > >> cleanups for staging.  When doing that, I require that they do "one
> > >> logical change per patch", which means that many of the individual
> > >> patches themselves will not be checkpatch.pl clean, because many lines
> > >> have multiple issues with them (tabs, spaces, format, length, etc.)
> > >>
> > >> So other than that minor thing, sounds interesting.  It's hard to
> > >> determine just how difficult the whole "set up git and send a patch out"
> > >> process is for people these days given the _huge_ numbers of new
> > >> contributions we keep getting, and the numerous good tutorials we have
> > >> created that spell out exactly how to do this.
> > >>
> > >> So you might be "solving" a problem that we don't really have.  It's
> > >> hard to tell :(
> > >
> > > I agree with this. I don't think this a problem that is worth solving.
> > > When a new developer wants to send a patch, they don't need to create
> > > any accounts. They setup their email client and send patch.
> > >
> > > We have several resources that walk them through setting up email
> > > clients and sending patches. checkpatch.pl can be automated with
> > > git hooks.
> > >
> > >>> I know this is a pretty big RFE, and I would like to hear your thoughts
> > >>> about this. If there is general agreement that this is doable/good idea, I
> > >>> may be able to come up with funding for this development as part of the
> > >>> overall tooling improvement proposal.
> > >>
> > >> The workflow seems sane, and matches what most people do today, with the
> > >> exception that it "solves" the git send-email issue, right?  Is that our
> > >> biggest barrier?
> > >>
> > >> I would recommend interviewing some of the recent kernel mentor project
> > >> and outreachy applicants first, to try to determine exactly what their
> > >> problems, if any, were with our development process.  If they say that
> > >> this type of tool/workflow would have saved them hours of time and
> > >> energy, then that's a great indication that we should try to do this.
> > >
> > > I would say considering the number of applicants to mentorship program
> > > and new developers it will be lot overhead to require them to create
> > > patchwork accounts, and it might even be hard overtime. A lot of them
> > > start out and drop out in the middle. With the current setup, nothing
> > > to cleanup.
> > >
> > > Setting up email clients and git hooks is one time task. It is the
> > > easiest of the learning curve for many new developers. New developers
> > > struggle with getting the change logs right, coding styles right, and
> > > responding to review comments and acting on them.
> > >
> > > These aren't something that can be automated and they just have to
> > > learn through experience of sending patches.
> > >
> > > My opinion based on contact with new developers as well running the
> > > mentorship program, I would sat this isn't something that needs
> > > solving.
> >
> > As one data point, I cannot send emails with git send-email anymore.
> > It used to work, then broke and I don't know how to fix it. Now it says:
> >
> > 5.7.8 Username and Password not accepted. Learn more at
> > 5.7.8  https://support.google.com/mail/?p=BadCredentials
> > s10sm8376885wrr.5 - gsmtp
> >
> > I suspect it has something to do with two factor auth.
> > So that's it: it cannot contribute to kernel right now.
> > I will not consider time spent fixing it as useful time investment.
>
> Starting from an estalished working process, a change on your e-mail
> provider side broke your workflow. The exact same problem could happen
> regardless of how changes get submitted, a corporate HTTP proxy or
> firewall could also break HTTP-based submissions.

If the network is down or firewalled, that's one problem. But for
email we involve another third party. So it's not exactly as own app
server.

> Now, gmail being one of the largest e-mail providers, I think it's fair
> to consider that the kernel community should provide clear and easy to
> follow instructions on how to use git-send-email with gmail. In
> particular, with two-factor authentication being widespread, how to set
> it up with git-send-email should likely be described in
> https://www.kernel.org/doc/html/latest/process/email-clients.html.
> Blaming it solely on the SMTP protocol is a bit of a shortcut.
>
> > Any kernel documentation that I can find for gmail, mentions config
> > that I am already using and that is not working:
> > https://www.kernel.org/doc/html/latest/search.html?q=gmail&check_keywords=yes&area=default#
> > https://www.kernel.org/doc/html/latest/process/email-clients.html?highlight=gmail
> >
> > As another data point, I spoke to KP Singh at the Plumbers. He is a
> > "returning" kernel developer (so already did this before), he said it
> > took him 3 days and 52 configurations changes (all were committed to
> > git, so was possible to count exactly) to setup mail client properly.
> > And he is "staffed" to do kernel work, I would expect that most people
> > who don't _have_ to do kernel contributions will turn away half-way.
>
> That's very interesting information, is there any way that more details
> about the 52 steps could be shared ?

+KP you told me that it was hard to setup a mail client on Plumbers.
Did I get the numbers right? You have any details for that?

> > As another data point, several people told me that they are afraid of
> > sending kernel patches b/c there is so much "on you" to do right.
>
> Is that related to the submission mechanism, or to all the other things
> you need to get right ?

Email is part of this.

> > I would say that we need to aim at  a process that does not require a
> > friendly experienced person to answer any of your questions in the
> > common case. Lots of people will simply not ask any questions.
>
> I fully agree with that.
>
> --
> Regards,
>
> Laurent Pinchart

Re: RFE: use patchwork to submit a patch

[Eric Wong]

Dmitry Vyukov <dvyukov@google.com> wrote:
> As one data point, I cannot send emails with git send-email anymore.
> It used to work, then broke and I don't know how to fix it. Now it says:
> 
> 5.7.8 Username and Password not accepted. Learn more at
> 5.7.8  https://support.google.com/mail/?p=BadCredentials
> s10sm8376885wrr.5 - gsmtp
> 
> I suspect it has something to do with two factor auth.
> So that's it: it cannot contribute to kernel right now.
> I will not consider time spent fixing it as useful time investment.

I'm sorry you feel that way about time investments...
But I've always assumed that's also the sentiment for time spent
learning ANY new tools or workflow changes that come along.

> Any kernel documentation that I can find for gmail, mentions config
> that I am already using and that is not working:
> https://www.kernel.org/doc/html/latest/search.html?q=gmail&check_keywords=yes&area=default#
> https://www.kernel.org/doc/html/latest/process/email-clients.html?highlight=gmail

Fwiw, git-send-email(1) manpage also has a special section for gmail:

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

and a link for app-specific passwords:

  https://security.google.com/settings/security/apppasswords

Perhaps that helps?

Re: RFE: use patchwork to submit a patch

[Dmitry Vyukov]

On Tue, Oct 15, 2019 at 10:57 AM Eric Wong <e@80x24.org> wrote:
>
> Dmitry Vyukov <dvyukov@google.com> wrote:
> > As one data point, I cannot send emails with git send-email anymore.
> > It used to work, then broke and I don't know how to fix it. Now it says:
> >
> > 5.7.8 Username and Password not accepted. Learn more at
> > 5.7.8  https://support.google.com/mail/?p=BadCredentials
> > s10sm8376885wrr.5 - gsmtp
> >
> > I suspect it has something to do with two factor auth.
> > So that's it: it cannot contribute to kernel right now.
> > I will not consider time spent fixing it as useful time investment.
>
> I'm sorry you feel that way about time investments...
> But I've always assumed that's also the sentiment for time spent
> learning ANY new tools or workflow changes that come along.

This is true. But the fact that there is a learning curve to anything
does not justify any learning curve for everything. Some parts of
technology may be isolated completely and one does not need to learn
anything about that part. For example, today to compile a high-level
language one generally does not need to learn anything about machine
instructions. So the question is: is SMTP/IMAP is something that
inherently needs to be learned for contribution to kernel or it can be
hidden/not required/made simpler? And looking at github/facebook I
would assume that contributors do not have to be exposed to that.


> > Any kernel documentation that I can find for gmail, mentions config
> > that I am already using and that is not working:
> > https://www.kernel.org/doc/html/latest/search.html?q=gmail&check_keywords=yes&area=default#
> > https://www.kernel.org/doc/html/latest/process/email-clients.html?highlight=gmail
>
> Fwiw, git-send-email(1) manpage also has a special section for gmail:
>
>   https://kernel.org/pub/software/scm/git/docs/git-send-email.html
>
> and a link for app-specific passwords:
>
>   https://security.google.com/settings/security/apppasswords
>
> Perhaps that helps?

For me that page says "The setting you are looking for is not
available for your account".
I suspect app passwords work if 2-factor auth is enabled, but what
enabled on my account is "Use your phone to sign in", which is
different from 2-factor auth setting.

Re: RFE: use patchwork to submit a patch

[Steven Rostedt]

On Mon, 14 Oct 2019 16:56:58 -0400
"Theodore Y. Ts'o" <tytso@mit.edu> wrote:

> So it's an extreme case, and is an example of corp e-mail insanity.
> The good news is that for companies that are really serious about
> working with upstream, they will roll out solutions for their own
> customers.  For example, with IBM, since Lotus notes was maximally
> unfriendly to OSS development practices, they created a separate mail
> system with ldap@linux.ibm.com addresses be cause it was simply
> impossible to engage with upstream development using ibm.com
> addresses.  So there are work arounds, and I encourage you to reach
> out to other Google kernel developers --- and if you are unhappy,
> please file internal bugs against the corp infrastructure.

Note, one issue is that some corporations do not want to support two
email services. It's hard enough securing one, opening up another one
is just opening another door to be cracked.

We do file bugs when there's issues, but that doesn't mean they will be
resolved. I have special permission to use my personal account, but
most people at my company do not have that permission, and this is a
heart ache for those that need to send email via the corporate email
servers.

Now, if we had a way to send and receive upstream patches via a web
site, that would actually make things easier.

-- Steve

Re: RFE: use patchwork to submit a patch

[Theodore Y. Ts'o]

On Tue, Oct 15, 2019 at 08:37:41AM -0400, Steven Rostedt wrote:
> Now, if we had a way to send and receive upstream patches via a web
> site, that would actually make things easier.

Alas, for those corporations who choose to enable DMARC in hard-fail
mode (which includes google.com BTW, although thankfully, not
gmail.com) sending (from DMARC's perspective, "forging") e-mail from
someone at that particular domain won't work unless it comes from an
authorized e-mail server.

Of course, we do have a workaround for that, which is to use a
kernel.org address.

						- Ted

Re: RFE: use patchwork to submit a patch

[Steven Rostedt]

On Tue, 15 Oct 2019 09:35:27 -0400
"Theodore Y. Ts'o" <tytso@mit.edu> wrote:

> On Tue, Oct 15, 2019 at 08:37:41AM -0400, Steven Rostedt wrote:
> > Now, if we had a way to send and receive upstream patches via a web
> > site, that would actually make things easier.  
> 
> Alas, for those corporations who choose to enable DMARC in hard-fail
> mode (which includes google.com BTW, although thankfully, not
> gmail.com) sending (from DMARC's perspective, "forging") e-mail from
> someone at that particular domain won't work unless it comes from an
> authorized e-mail server.

I'm not talking about the web interface forging an email. I'm assuming
that we have a way to communicate with something other than email.
Although, when submitting to this interface, you can be subscribed to
receive an email from the interface with something that looks just like
a patch.

-- Steve

Re: RFE: use patchwork to submit a patch

[Konstantin Ryabitsev]

On Tue, Oct 15, 2019 at 09:35:27AM -0400, Theodore Y. Ts'o wrote:
>On Tue, Oct 15, 2019 at 08:37:41AM -0400, Steven Rostedt wrote:
>> Now, if we had a way to send and receive upstream patches via a web
>> site, that would actually make things easier.
>
>Alas, for those corporations who choose to enable DMARC in hard-fail
>mode (which includes google.com BTW, although thankfully, not
>gmail.com) sending (from DMARC's perspective, "forging") e-mail from
>someone at that particular domain won't work unless it comes from an
>authorized e-mail server.

This is not a huge deal for web services mailing patches, because they 
just need to specify the `From:` header in the body of the patch. From 
`man git-am`:

  "From: " and "Subject: " lines starting the body override the 
  respective commit author name and title values taken from the headers.

So, a web service emailing patches can have:

  From jdoe@webserviceaddr.example.org
  From: J.Doe <jdoe@webserviceaddr.example.org>
  Reply-to: J.Doe <jdoe@corpaddress.example.com>
  Cc: jdoe@corpaddress.example.com
  Subject: [PATCH] Fix foo in bar

  From: J.Doe <jdoe@corpaddress.example.com>

  Signed-off-by: J.Doe <jdoe@corpaddress.example.com>
  ---

This *should* be kosher with legal requirements in most companies:

- the email traverses the corporate server when the cc is received, so 
  there is a log of it in their legally required auto-cc inbox
- all replies will be sent to the corporate server
- when the patch is applied, the commit will be properly listing the 
  corporate address

>Of course, we do have a workaround for that, which is to use a
>kernel.org address.

Only a very small subset of people have these, and it may still be not 
allowed by individual company's legal departments even when this is 
available.

-K

Re: RFE: use patchwork to submit a patch

[Konstantin Ryabitsev]

On Sun, Oct 13, 2019 at 11:39:30PM +0000, Eric Wong wrote:
>> Well, this is the bit where I say that it may not be allowed by 
>> corporate
>> rules. I see this all the time in CAF/Android world where companies
>> *require* that all email goes through their SMTP server so that it can be
>> properly logged (often for legal reasons). And it is often equally required
>> that any code submissions come from person@corporate.com and not
>> person@free-email-provider.com for License/CLA reasons, so setting up a
>> webmail server is not a solution either.
>
>Aren't they still allowed to submit stuff via forges the same way
>they'd use a potential hacker-oriented webmail/SMTP/IMAP solution?

You're trying to apply logic to legal requirements. :) It's actually 
fine for someone to submit a Github pull request using their corporate 
"From:", because it's not an action that sends email from their 
corporate address. Email is considered official company correspondence 
and therefore must be preserved in case it needs to be turned over 
during lawsuits (for fact-finding).

>We can also find creative ways to subvert corporate policies:
>For example; if their policy specifically prevents outgoing SMTP,
>"git imap-send" could be used.

I'm generally against recommending solutions that can get people into 
legal trouble. If their company requires that all work correspondence is 
sent via the company's SMTP server, then we shouldn't be putting forth 
technical solutions defeating this.

-K

Re: RFE: use patchwork to submit a patch

[Konstantin Ryabitsev]

On Mon, Oct 14, 2019 at 06:33:33PM +0300, Laurent Pinchart wrote:
>If the goal is to work around SMTP-related technical issues, is a web UI
>really the best way to go ? Wouldn't it be better to do the same through
>a git push ? We could setup a git server that requires authentication,
>and implement a push-to-email bridge. The information that would need to
>be entered in a web UI could be put in a tag message, and we could have
>a CLI to create the tag from a list of questions.

Well, this is largely what GitGitGadget does 
(https://gitgitgadget.github.io), and we could go that route, sure. I'm 
reluctant only because, quoth:

  GitGitGadget itself is a GitHub App that is backed by an Azure 
  Function written in pure Javascript which in turn triggers an Azure 
  Pipeline written in Typescript (which is really easy to understand and 
  write for everybody who knows even just a little Javascript), 
  maintained at https://github.com/gitgitgadget/gitgitgadget.

I have zero familiarity with any of the above. That said, we do have a 
bunch of CI engineers working at the LF, and I can probably avail myself 
of their expertise if we decide to set this up.

-K

Re: RFE: use patchwork to submit a patch

[Greg KH]

On Tue, Oct 15, 2019 at 06:41:03AM +0200, Dmitry Vyukov wrote:
> On Mon, Oct 14, 2019 at 5:17 PM Greg KH <gregkh@linuxfoundation.org> wrote:
> > > As one data point, I cannot send emails with git send-email anymore.
> > > It used to work, then broke and I don't know how to fix it. Now it says:
> > >
> > > 5.7.8 Username and Password not accepted. Learn more at
> > > 5.7.8  https://support.google.com/mail/?p=BadCredentials
> > > s10sm8376885wrr.5 - gsmtp
> > >
> > > I suspect it has something to do with two factor auth.
> > > So that's it: it cannot contribute to kernel right now.
> >
> > That is because your employer changed how it manages imap.  So yes, this
> > configuration is now broken, you can not contribute to the kernel this
> > way.  They know about it, and there's an "opt-out" list you can sign up
> > for if you want to fix it.  Nothing the community can do about something
> > crazy like this.
> 
> I used my private gmail account to send emails. My private email
> account stopped working.

I think you need a better email provider :)

Re: RFE: use patchwork to submit a patch

[Konstantin Ryabitsev]

On Sat, Oct 12, 2019 at 09:19:11AM +0200, Greg KH wrote:
>> This is basically why SMTP sucks in my view -- and it's worthless 
>> trying to
>> pick fights with IT departments, because they are told to do so by lawyers.
>> So, I want to take SMTP out of the equation:
>>
>> 1. provide a way for someone to submit a patch using a web interface   (but
>> still in a way that From: is their corporate ID)
>
>If you do this, what happens when a maintainer/reviewer responds to that
>patch and says "looks good, but can you change X and resend it?"
>
>How will they get that message if it didn't go through their email
>system?  How will they be able to respond to it?

The magic of git and email headers. Only the initial submission will go 
out of this web service -- the follow-up is expected to arrive into 
their mailbox.

>> 2. use individual git feeds as a way to send out patches instead of   always
>> being secondary to SMTP
>
>Sending patches that way is one thing, the interaction based on those
>patches is another.
>
>Everyone needs to remember that only 1/3 of the patches submitted are
>applied.  The "normal" path of development is at least a review/resend
>cycle for submissions (2/3 of patches).  So that 2/3 can't be ignored as
>the "new/drive-by submissions" are probably more in that category than
>not.

Right, the idea is that the imaginary tool that is backed by 
public-inbox will use multiple sources of content:

- the mailing list repository
- individual developer feeds
- the person's imap folder

Anything that shows up in the individual feeds should have a 
corresponding entry in the mailing list repository, but the latter is 
also cryptographically signed and therefore end-to-end attestable.  
Mailing lists and SMTP continue to be the fallback delivery method for 
the vast majority of the content.

I expect to lay this out in more detail as I prepare the "kthul MVP" 
roadplan.

-K

Re: RFE: use patchwork to submit a patch

[Laurent Pinchart]

On Tue, Oct 15, 2019 at 11:11:57AM +0200, Dmitry Vyukov wrote:
> On Tue, Oct 15, 2019 at 10:57 AM Eric Wong wrote:
> > Dmitry Vyukov wrote:
> >> As one data point, I cannot send emails with git send-email anymore.
> >> It used to work, then broke and I don't know how to fix it. Now it says:
> >>
> >> 5.7.8 Username and Password not accepted. Learn more at
> >> 5.7.8  https://support.google.com/mail/?p=BadCredentials
> >> s10sm8376885wrr.5 - gsmtp
> >>
> >> I suspect it has something to do with two factor auth.
> >> So that's it: it cannot contribute to kernel right now.
> >> I will not consider time spent fixing it as useful time investment.
> >
> > I'm sorry you feel that way about time investments...
> > But I've always assumed that's also the sentiment for time spent
> > learning ANY new tools or workflow changes that come along.
> 
> This is true. But the fact that there is a learning curve to anything
> does not justify any learning curve for everything. Some parts of
> technology may be isolated completely and one does not need to learn
> anything about that part. For example, today to compile a high-level
> language one generally does not need to learn anything about machine
> instructions.

That's only true to some extent. Yes, you can develop in java without
knowing what a CPU is, but to develop efficient and safe software, a
knowledge of the whole stack is very useful, when not required.

> So the question is: is SMTP/IMAP is something that
> inherently needs to be learned for contribution to kernel or it can be
> hidden/not required/made simpler? And looking at github/facebook I
> would assume that contributors do not have to be exposed to that.

Could we develop a patch submission and review based on facebook if we
wanted to ? Most likely yes. Would we consider that as a good idea ?
Most likely no. There are always pros and cons in any workflow, and
while I could personally move away from e-mail, I would want a solution
that brings me most of the benefits of e-mail, in particular
decentralisation. git**b creates both a central point of failure and a
central point of trust, so it's a big no-go as far as I'm concerned. On
the other hand, creating a solution that enables optional use of forges
for contributors would prefer using them doesn't bother me at all, quite
the contrary.

> >> Any kernel documentation that I can find for gmail, mentions config
> >> that I am already using and that is not working:
> >> https://www.kernel.org/doc/html/latest/search.html?q=gmail&check_keywords=yes&area=default#
> >> https://www.kernel.org/doc/html/latest/process/email-clients.html?highlight=gmail
> >
> > Fwiw, git-send-email(1) manpage also has a special section for gmail:
> >
> >   https://kernel.org/pub/software/scm/git/docs/git-send-email.html
> >
> > and a link for app-specific passwords:
> >
> >   https://security.google.com/settings/security/apppasswords
> >
> > Perhaps that helps?
> 
> For me that page says "The setting you are looking for is not
> available for your account".
> I suspect app passwords work if 2-factor auth is enabled, but what
> enabled on my account is "Use your phone to sign in", which is
> different from 2-factor auth setting.

Did git-send-email start failing when enabling phone as a mean to sign
in, or was it unrelated ?

-- 
Regards,

Laurent Pinchart

Re: RFE: use patchwork to submit a patch

[Laurent Pinchart]

On Tue, Oct 15, 2019 at 11:11:57AM +0200, Dmitry Vyukov wrote:
> On Tue, Oct 15, 2019 at 10:57 AM Eric Wong wrote:
> > Dmitry Vyukov wrote:
> > > As one data point, I cannot send emails with git send-email anymore.
> > > It used to work, then broke and I don't know how to fix it. Now it says:
> > >
> > > 5.7.8 Username and Password not accepted. Learn more at
> > > 5.7.8  https://support.google.com/mail/?p=BadCredentials
> > > s10sm8376885wrr.5 - gsmtp
> > >
> > > I suspect it has something to do with two factor auth.
> > > So that's it: it cannot contribute to kernel right now.
> > > I will not consider time spent fixing it as useful time investment.
> >
> > I'm sorry you feel that way about time investments...
> > But I've always assumed that's also the sentiment for time spent
> > learning ANY new tools or workflow changes that come along.
> 
> This is true. But the fact that there is a learning curve to anything
> does not justify any learning curve for everything. Some parts of
> technology may be isolated completely and one does not need to learn
> anything about that part. For example, today to compile a high-level
> language one generally does not need to learn anything about machine
> instructions. So the question is: is SMTP/IMAP is something that
> inherently needs to be learned for contribution to kernel or it can be
> hidden/not required/made simpler? And looking at github/facebook I
> would assume that contributors do not have to be exposed to that.

I also forgot to mention one thing here. It can be argue that forges and
large e-mail providers such as gmail made the world a better place by
managing to hide SMTP and IMAP. But if you want to defend that point of
view, then it should be their responsibility to provide integration with
git-send-email. Golden handcuffs are, at the end of the day, just
handcuffs.

> > > Any kernel documentation that I can find for gmail, mentions config
> > > that I am already using and that is not working:
> > > https://www.kernel.org/doc/html/latest/search.html?q=gmail&check_keywords=yes&area=default#
> > > https://www.kernel.org/doc/html/latest/process/email-clients.html?highlight=gmail
> >
> > Fwiw, git-send-email(1) manpage also has a special section for gmail:
> >
> >   https://kernel.org/pub/software/scm/git/docs/git-send-email.html
> >
> > and a link for app-specific passwords:
> >
> >   https://security.google.com/settings/security/apppasswords
> >
> > Perhaps that helps?
> 
> For me that page says "The setting you are looking for is not
> available for your account".
> I suspect app passwords work if 2-factor auth is enabled, but what
> enabled on my account is "Use your phone to sign in", which is
> different from 2-factor auth setting.

-- 
Regards,

Laurent Pinchart

Re: RFE: use patchwork to submit a patch

[Laurent Pinchart]

Hi Dmitry,

On Tue, Oct 15, 2019 at 06:49:27AM +0200, Dmitry Vyukov wrote:
> On Mon, Oct 14, 2019 at 5:12 PM Laurent Pinchart wrote:
> > On Mon, Oct 14, 2019 at 04:58:17PM +0200, Dmitry Vyukov wrote:
> >> On Fri, Oct 11, 2019 at 7:20 PM Shuah Khan wrote:
> >>> On 10/11/19 2:57 AM, Greg KH wrote:
> >>>> On Thu, Oct 10, 2019 at 10:41:50AM -0400, Konstantin Ryabitsev wrote:
> >>>>> Hi, all:
> >>>>>
> >>>>> I would like to propose a new (large) feature to patchwork with the goal to
> >>>>> make the process of submitting a patch easier for newbies and people
> >>>>> generally less familiar with patch-based development. This was discussed
> >>>>> previously on the workflows list:
> >>>>> https://lore.kernel.org/workflows/20190930202451.GA14403@pure.paranoia.local/
> >>>>>
> >>>>> How I envision this would work:
> >>>>>
> >>>>> - user creates an account (which requires a mail confirmation) >> - they choose a "submit patch" option from the menu
> >>>>> - the patch submission screen has a succession of screens:
> >>>>>
> >>>>>   1. a screen with a single field allowing a user to paste a URL to     their
> >>>>> fork of the git repository. Once submitted, patchwork does a     "git
> >>>>> ls-remote" to attempt to get a list of refs and to verify that     this is
> >>>>> indeed a valid git repository
> >>>>
> >>>> s/valid git repository/valid git repository based on the kernel git tree/
> >>>>
> >>>> Otherwise you might be sending out lots of emails for other projects :)
> >>>>
> >>>>>   2. next screen asks the user to select the ref to work from using the
> >>>>> list obtained from the remote. Once submitted, patchwork performs a     `git
> >>>>> clone --reference` to clone the repository locally using a     local fork of
> >>>>> the same repo to minimize object transfer. This part     requires that:
> >>>>>        a. patchwork project is configured with a path to a local fork,
> >>>>> if this feature is enabled for a project
> >>>>>        b. that fork is kept current via some mechanism outside of
> >>>>> patchwork (e.g. with grokmirror)
> >>>>>        c. there is some sanity-checking during the clone process to
> >>>>> avoid abuse (e.g. a sane timeout, a tmpdir with limited size,          etc
> >>>>> -- other suggestions welcome)
> >>>>>
> >>>>>   3. next screen asks the user to pick a starting commit from the log.
> >>>>> Once submitted, patchwork generates the patch from the commit     provided
> >>>>> to the tip of the branch selected by the user earlier,
> >>>>>      using git format-patch.
> >>>>>
> >>>>>   4. next screen asks the user to review the patch to make sure this is
> >>>>> what they want to submit. Once confirmed, patchwork performs two
> >>>>> admin-defined optional hooks:
> >>>>>
> >>>>>        a. a hook to generate a list of cc's (e.g. get_maintainer.pl)
> >>>>>        b. a sanity check hook (e.g. checkpatch.pl)
> >>>>
> >>>> I will note that many "first patch" submissions are checkpatch.pl
> >>>> cleanups for staging.  When doing that, I require that they do "one
> >>>> logical change per patch", which means that many of the individual
> >>>> patches themselves will not be checkpatch.pl clean, because many lines
> >>>> have multiple issues with them (tabs, spaces, format, length, etc.)
> >>>>
> >>>> So other than that minor thing, sounds interesting.  It's hard to
> >>>> determine just how difficult the whole "set up git and send a patch out"
> >>>> process is for people these days given the _huge_ numbers of new
> >>>> contributions we keep getting, and the numerous good tutorials we have
> >>>> created that spell out exactly how to do this.
> >>>>
> >>>> So you might be "solving" a problem that we don't really have.  It's
> >>>> hard to tell :(
> >>>
> >>> I agree with this. I don't think this a problem that is worth solving.
> >>> When a new developer wants to send a patch, they don't need to create
> >>> any accounts. They setup their email client and send patch.
> >>>
> >>> We have several resources that walk them through setting up email
> >>> clients and sending patches. checkpatch.pl can be automated with
> >>> git hooks.
> >>>
> >>>>> I know this is a pretty big RFE, and I would like to hear your thoughts
> >>>>> about this. If there is general agreement that this is doable/good idea, I
> >>>>> may be able to come up with funding for this development as part of the
> >>>>> overall tooling improvement proposal.
> >>>>
> >>>> The workflow seems sane, and matches what most people do today, with the
> >>>> exception that it "solves" the git send-email issue, right?  Is that our
> >>>> biggest barrier?
> >>>>
> >>>> I would recommend interviewing some of the recent kernel mentor project
> >>>> and outreachy applicants first, to try to determine exactly what their
> >>>> problems, if any, were with our development process.  If they say that
> >>>> this type of tool/workflow would have saved them hours of time and
> >>>> energy, then that's a great indication that we should try to do this.
> >>>
> >>> I would say considering the number of applicants to mentorship program
> >>> and new developers it will be lot overhead to require them to create
> >>> patchwork accounts, and it might even be hard overtime. A lot of them
> >>> start out and drop out in the middle. With the current setup, nothing
> >>> to cleanup.
> >>>
> >>> Setting up email clients and git hooks is one time task. It is the
> >>> easiest of the learning curve for many new developers. New developers
> >>> struggle with getting the change logs right, coding styles right, and
> >>> responding to review comments and acting on them.
> >>>
> >>> These aren't something that can be automated and they just have to
> >>> learn through experience of sending patches.
> >>>
> >>> My opinion based on contact with new developers as well running the
> >>> mentorship program, I would sat this isn't something that needs
> >>> solving.
> >>
> >> As one data point, I cannot send emails with git send-email anymore.
> >> It used to work, then broke and I don't know how to fix it. Now it says:
> >>
> >> 5.7.8 Username and Password not accepted. Learn more at
> >> 5.7.8  https://support.google.com/mail/?p=BadCredentials
> >> s10sm8376885wrr.5 - gsmtp
> >>
> >> I suspect it has something to do with two factor auth.
> >> So that's it: it cannot contribute to kernel right now.
> >> I will not consider time spent fixing it as useful time investment.
> >
> > Starting from an estalished working process, a change on your e-mail
> > provider side broke your workflow. The exact same problem could happen
> > regardless of how changes get submitted, a corporate HTTP proxy or
> > firewall could also break HTTP-based submissions.
> 
> If the network is down or firewalled, that's one problem. But for
> email we involve another third party. So it's not exactly as own app
> server.

Network or firewall are most of the time third parties as well, and it's
up to you to pick providers that play fair. There's more to the world
than gmail, and if one prefers to give up freedom for some temporary
ease of use, then one shouldn't be surprised to end up with neither.

> > Now, gmail being one of the largest e-mail providers, I think it's fair
> > to consider that the kernel community should provide clear and easy to
> > follow instructions on how to use git-send-email with gmail. In
> > particular, with two-factor authentication being widespread, how to set
> > it up with git-send-email should likely be described in
> > https://www.kernel.org/doc/html/latest/process/email-clients.html.
> > Blaming it solely on the SMTP protocol is a bit of a shortcut.
> >
> >> Any kernel documentation that I can find for gmail, mentions config
> >> that I am already using and that is not working:
> >> https://www.kernel.org/doc/html/latest/search.html?q=gmail&check_keywords=yes&area=default#
> >> https://www.kernel.org/doc/html/latest/process/email-clients.html?highlight=gmail
> >>
> >> As another data point, I spoke to KP Singh at the Plumbers. He is a
> >> "returning" kernel developer (so already did this before), he said it
> >> took him 3 days and 52 configurations changes (all were committed to
> >> git, so was possible to count exactly) to setup mail client properly.
> >> And he is "staffed" to do kernel work, I would expect that most people
> >> who don't _have_ to do kernel contributions will turn away half-way.
> >
> > That's very interesting information, is there any way that more details
> > about the 52 steps could be shared ?
> 
> +KP you told me that it was hard to setup a mail client on Plumbers.
> Did I get the numbers right? You have any details for that?
> 
> >> As another data point, several people told me that they are afraid of
> >> sending kernel patches b/c there is so much "on you" to do right.
> >
> > Is that related to the submission mechanism, or to all the other things
> > you need to get right ?
> 
> Email is part of this.
> 
> >> I would say that we need to aim at  a process that does not require a
> >> friendly experienced person to answer any of your questions in the
> >> common case. Lots of people will simply not ask any questions.
> >
> > I fully agree with that.

-- 
Regards,

Laurent Pinchart

Re: RFE: use patchwork to submit a patch

[Laurent Pinchart]

Hi Konstantin,

On Tue, Oct 15, 2019 at 11:40:10AM -0400, Konstantin Ryabitsev wrote:
> On Mon, Oct 14, 2019 at 06:33:33PM +0300, Laurent Pinchart wrote:
> > If the goal is to work around SMTP-related technical issues, is a web UI
> > really the best way to go ? Wouldn't it be better to do the same through
> > a git push ? We could setup a git server that requires authentication,
> > and implement a push-to-email bridge. The information that would need to
> > be entered in a web UI could be put in a tag message, and we could have
> > a CLI to create the tag from a list of questions.
> 
> Well, this is largely what GitGitGadget does 
> (https://gitgitgadget.github.io), and we could go that route, sure. I'm 
> reluctant only because, quoth:
> 
>   GitGitGadget itself is a GitHub App that is backed by an Azure 
>   Function written in pure Javascript which in turn triggers an Azure 
>   Pipeline written in Typescript (which is really easy to understand and 
>   write for everybody who knows even just a little Javascript), 
>   maintained at https://github.com/gitgitgadget/gitgitgadget.
> 
> I have zero familiarity with any of the above. That said, we do have a 
> bunch of CI engineers working at the LF, and I can probably avail myself 
> of their expertise if we decide to set this up.

I certainly wouldn't recommend a solution based on a proprietary
closed-source stack :-) But as we're talking about performing new
development for patchwork, I wanted to point out that we could also
consider a different technical approach that would involve new
development for a different open-source project. For instance, is the
above idea something that could be developed on top of gitolite ? Or
possibly even as a tiny standalone git server ?

-- 
Regards,

Laurent Pinchart

Re: RFE: use patchwork to submit a patch

[Drew DeVault]

On Tue Oct 15, 2019 at 7:32 PM Laurent Pinchart wrote:
> I certainly wouldn't recommend a solution based on a proprietary
> closed-source stack :-) But as we're talking about performing new
> development for patchwork, I wanted to point out that we could also
> consider a different technical approach that would involve new
> development for a different open-source project. For instance, is the
> above idea something that could be developed on top of gitolite ? Or
> possibly even as a tiny standalone git server ?

Some of the work I'm doing with code review on sourcehut can be
generalized, to support writing up arbitrary mailing lists with
arbitrary review tools with a bidirectional workflow. I intend to make a
reality of this for at least lists.sr.ht <-> gitlab and lists.sr.ht <->
github, so that patchsets and the resulting discussion gets turned into
git{hub,lab} code reviews, and vise versa. I think it's possible to make
a system which presents both sides with an experience that is idiomatic
for each context, so that you can't really tell whose using which tool
because they're both speaking each other's conventional language.

Re: RFE: use patchwork to submit a patch

[Laurent Pinchart]

Hi Steven,

On Tue, Oct 15, 2019 at 08:37:41AM -0400, Steven Rostedt wrote:
> On Mon, 14 Oct 2019 16:56:58 -0400 "Theodore Y. Ts'o" wrote:
> 
> > So it's an extreme case, and is an example of corp e-mail insanity.
> > The good news is that for companies that are really serious about
> > working with upstream, they will roll out solutions for their own
> > customers.  For example, with IBM, since Lotus notes was maximally
> > unfriendly to OSS development practices, they created a separate mail
> > system with ldap@linux.ibm.com addresses be cause it was simply
> > impossible to engage with upstream development using ibm.com
> > addresses.  So there are work arounds, and I encourage you to reach
> > out to other Google kernel developers --- and if you are unhappy,
> > please file internal bugs against the corp infrastructure.
> 
> Note, one issue is that some corporations do not want to support two
> email services. It's hard enough securing one, opening up another one
> is just opening another door to be cracked.
> 
> We do file bugs when there's issues, but that doesn't mean they will be
> resolved. I have special permission to use my personal account, but
> most people at my company do not have that permission, and this is a
> heart ache for those that need to send email via the corporate email
> servers.

But how far could we push this reasoning ? I've worked for a company
whose corporate policy was that all source code had to be stored in SVN,
not exception. I didn't reach the community to move kernel development
away from git. I've also worked with people whose corporate policy was
that they had to do Linux kernel development on Windows, without using
any virtual machine. There are all kind of crazy corporate policies, and
if we don't push the pain it inflicts on all of us back to the people
who enact those crazy policies, we'll always lose.

> Now, if we had a way to send and receive upstream patches via a web
> site, that would actually make things easier.

-- 
Regards,

Laurent Pinchart

Re: RFE: use patchwork to submit a patch

[Laurent Pinchart]

Hi Drew,

On Tue, Oct 15, 2019 at 12:34:57PM -0400, Drew DeVault wrote:
> On Tue Oct 15, 2019 at 7:32 PM Laurent Pinchart wrote:
> > I certainly wouldn't recommend a solution based on a proprietary
> > closed-source stack :-) But as we're talking about performing new
> > development for patchwork, I wanted to point out that we could also
> > consider a different technical approach that would involve new
> > development for a different open-source project. For instance, is the
> > above idea something that could be developed on top of gitolite ? Or
> > possibly even as a tiny standalone git server ?
> 
> Some of the work I'm doing with code review on sourcehut can be
> generalized, to support writing up arbitrary mailing lists with
> arbitrary review tools with a bidirectional workflow. I intend to make a
> reality of this for at least lists.sr.ht <-> gitlab and lists.sr.ht <->
> github, so that patchsets and the resulting discussion gets turned into
> git{hub,lab} code reviews, and vise versa. I think it's possible to make
> a system which presents both sides with an experience that is idiomatic
> for each context, so that you can't really tell whose using which tool
> because they're both speaking each other's conventional language.

First of all, please let me thank you for all the effort you're putting
into this.

I'm not very confident that perfect transparent e-mail bridges could be
developed, and the culprit here it e-mail. From forge to e-mail there's
no real issue, we have structure data, and we write it out in text form.
The other way around, though, involves recovering structure from text.
If the MTAs, MDAs, MUAs and, quite importanttly, users, behave
correctly, that's doable. We can handle some of the "features" of common
M*A, but if someone decides to throw the formatting through the window
completely, we'll be screwed.

An idea I was toying with in the past was to create a structured format
for review, that would match what we use now in e-mail very closely, but
with a clearly defined syntax and grammar. The format would appear as
just plain e-mails to users, and a MUA that behaves reasonably would
likely produce valid structured data as e-mail replies. Over time we
could develop clients or teach some MUAs about the structured format,
removing the risk that they, or the end users, would mess up a reply.
The migration would be mostly transparent as it wouldn't really impact
existing users of e-mail, and could thus be rolled out over a long
period of time.

-- 
Regards,

Laurent Pinchart

Re: RFE: use patchwork to submit a patch

[Steven Rostedt]

On Tue, 15 Oct 2019 19:37:04 +0300
Laurent Pinchart <laurent.pinchart@ideasonboard.com> wrote:

> But how far could we push this reasoning ? I've worked for a company
> whose corporate policy was that all source code had to be stored in SVN,
> not exception. I didn't reach the community to move kernel development
> away from git. I've also worked with people whose corporate policy was
> that they had to do Linux kernel development on Windows, without using
> any virtual machine. There are all kind of crazy corporate policies, and
> if we don't push the pain it inflicts on all of us back to the people
> who enact those crazy policies, we'll always lose.

I understand your sentiment. But most of your examples are one-off
"crazy corporate policies". The "sucky email server" and "you must use
this sucky email server" is rather standard. Not saying that we don't
want to pressure them to change, but I'm hearing from people (like Dave
Miller), that new developers are moving away from email for
development.

I thought part of this thread was talking about having other ways
besides email to submit a patch. Something that can help people
correctly send a patch (at least their formatting) by making sure they
fill out the proper fields and have a tool that helps them do so. I
brought up the corporate email servers just as an example of having
this help out there too.

I plan on continuing to develop mostly in email (I still send my patch
series via quilt!). But I'm not going to enforce everyone to continue
to use email if we can come up with a better way. I also want to make
sure that whatever we do come up with will still support email.

-- Steve

Re: RFE: use patchwork to submit a patch

[Drew DeVault]

On Tue Oct 15, 2019 at 7:44 PM Laurent Pinchart wrote:
> I'm not very confident that perfect transparent e-mail bridges could be
> developed, and the culprit here it e-mail. From forge to e-mail there's
> no real issue, we have structure data, and we write it out in text form.
> The other way around, though, involves recovering structure from text.
> If the MTAs, MDAs, MUAs and, quite importanttly, users, behave
> correctly, that's doable. We can handle some of the "features" of common
> M*A, but if someone decides to throw the formatting through the window
> completely, we'll be screwed.

Well, no, it can never be perfect. But I'm not trying to accomodate for,
as an example, someone who's deliberately trying to confusing the
system. It will probably always be based on heuristics in some degrees.

However, I think that social conformance is a force to be reckoned with.
Most people don't have to be taught how code review works on mailing
lists, they just watch others do it and naturally fall in line. Without
any formal structure at all, we've already more or less settled on a
shared set of patterns for email-driven code review. However, I think
that the following:

> An idea I was toying with in the past was to create a structured format
> for review, that would match what we use now in e-mail very closely, but
> with a clearly defined syntax and grammar. The format would appear as
> just plain e-mails to users, and a MUA that behaves reasonably would
> likely produce valid structured data as e-mail replies. Over time we
> could develop clients or teach some MUAs about the structured format,
> removing the risk that they, or the end users, would mess up a reply.
> The migration would be mostly transparent as it wouldn't really impact
> existing users of e-mail, and could thus be rolled out over a long
> period of time.

Is very reasonable. If we can come up with a standard which feels like
what we're already doing, but is written down, then tools have something
to conform to and users have some expectations for how to behave to make
their tools happy.

I would be up for putting up some kind of simple spec for bikeshedding
purposes, and teaching it to the thread parser that was built for
lists.sr.ht:

https://git.sr.ht/~emersion/python-emailthreads

Re: RFE: use patchwork to submit a patch

[Konstantin Ryabitsev]

On Tue, Oct 15, 2019 at 07:32:41PM +0300, Laurent Pinchart wrote:
>> Well, this is largely what GitGitGadget does
>> (https://gitgitgadget.github.io), and we could go that route, sure. I'm
>> reluctant only because, quoth:
>>
>>   GitGitGadget itself is a GitHub App that is backed by an Azure
>>   Function written in pure Javascript which in turn triggers an Azure
>>   Pipeline written in Typescript (which is really easy to understand and
>>   write for everybody who knows even just a little Javascript),
>>   maintained at https://github.com/gitgitgadget/gitgitgadget.
>>
>> I have zero familiarity with any of the above. That said, we do have a
>> bunch of CI engineers working at the LF, and I can probably avail myself
>> of their expertise if we decide to set this up.
>
>I certainly wouldn't recommend a solution based on a proprietary
>closed-source stack :-) But as we're talking about performing new
>development for patchwork, I wanted to point out that we could also
>consider a different technical approach that would involve new
>development for a different open-source project. For instance, is the
>above idea something that could be developed on top of gitolite ? Or
>possibly even as a tiny standalone git server ?

I wouldn't do it on top of gitolite, because the administrative overhead 
would be too large. It certainly can be done as a separate service -- 
after all, any relation between this tool and patchwork is pretty 
tenuous.

That said, I'm leaning towards shelving this idea for the moment, at 
least as an official service provided by kernel.org -- it is easier to 
limit the scope at first and target maintainer tools and communication 
frameworks. It seems that sr.ht folks are already making something like 
this available, so we can just sit on our hands for a bit and see where 
this takes us.

-K

Re: RFE: use patchwork to submit a patch

[Dmitry Vyukov]

On Mon, Oct 14, 2019 at 4:58 PM Dmitry Vyukov <dvyukov@google.com> wrote:
>
> On Fri, Oct 11, 2019 at 7:20 PM Shuah Khan <skhan@linuxfoundation.org> wrote:
> >
> > On 10/11/19 2:57 AM, Greg KH wrote:
> > > On Thu, Oct 10, 2019 at 10:41:50AM -0400, Konstantin Ryabitsev wrote:
> > >> Hi, all:
> > >>
> > >> I would like to propose a new (large) feature to patchwork with the goal to
> > >> make the process of submitting a patch easier for newbies and people
> > >> generally less familiar with patch-based development. This was discussed
> > >> previously on the workflows list:
> > >> https://lore.kernel.org/workflows/20190930202451.GA14403@pure.paranoia.local/
> > >>
> > >> How I envision this would work:
> > >>
> > >> - user creates an account (which requires a mail confirmation) >> - they choose a "submit patch" option from the menu
> > >> - the patch submission screen has a succession of screens:
> > >>
> > >>   1. a screen with a single field allowing a user to paste a URL to     their
> > >> fork of the git repository. Once submitted, patchwork does a     "git
> > >> ls-remote" to attempt to get a list of refs and to verify that     this is
> > >> indeed a valid git repository
> > >
> > > s/valid git repository/valid git repository based on the kernel git tree/
> > >
> > > Otherwise you might be sending out lots of emails for other projects :)
> > >
> > >>
> > >>   2. next screen asks the user to select the ref to work from using the
> > >> list obtained from the remote. Once submitted, patchwork performs a     `git
> > >> clone --reference` to clone the repository locally using a     local fork of
> > >> the same repo to minimize object transfer. This part     requires that:
> > >>        a. patchwork project is configured with a path to a local fork,
> > >> if this feature is enabled for a project
> > >>        b. that fork is kept current via some mechanism outside of
> > >> patchwork (e.g. with grokmirror)
> > >>        c. there is some sanity-checking during the clone process to
> > >> avoid abuse (e.g. a sane timeout, a tmpdir with limited size,          etc
> > >> -- other suggestions welcome)
> > >>
> > >>   3. next screen asks the user to pick a starting commit from the log.
> > >> Once submitted, patchwork generates the patch from the commit     provided
> > >> to the tip of the branch selected by the user earlier,
> > >>      using git format-patch.
> > >>
> > >>   4. next screen asks the user to review the patch to make sure this is
> > >> what they want to submit. Once confirmed, patchwork performs two
> > >> admin-defined optional hooks:
> > >>
> > >>        a. a hook to generate a list of cc's (e.g. get_maintainer.pl)
> > >>        b. a sanity check hook (e.g. checkpatch.pl)
> > >
> > > I will note that many "first patch" submissions are checkpatch.pl
> > > cleanups for staging.  When doing that, I require that they do "one
> > > logical change per patch", which means that many of the individual
> > > patches themselves will not be checkpatch.pl clean, because many lines
> > > have multiple issues with them (tabs, spaces, format, length, etc.)
> > >
> > > So other than that minor thing, sounds interesting.  It's hard to
> > > determine just how difficult the whole "set up git and send a patch out"
> > > process is for people these days given the _huge_ numbers of new
> > > contributions we keep getting, and the numerous good tutorials we have
> > > created that spell out exactly how to do this.
> > >
> > > So you might be "solving" a problem that we don't really have.  It's
> > > hard to tell :(
> > >
> >
> > I agree with this. I don't think this a problem that is worth solving.
> > When a new developer wants to send a patch, they don't need to create
> > any accounts. They setup their email client and send patch.
> >
> > We have several resources that walk them through setting up email
> > clients and sending patches. checkpatch.pl can be automated with
> > git hooks.
> >
> > >> I know this is a pretty big RFE, and I would like to hear your thoughts
> > >> about this. If there is general agreement that this is doable/good idea, I
> > >> may be able to come up with funding for this development as part of the
> > >> overall tooling improvement proposal.
> > >
> > > The workflow seems sane, and matches what most people do today, with the
> > > exception that it "solves" the git send-email issue, right?  Is that our
> > > biggest barrier?
> > >
> > > I would recommend interviewing some of the recent kernel mentor project
> > > and outreachy applicants first, to try to determine exactly what their
> > > problems, if any, were with our development process.  If they say that
> > > this type of tool/workflow would have saved them hours of time and
> > > energy, then that's a great indication that we should try to do this.
> > >
> >
> > I would say considering the number of applicants to mentorship program
> > and new developers it will be lot overhead to require them to create
> > patchwork accounts, and it might even be hard overtime. A lot of them
> > start out and drop out in the middle. With the current setup, nothing
> > to cleanup.
> >
> > Setting up email clients and git hooks is one time task. It is the
> > easiest of the learning curve for many new developers. New developers
> > struggle with getting the change logs right, coding styles right, and
> > responding to review comments and acting on them.
> >
> > These aren't something that can be automated and they just have to
> > learn through experience of sending patches.
> >
> > My opinion based on contact with new developers as well running the
> > mentorship program, I would sat this isn't something that needs
> > solving.
> >
> > thanks,
> > -- Shuah
>
>
> As one data point, I cannot send emails with git send-email anymore.
> It used to work, then broke and I don't know how to fix it. Now it says:
>
> 5.7.8 Username and Password not accepted. Learn more at
> 5.7.8  https://support.google.com/mail/?p=BadCredentials
> s10sm8376885wrr.5 - gsmtp
>
> I suspect it has something to do with two factor auth.
> So that's it: it cannot contribute to kernel right now.
> I will not consider time spent fixing it as useful time investment.
>
> Any kernel documentation that I can find for gmail, mentions config
> that I am already using and that is not working:
> https://www.kernel.org/doc/html/latest/search.html?q=gmail&check_keywords=yes&area=default#
> https://www.kernel.org/doc/html/latest/process/email-clients.html?highlight=gmail
>
> As another data point, I spoke to KP Singh at the Plumbers. He is a
> "returning" kernel developer (so already did this before), he said it
> took him 3 days and 52 configurations changes (all were committed to
> git, so was possible to count exactly) to setup mail client properly.
> And he is "staffed" to do kernel work, I would expect that most people
> who don't _have_ to do kernel contributions will turn away half-way.
>
> As another data point, several people told me that they are afraid of
> sending kernel patches b/c there is so much "on you" to do right.
>
> I would say that we need to aim at  a process that does not require a
> friendly experienced person to answer any of your questions in the
> common case. Lots of people will simply not ask any questions.

Just arrived: lost emails and patches:
https://groups.google.com/d/msg/syzkaller-bugs/GSD7eiEJm5M/XyKMJ_v8BAAJ

Re: RFE: use patchwork to submit a patch

[Laurent Pinchart]

Hi Steven,

On Tue, Oct 15, 2019 at 12:47:49PM -0400, Steven Rostedt wrote:
> On Tue, 15 Oct 2019 19:37:04 +0300 Laurent Pinchart wrote:
> 
> > But how far could we push this reasoning ? I've worked for a company
> > whose corporate policy was that all source code had to be stored in SVN,
> > not exception. I didn't reach the community to move kernel development
> > away from git. I've also worked with people whose corporate policy was
> > that they had to do Linux kernel development on Windows, without using
> > any virtual machine. There are all kind of crazy corporate policies, and
> > if we don't push the pain it inflicts on all of us back to the people
> > who enact those crazy policies, we'll always lose.
> 
> I understand your sentiment. But most of your examples are one-off
> "crazy corporate policies". The "sucky email server" and "you must use
> this sucky email server" is rather standard. Not saying that we don't
> want to pressure them to change, but I'm hearing from people (like Dave
> Miller), that new developers are moving away from email for
> development.

I think that's two different issues though, which certainly overlap (but
I'm not sure to what extent). I'm also not sure if we can say that new
developers are moving away from e-mail, or simply start development with
a different, non e-mail-based process. It makes a bit of a difference,
because some new developers may not have any issue with e-mail, they may
just not have considered it. This of course doesn't mean that nobody has
issues with e-mail, that nobody is moving away from e-mail, or that
nobody would have trouble moving to e-mail.

> I thought part of this thread was talking about having other ways
> besides email to submit a patch. Something that can help people
> correctly send a patch (at least their formatting) by making sure they
> fill out the proper fields and have a tool that helps them do so. I
> brought up the corporate email servers just as an example of having
> this help out there too.

I'm very biased here, but for me a CLI tool that asks me questions and
generates the patch series (or directly sends it) would be easier to
use than a web UI. That's because I would stay on the CLI that I'm
already using for git. This of course assumes a working git-send-email
configuration, which we all no is not a given. I thus wonder if it
wouldn't be better to have a CLI helper to create the patches, and
provide some sort of https-to-smtp service (possibly in the form of
pushing a tag with a message for instance). There are many options
possible to work around broken e-mail workflows, so instead of focussing
on a web UI because it was the first idea that was proposed, let's make
sure we consider other ones. And maybe we'll decide that a web UI is the
best option.

> I plan on continuing to develop mostly in email (I still send my patch
> series via quilt!). But I'm not going to enforce everyone to continue
> to use email if we can come up with a better way. I also want to make
> sure that whatever we do come up with will still support email.

Hypothetically speaking, if there was a service that allowed sending a
patch series through a git push with a tag containing a cover letter in
its message, and that service would send out e-mails to mailing lists
(with the option to self-host it if you want), would you consider using
it ?

-- 
Regards,

Laurent Pinchart

Re: RFE: use patchwork to submit a patch

[Steven Rostedt]

On Mon, 21 Oct 2019 18:39:18 +0300
Laurent Pinchart <laurent.pinchart@ideasonboard.com> wrote:

> > I plan on continuing to develop mostly in email (I still send my patch
> > series via quilt!). But I'm not going to enforce everyone to continue
> > to use email if we can come up with a better way. I also want to make
> > sure that whatever we do come up with will still support email.  
> 
> Hypothetically speaking, if there was a service that allowed sending a
> patch series through a git push with a tag containing a cover letter in
> its message, and that service would send out e-mails to mailing lists
> (with the option to self-host it if you want), would you consider using
> it ?

Only if I was forced to ;-)

Seriously, it's hard to teach an old dog new tricks. I wont change
until it becomes necessary to do so. As the old saying goes: "Don't fix
what ain't broke!"

I've developed a workflow over the past decade or so, that I've
optimized to fit my needs. Why would I want to change that? Unless
there's something that changes that can make an impact on my
efficiency. I've created a bunch of scripts that do exactly what I
expect them to. To change, I'm going back into the world of the unknown,
and will need to take up time (which I'm struggling to come up with),
to learn something new that may or may not make it better for me.

The tool I would like is simply something where all the patches I
receive are stored in a nice formatted database that I can process
offline somewhere (like when I'm flying to Europe).

Right now I hacked an old version of patchwork (when I upgraded,
everything broke, and I had to revert it), along with procmail, that
creates a local patchwork queue I can access locally. Unfortunately,
this doesn't work when I'm away from home without an internet
connection.

I would love to be able to take this database with me and process the
patches while traveling without connectivity.

-- Steve

Re: RFE: use patchwork to submit a patch

[Dmitry Vyukov]

On Thu, Oct 24, 2019 at 3:15 PM Steven Rostedt <rostedt@goodmis.org> wrote:
>
> On Mon, 21 Oct 2019 18:39:18 +0300
> Laurent Pinchart <laurent.pinchart@ideasonboard.com> wrote:
>
> > > I plan on continuing to develop mostly in email (I still send my patch
> > > series via quilt!). But I'm not going to enforce everyone to continue
> > > to use email if we can come up with a better way. I also want to make
> > > sure that whatever we do come up with will still support email.
> >
> > Hypothetically speaking, if there was a service that allowed sending a
> > patch series through a git push with a tag containing a cover letter in
> > its message, and that service would send out e-mails to mailing lists
> > (with the option to self-host it if you want), would you consider using
> > it ?
>
> Only if I was forced to ;-)
>
> Seriously, it's hard to teach an old dog new tricks. I wont change
> until it becomes necessary to do so. As the old saying goes: "Don't fix
> what ain't broke!"
>
> I've developed a workflow over the past decade or so, that I've
> optimized to fit my needs. Why would I want to change that? Unless
> there's something that changes that can make an impact on my
> efficiency.


Purely theoretically let's consider that the changes do not improve
_your_ efficiency, but they significantly improve overall project
efficiency by positively affecting people who did not develop a
workflow over the past decades (maybe there were not around 2 decades
ago) and positively affecting various tooling that _you_ may be
directly interested in, but otherwise they are important for the
project overall. So for you it's no change in efficiency except that
you now need to do things differently. What do you think about such
changes? Are you ready to force yourself? :)
I think it's quite cornerstone question here. All (?) major figures in
the kernel (who are ~~98% of decision making, but ~~2% of kernel
developers overall) have developed workflows over the past decades
that work reasonably well for them. If they veto all proposed changes
based on the criteria you described, every new contributor will need
decades to develop own workflows to become an efficient contributor
and lots of tooling will be painfully hard to do.


>I've created a bunch of scripts that do exactly what I
> expect them to. To change, I'm going back into the world of the unknown,
> and will need to take up time (which I'm struggling to come up with),
> to learn something new that may or may not make it better for me.
>
> The tool I would like is simply something where all the patches I
> receive are stored in a nice formatted database that I can process
> offline somewhere (like when I'm flying to Europe).
>
> Right now I hacked an old version of patchwork (when I upgraded,
> everything broke, and I had to revert it), along with procmail, that
> creates a local patchwork queue I can access locally. Unfortunately,
> this doesn't work when I'm away from home without an internet
> connection.
>
> I would love to be able to take this database with me and process the
> patches while traveling without connectivity.
>
> -- Steve

Re: RFE: use patchwork to submit a patch

[Steven Rostedt]

On Thu, 24 Oct 2019 15:33:04 +0200
Dmitry Vyukov <dvyukov@google.com> wrote:

> On Thu, Oct 24, 2019 at 3:15 PM Steven Rostedt <rostedt@goodmis.org> wrote:
> >
> > On Mon, 21 Oct 2019 18:39:18 +0300
> > Laurent Pinchart <laurent.pinchart@ideasonboard.com> wrote:


> 
> Purely theoretically let's consider that the changes do not improve
> _your_ efficiency, but they significantly improve overall project
> efficiency by positively affecting people who did not develop a
> workflow over the past decades (maybe there were not around 2 decades
> ago) and positively affecting various tooling that _you_ may be
> directly interested in, but otherwise they are important for the
> project overall. So for you it's no change in efficiency except that
> you now need to do things differently. What do you think about such
> changes? Are you ready to force yourself? :)
> I think it's quite cornerstone question here. All (?) major figures in
> the kernel (who are ~~98% of decision making, but ~~2% of kernel
> developers overall) have developed workflows over the past decades
> that work reasonably well for them. If they veto all proposed changes
> based on the criteria you described, every new contributor will need
> decades to develop own workflows to become an efficient contributor
> and lots of tooling will be painfully hard to do.
>

The above sound like a one size fits all approach, which I would caste
a veto to. I would like a solution that works for multiple workflows.
One where mine and others still work too.

Please, lets work on a infrastructure that is robust and flexible, that
is split into back and front ends. That way, we have a single "back
end" and multiple front ends that suite everyone's needs.

-- Steve

Re: RFE: use patchwork to submit a patch

[Dmitry Vyukov]

On Thu, Oct 24, 2019 at 3:58 PM Steven Rostedt <rostedt@goodmis.org> wrote:
>
> On Thu, 24 Oct 2019 15:33:04 +0200
> Dmitry Vyukov <dvyukov@google.com> wrote:
>
> > On Thu, Oct 24, 2019 at 3:15 PM Steven Rostedt <rostedt@goodmis.org> wrote:
> > >
> > > On Mon, 21 Oct 2019 18:39:18 +0300
> > > Laurent Pinchart <laurent.pinchart@ideasonboard.com> wrote:
> >
> > Purely theoretically let's consider that the changes do not improve
> > _your_ efficiency, but they significantly improve overall project
> > efficiency by positively affecting people who did not develop a
> > workflow over the past decades (maybe there were not around 2 decades
> > ago) and positively affecting various tooling that _you_ may be
> > directly interested in, but otherwise they are important for the
> > project overall. So for you it's no change in efficiency except that
> > you now need to do things differently. What do you think about such
> > changes? Are you ready to force yourself? :)
> > I think it's quite cornerstone question here. All (?) major figures in
> > the kernel (who are ~~98% of decision making, but ~~2% of kernel
> > developers overall) have developed workflows over the past decades
> > that work reasonably well for them. If they veto all proposed changes
> > based on the criteria you described, every new contributor will need
> > decades to develop own workflows to become an efficient contributor
> > and lots of tooling will be painfully hard to do.
> >
>
> The above sound like a one size fits all approach, which I would caste
> a veto to. I would like a solution that works for multiple workflows.
> One where mine and others still work too.
>
> Please, lets work on a infrastructure that is robust and flexible, that
> is split into back and front ends. That way, we have a single "back
> end" and multiple front ends that suite everyone's needs.
>
> -- Steve

I agree that the system must not lock in into a single UI.
However, (1) somebody will still need to glue all old frontends with
the new backend. (2) some workflows may still break regardless of a
perfect backend/frontend split. Consider, previously one could say in
arbitrary plain English that a bug X is fixed by commit Y, but if we
want to build anything on top of that can't be plain English anymore
and will require changes in workflows.

FWIW, Gerrit satisfies most of the criteria. It provides a nicely
formatted database that you can easily git fetch and take with you
offline. In this sense it is a robust and flexible backend. However,
one won't get all of hundreds of existing scripts and systems work on
top of it out-of-the-box.

Re: RFE: use patchwork to submit a patch

[Dmitry Vyukov]

On Mon, Oct 14, 2019 at 4:58 PM Dmitry Vyukov <dvyukov@google.com> wrote:
> > On 10/11/19 2:57 AM, Greg KH wrote:
> > > On Thu, Oct 10, 2019 at 10:41:50AM -0400, Konstantin Ryabitsev wrote:
> > >> Hi, all:
> > >>
> > >> I would like to propose a new (large) feature to patchwork with the goal to
> > >> make the process of submitting a patch easier for newbies and people
> > >> generally less familiar with patch-based development. This was discussed
> > >> previously on the workflows list:
> > >> https://lore.kernel.org/workflows/20190930202451.GA14403@pure.paranoia.local/
> > >>
> > >> How I envision this would work:
> > >>
> > >> - user creates an account (which requires a mail confirmation) >> - they choose a "submit patch" option from the menu
> > >> - the patch submission screen has a succession of screens:
> > >>
> > >>   1. a screen with a single field allowing a user to paste a URL to     their
> > >> fork of the git repository. Once submitted, patchwork does a     "git
> > >> ls-remote" to attempt to get a list of refs and to verify that     this is
> > >> indeed a valid git repository
> > >
> > > s/valid git repository/valid git repository based on the kernel git tree/
> > >
> > > Otherwise you might be sending out lots of emails for other projects :)
> > >
> > >>
> > >>   2. next screen asks the user to select the ref to work from using the
> > >> list obtained from the remote. Once submitted, patchwork performs a     `git
> > >> clone --reference` to clone the repository locally using a     local fork of
> > >> the same repo to minimize object transfer. This part     requires that:
> > >>        a. patchwork project is configured with a path to a local fork,
> > >> if this feature is enabled for a project
> > >>        b. that fork is kept current via some mechanism outside of
> > >> patchwork (e.g. with grokmirror)
> > >>        c. there is some sanity-checking during the clone process to
> > >> avoid abuse (e.g. a sane timeout, a tmpdir with limited size,          etc
> > >> -- other suggestions welcome)
> > >>
> > >>   3. next screen asks the user to pick a starting commit from the log.
> > >> Once submitted, patchwork generates the patch from the commit     provided
> > >> to the tip of the branch selected by the user earlier,
> > >>      using git format-patch.
> > >>
> > >>   4. next screen asks the user to review the patch to make sure this is
> > >> what they want to submit. Once confirmed, patchwork performs two
> > >> admin-defined optional hooks:
> > >>
> > >>        a. a hook to generate a list of cc's (e.g. get_maintainer.pl)
> > >>        b. a sanity check hook (e.g. checkpatch.pl)
> > >
> > > I will note that many "first patch" submissions are checkpatch.pl
> > > cleanups for staging.  When doing that, I require that they do "one
> > > logical change per patch", which means that many of the individual
> > > patches themselves will not be checkpatch.pl clean, because many lines
> > > have multiple issues with them (tabs, spaces, format, length, etc.)
> > >
> > > So other than that minor thing, sounds interesting.  It's hard to
> > > determine just how difficult the whole "set up git and send a patch out"
> > > process is for people these days given the _huge_ numbers of new
> > > contributions we keep getting, and the numerous good tutorials we have
> > > created that spell out exactly how to do this.
> > >
> > > So you might be "solving" a problem that we don't really have.  It's
> > > hard to tell :(
> > >
> >
> > I agree with this. I don't think this a problem that is worth solving.
> > When a new developer wants to send a patch, they don't need to create
> > any accounts. They setup their email client and send patch.
> >
> > We have several resources that walk them through setting up email
> > clients and sending patches. checkpatch.pl can be automated with
> > git hooks.
> >
> > >> I know this is a pretty big RFE, and I would like to hear your thoughts
> > >> about this. If there is general agreement that this is doable/good idea, I
> > >> may be able to come up with funding for this development as part of the
> > >> overall tooling improvement proposal.
> > >
> > > The workflow seems sane, and matches what most people do today, with the
> > > exception that it "solves" the git send-email issue, right?  Is that our
> > > biggest barrier?
> > >
> > > I would recommend interviewing some of the recent kernel mentor project
> > > and outreachy applicants first, to try to determine exactly what their
> > > problems, if any, were with our development process.  If they say that
> > > this type of tool/workflow would have saved them hours of time and
> > > energy, then that's a great indication that we should try to do this.
> > >
> >
> > I would say considering the number of applicants to mentorship program
> > and new developers it will be lot overhead to require them to create
> > patchwork accounts, and it might even be hard overtime. A lot of them
> > start out and drop out in the middle. With the current setup, nothing
> > to cleanup.
> >
> > Setting up email clients and git hooks is one time task. It is the
> > easiest of the learning curve for many new developers. New developers
> > struggle with getting the change logs right, coding styles right, and
> > responding to review comments and acting on them.
> >
> > These aren't something that can be automated and they just have to
> > learn through experience of sending patches.
> >
> > My opinion based on contact with new developers as well running the
> > mentorship program, I would sat this isn't something that needs
> > solving.

As yet another data point re emails.
Shuah, you are aware that you can't send proper emails either? ;)

syzbot asks to provide fixing commit in a particular format (and on 1
line, otherwise not possible to parse back).

It sent this yesterday:
https://groups.google.com/forum/message/raw?msg=syzkaller-bugs/IZZUu-BobtI/xAkxm6H4EQAJ
-------------------
Content-Type: text/plain; charset="UTF-8"; format=flowed; delsp=yes

If the result looks correct, please mark the bug fixed by replying with:
#syz fix: usb: usbip: Fix BUG: KASAN: slab-out-of-bounds in
vhci_hub_control()
-------------------

(Not sure you will be able to see what exactly I am sending after my
email client, so resort to the link for exact content :))

Which is still presumably correct: "format=flowed; delsp=yes" + 2
spaces before line wrap.

However, what you replied is corrupted:
https://groups.google.com/forum/message/raw?msg=syzkaller-bugs/IZZUu-BobtI/xAkxm6H4EQAJ
-------------------
#syz fix: usb: usbip: Fix BUG: KASAN: slab-out-of-bounds in
vhci_hub_control()
-------------------

Which means that the bug is fixed with "usb: usbip: Fix BUG: KASAN:
slab-out-of-bounds in" (last part is omitted). And this is what syzbot
has recorded:
https://syzkaller.appspot.com/bug?extid=600b03e0cf1b73bb23c4

Such commit never existed, so syzbot will get back to you in future
suggesting that the commit you provided is incorrect and asking to
provide a correct one.

That's what happens when you try to build processes around email...



> > thanks,
> > -- Shuah
>
>
> As one data point, I cannot send emails with git send-email anymore.
> It used to work, then broke and I don't know how to fix it. Now it says:
>
> 5.7.8 Username and Password not accepted. Learn more at
> 5.7.8  https://support.google.com/mail/?p=BadCredentials
> s10sm8376885wrr.5 - gsmtp
>
> I suspect it has something to do with two factor auth.
> So that's it: it cannot contribute to kernel right now.
> I will not consider time spent fixing it as useful time investment.
>
> Any kernel documentation that I can find for gmail, mentions config
> that I am already using and that is not working:
> https://www.kernel.org/doc/html/latest/search.html?q=gmail&check_keywords=yes&area=default#
> https://www.kernel.org/doc/html/latest/process/email-clients.html?highlight=gmail
>
> As another data point, I spoke to KP Singh at the Plumbers. He is a
> "returning" kernel developer (so already did this before), he said it
> took him 3 days and 52 configurations changes (all were committed to
> git, so was possible to count exactly) to setup mail client properly.
> And he is "staffed" to do kernel work, I would expect that most people
> who don't _have_ to do kernel contributions will turn away half-way.
>
> As another data point, several people told me that they are afraid of
> sending kernel patches b/c there is so much "on you" to do right.
>
> I would say that we need to aim at  a process that does not require a
> friendly experienced person to answer any of your questions in the
> common case. Lots of people will simply not ask any questions.

Re: RFE: use patchwork to submit a patch

[Theodore Y. Ts'o]

On Fri, Nov 08, 2019 at 10:44:25AM +0100, Dmitry Vyukov wrote:
> syzbot asks to provide fixing commit in a particular format (and on 1
> line, otherwise not possible to parse back).
> 
> It sent this yesterday:
> https://groups.google.com/forum/message/raw?msg=syzkaller-bugs/IZZUu-BobtI/xAkxm6H4EQAJ

This is Shuah's reply, not the syzbot original message.  Looks like
you cut and pasted the same URL twice?

The correct URL is:

https://groups.google.com/forum/message/raw?msg=syzkaller-bugs/IZZUu-BobtI/wQdqUSHuEQAJ

.. and it looks like what Syzbot sent was already line-wrapped:

>If the result looks correct, please mark the bug fixed by replying with:
>
>#syz fix: usb: usbip: Fix BUG: KASAN: slab-out-of-bounds in
>vhci_hub_control()

So I don't think you can blame this on how Shuah replied, or how
Mozilla handled things.  It looks like either Syzbot or more likely,
GMail on its outgoing path "added value" by line-wrapping the message,
probably because of the following MIME setting:

> Content-Type: text/plain; charset="UTF-8"; format=flowed; delsp=yes

Flowing text lines and deleting spaces is always, ALWAYS, a bad idea
when sending plain text.  In this case, it wasn't just a patch that
got corrupted, but the instructions from Syzbot, which it appears
Shuah followed faithfully.  :-)

IBM handled this problem by standing up an open source e-mail system
because Lotus Notes couldn't be fixed.  If we can't fix GMail, then
David Miller may be right, and e-mail may be doomed.  (I don't like
that answer myself, since so far all of the alternatives seem to be
categorically worse for our use case, but...)

						- Ted

Re: RFE: use patchwork to submit a patch

[Dmitry Vyukov]

On Fri, Nov 8, 2019 at 3:02 PM Theodore Y. Ts'o <tytso@mit.edu> wrote:
>
> On Fri, Nov 08, 2019 at 10:44:25AM +0100, Dmitry Vyukov wrote:
> > syzbot asks to provide fixing commit in a particular format (and on 1
> > line, otherwise not possible to parse back).
> >
> > It sent this yesterday:
> > https://groups.google.com/forum/message/raw?msg=syzkaller-bugs/IZZUu-BobtI/xAkxm6H4EQAJ
>
> This is Shuah's reply, not the syzbot original message.  Looks like
> you cut and pasted the same URL twice?
>
> The correct URL is:
>
> https://groups.google.com/forum/message/raw?msg=syzkaller-bugs/IZZUu-BobtI/wQdqUSHuEQAJ
>
> .. and it looks like what Syzbot sent was already line-wrapped:
>
> >If the result looks correct, please mark the bug fixed by replying with:
> >
> >#syz fix: usb: usbip: Fix BUG: KASAN: slab-out-of-bounds in
> >vhci_hub_control()
>
> So I don't think you can blame this on how Shuah replied, or how
> Mozilla handled things.  It looks like either Syzbot or more likely,
> GMail on its outgoing path "added value" by line-wrapping the message,
> probably because of the following MIME setting:
>
> > Content-Type: text/plain; charset="UTF-8"; format=flowed; delsp=yes
>
> Flowing text lines and deleting spaces is always, ALWAYS, a bad idea
> when sending plain text.  In this case, it wasn't just a patch that
> got corrupted, but the instructions from Syzbot, which it appears
> Shuah followed faithfully.  :-)

Looks correct to me. See the DelSp=yes part here:
https://tools.ietf.org/html/rfc3676
But I agree this is insanity. But that's exactly the point.


> IBM handled this problem by standing up an open source e-mail system
> because Lotus Notes couldn't be fixed.  If we can't fix GMail, then
> David Miller may be right, and e-mail may be doomed.  (I don't like
> that answer myself, since so far all of the alternatives seem to be
> categorically worse for our use case, but...)
>
>                                                 - Ted

Re: RFE: use patchwork to submit a patch

[Dmitry Vyukov]

On Fri, Nov 8, 2019 at 3:11 PM Dmitry Vyukov <dvyukov@google.com> wrote:
>
> On Fri, Nov 8, 2019 at 3:02 PM Theodore Y. Ts'o <tytso@mit.edu> wrote:
> >
> > On Fri, Nov 08, 2019 at 10:44:25AM +0100, Dmitry Vyukov wrote:
> > > syzbot asks to provide fixing commit in a particular format (and on 1
> > > line, otherwise not possible to parse back).
> > >
> > > It sent this yesterday:
> > > https://groups.google.com/forum/message/raw?msg=syzkaller-bugs/IZZUu-BobtI/xAkxm6H4EQAJ
> >
> > This is Shuah's reply, not the syzbot original message.  Looks like
> > you cut and pasted the same URL twice?
> >
> > The correct URL is:
> >
> > https://groups.google.com/forum/message/raw?msg=syzkaller-bugs/IZZUu-BobtI/wQdqUSHuEQAJ
> >
> > .. and it looks like what Syzbot sent was already line-wrapped:
> >
> > >If the result looks correct, please mark the bug fixed by replying with:
> > >
> > >#syz fix: usb: usbip: Fix BUG: KASAN: slab-out-of-bounds in
> > >vhci_hub_control()
> >
> > So I don't think you can blame this on how Shuah replied, or how
> > Mozilla handled things.  It looks like either Syzbot or more likely,
> > GMail on its outgoing path "added value" by line-wrapping the message,
> > probably because of the following MIME setting:
> >
> > > Content-Type: text/plain; charset="UTF-8"; format=flowed; delsp=yes
> >
> > Flowing text lines and deleting spaces is always, ALWAYS, a bad idea
> > when sending plain text.  In this case, it wasn't just a patch that
> > got corrupted, but the instructions from Syzbot, which it appears
> > Shuah followed faithfully.  :-)
>
> Looks correct to me. See the DelSp=yes part here:
> https://tools.ietf.org/html/rfc3676
> But I agree this is insanity. But that's exactly the point.

Just to clarify: syzbot does not do this on purpose. It generates and
hands off unwrapped lines. The "format=flowed; delsp=yes" part was
done by somebody else.

> > IBM handled this problem by standing up an open source e-mail system
> > because Lotus Notes couldn't be fixed.  If we can't fix GMail, then
> > David Miller may be right, and e-mail may be doomed.  (I don't like
> > that answer myself, since so far all of the alternatives seem to be
> > categorically worse for our use case, but...)
> >
> >                                                 - Ted

Re: RFE: use patchwork to submit a patch

[Laurent Pinchart]

Hi Ted,

On Fri, Nov 08, 2019 at 09:02:49AM -0500, Theodore Y. Ts'o wrote:
> On Fri, Nov 08, 2019 at 10:44:25AM +0100, Dmitry Vyukov wrote:
> > syzbot asks to provide fixing commit in a particular format (and on 1
> > line, otherwise not possible to parse back).
> > 
> > It sent this yesterday:
> > https://groups.google.com/forum/message/raw?msg=syzkaller-bugs/IZZUu-BobtI/xAkxm6H4EQAJ
> 
> This is Shuah's reply, not the syzbot original message.  Looks like
> you cut and pasted the same URL twice?
> 
> The correct URL is:
> 
> https://groups.google.com/forum/message/raw?msg=syzkaller-bugs/IZZUu-BobtI/wQdqUSHuEQAJ
> 
> .. and it looks like what Syzbot sent was already line-wrapped:
> 
> > If the result looks correct, please mark the bug fixed by replying with:
> >
> >#syz fix: usb: usbip: Fix BUG: KASAN: slab-out-of-bounds in
> >vhci_hub_control()
> 
> So I don't think you can blame this on how Shuah replied, or how
> Mozilla handled things.  It looks like either Syzbot or more likely,
> GMail on its outgoing path "added value" by line-wrapping the message,
> probably because of the following MIME setting:
> 
> > Content-Type: text/plain; charset="UTF-8"; format=flowed; delsp=yes
> 
> Flowing text lines and deleting spaces is always, ALWAYS, a bad idea
> when sending plain text.  In this case, it wasn't just a patch that
> got corrupted, but the instructions from Syzbot, which it appears
> Shuah followed faithfully.  :-)
> 
> IBM handled this problem by standing up an open source e-mail system
> because Lotus Notes couldn't be fixed.  If we can't fix GMail, then
> David Miller may be right, and e-mail may be doomed.  (I don't like
> that answer myself, since so far all of the alternatives seem to be
> categorically worse for our use case, but...)

Since when did we decide to let Google dictate what tools were are or
are not allowed to use ? :-) If gmail isn't suitable as an e-mail
provider, we should be vocal about it.

-- 
Regards,

Laurent Pinchart

Re: RFE: use patchwork to submit a patch

[Konstantin Ryabitsev]

On Fri, Nov 08, 2019 at 03:12:51PM +0100, Dmitry Vyukov wrote:
> Just to clarify: syzbot does not do this on purpose. It generates and
> hands off unwrapped lines. The "format=flowed; delsp=yes" part was
> done by somebody else.

Looks like vger received it already mangled:
https://lore.kernel.org/linux-usb/000000000000c460630596c1d40d@google.com/raw

Would you like a mail sending account so you can use mail.kernel.org as 
your outgoing smtp? We promise we won't do anything that nasty. :)

-K

Re: RFE: use patchwork to submit a patch

[Dmitry Vyukov]

On Fri, Nov 8, 2019 at 3:17 PM Konstantin Ryabitsev
<konstantin@linuxfoundation.org> wrote:
>
> On Fri, Nov 08, 2019 at 03:12:51PM +0100, Dmitry Vyukov wrote:
> > Just to clarify: syzbot does not do this on purpose. It generates and
> > hands off unwrapped lines. The "format=flowed; delsp=yes" part was
> > done by somebody else.
>
> Looks like vger received it already mangled:
> https://lore.kernel.org/linux-usb/000000000000c460630596c1d40d@google.com/raw

It has the format=flowed; delsp=yes part:

Content-Type: text/plain; charset="UTF-8"; format=flowed; delsp=yes

Which according to https://tools.ietf.org/html/rfc3676 seems to mean
that that line is not wrapped: new line and 1 space needs to be
removed as part of decoding it.


> Would you like a mail sending account so you can use mail.kernel.org as
> your outgoing smtp? We promise we won't do anything that nasty. :)
>
> -K

You know that we also serve Fuchsia, Akaros, FreeBSD, NetBSD, OpenBSD
and potentially Windows and Darwin in the future, right? :)

How does that work? Currently we just call mail.Send:
https://cloud.google.com/appengine/docs/standard/go/mail/reference
and it magically does the thing. I can't even setup my own email
client, so don't expect lots from me :)

Re: RFE: use patchwork to submit a patch

[Theodore Y. Ts'o]

On Fri, Nov 08, 2019 at 03:25:00PM +0100, Dmitry Vyukov wrote:
> 
> It has the format=flowed; delsp=yes part:
> 
> Content-Type: text/plain; charset="UTF-8"; format=flowed; delsp=yes
> 
> Which according to https://tools.ietf.org/html/rfc3676 seems to mean
> that that line is not wrapped: new line and 1 space needs to be
> removed as part of decoding it.

Actually if you read RFC 3676 carefully, the receiver MAY decode as
you have described above.  It's not even a SHOULD or a MUST.

Essentially format=flowed is only supposed to be used when it's
considered OK for the receiver to reflow (or not) the lines any darned
way it wants.

If you use format=flowed in a context where (a) there might be
whitespace at the end of lines, and random reflowing is a really bad
idea (e.g., C/C++/Python/Java source files, or patches), or (b) where
you really care about whether line breaks when text is quoted and used
in replies (e.g., syzbot's parser which cares about line breaks), you
MUST NOT use format=flowed, or things will only end in tears.

If the problem here is that syzbot mail, and the responses to its
mails, can't be broken in random places, or syzbot parser will be
unhappy.  In that case, you shouldn't have sent it using format=flowed
in the first place.  There are other MIME text formats that are
appropriate in that particular case, but format=flowed is not one of
them.  And if GMail doesn't give you that option, then I suggest that
you either make the Syzbot parser more forgiving, or you find another
mail transport agent.

   		   	     		      - Ted

Re: RFE: use patchwork to submit a patch

[Dmitry Vyukov]

On Sat, Nov 9, 2019 at 5:31 AM Theodore Y. Ts'o <tytso@mit.edu> wrote:
>
> On Fri, Nov 08, 2019 at 03:25:00PM +0100, Dmitry Vyukov wrote:
> >
> > It has the format=flowed; delsp=yes part:
> >
> > Content-Type: text/plain; charset="UTF-8"; format=flowed; delsp=yes
> >
> > Which according to https://tools.ietf.org/html/rfc3676 seems to mean
> > that that line is not wrapped: new line and 1 space needs to be
> > removed as part of decoding it.
>
> Actually if you read RFC 3676 carefully, the receiver MAY decode as
> you have described above.  It's not even a SHOULD or a MUST.
>
> Essentially format=flowed is only supposed to be used when it's
> considered OK for the receiver to reflow (or not) the lines any darned
> way it wants.
>
> If you use format=flowed in a context where (a) there might be
> whitespace at the end of lines, and random reflowing is a really bad
> idea (e.g., C/C++/Python/Java source files, or patches), or (b) where
> you really care about whether line breaks when text is quoted and used
> in replies (e.g., syzbot's parser which cares about line breaks), you
> MUST NOT use format=flowed, or things will only end in tears.

You are right.
syzbot email only MAY be correct :)
But for me the point re email still stands: why are we even spending
time discussing this? Why are there such extensions with MAY status?
If one doesn't care if the received with recover the original message
or not, why caring adding/specifying these "format=flowed;
delsp=yes"?...
Obviously somebody in the middle got confused about these flowed/delsp
as well and either assumed that they are MUST or assumed that
preserving precise text for emails is just never important...

> If the problem here is that syzbot mail, and the responses to its
> mails, can't be broken in random places, or syzbot parser will be
> unhappy.  In that case, you shouldn't have sent it using format=flowed
> in the first place.  There are other MIME text formats that are
> appropriate in that particular case, but format=flowed is not one of
> them.  And if GMail doesn't give you that option, then I suggest that
> you either make the Syzbot parser more forgiving, or you find another
> mail transport agent.

I don't see how it's possible to make it more forgiving without also
appending unrelated text that happened to be on subsequent lines to
patch titles and breaking everything the other way.
I added one partial escape hatch in the form of:

#syz fix:
whole-title-goes-on-the-next-line

This allows to send slightly longer titles. This is still parsable as
we know that commit titles can't be empty.

Regarding another mail agent: again this only proves the point for me:
this is what tool developers are forced to be spending their resources
on, rather then working on adding more useful features...
I don't even know where to start re switching mail transport; how much
the switch will cost? what are other transport costs in the long term
maintenance? what are their problems?

Re: RFE: use patchwork to submit a patch

[Mark Brown]

[-- Attachment #1: Type: text/plain, Size: 1785 bytes --]

On Mon, Nov 11, 2019 at 10:35:10AM +0100, Dmitry Vyukov wrote:
> On Sat, Nov 9, 2019 at 5:31 AM Theodore Y. Ts'o <tytso@mit.edu> wrote:

> > Essentially format=flowed is only supposed to be used when it's
> > considered OK for the receiver to reflow (or not) the lines any darned
> > way it wants.

> But for me the point re email still stands: why are we even spending
> time discussing this? Why are there such extensions with MAY status?
> If one doesn't care if the received with recover the original message
> or not, why caring adding/specifying these "format=flowed;
> delsp=yes"?...

> Obviously somebody in the middle got confused about these flowed/delsp
> as well and either assumed that they are MUST or assumed that
> preserving precise text for emails is just never important...

Not sure if you're looking for a serious answer here or not and it's not
really your point but the use case for format=flowed is for plain text
mails.  The sending MUA should flow the mail into 80 columns so it will
render OK as text but if something wants to reflow (eg, to fit within a
window) like it would for a paragraph in HTML mail then it can.  It's
optional so that things where the formatting is important don't get
disrupted.

> Regarding another mail agent: again this only proves the point for me:
> this is what tool developers are forced to be spending their resources
> on, rather then working on adding more useful features...
> I don't even know where to start re switching mail transport; how much
> the switch will cost? what are other transport costs in the long term
> maintenance? what are their problems?

We're not going to get away from interoperability problems no matter
what we use, especially if we are mixing things intended for human and
machine consumption.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: RFE: use patchwork to submit a patch

[Theodore Y. Ts'o]

On Mon, Nov 11, 2019 at 10:35:10AM +0100, Dmitry Vyukov wrote:
> 
> Regarding another mail agent: again this only proves the point for me:
> this is what tool developers are forced to be spending their resources
> on, rather then working on adding more useful features...
> I don't even know where to start re switching mail transport; how much
> the switch will cost? what are other transport costs in the long term
> maintenance? what are their problems?

This is a GMail bug/problem.  Your reply to me was sent without
format=flowed, so it's certainly possible that GMail not use
format=flowed.

There are plenty of open source mail transport agents where sending
this without format=flowed would be trivial....

					- Ted

Re: RFE: use patchwork to submit a patch

[Konstantin Ryabitsev]

On Fri, Nov 08, 2019 at 03:25:00PM +0100, Dmitry Vyukov wrote:
> > Would you like a mail sending account so you can use mail.kernel.org 
> > as
> > your outgoing smtp? We promise we won't do anything that nasty. :)
> >
> > -K
> 
> You know that we also serve Fuchsia, Akaros, FreeBSD, NetBSD, OpenBSD
> and potentially Windows and Darwin in the future, right? :)
> 
> How does that work? Currently we just call mail.Send:
> https://cloud.google.com/appengine/docs/standard/go/mail/reference
> and it magically does the thing. I can't even setup my own email
> client, so don't expect lots from me :)

I didn't realize you were using appengine, so I have no idea if using an 
external SMTP submission server is even feasible. I'm sure it can be 
done, but since you're serving all those other communities as well, that 
makes the offer of acting as a submission point a bit more tricky. :) 
Perhaps we can revisit this idea once you start publishing a feed?

-K