git send-email friendly smtp provider anyone?

git send-email friendly smtp provider anyone?

[Noralf Trønnes]

Hi

A couple of years ago my email provider blocked me from using git
send-email with their smtp server. So I switched to the one my ISP
provides. Now my ISP have outsourced their email service so the first 3
emails gets through and the rest looks like it ends up in a tar pit or
something, 18 hours later and 5 of 7 emails have gotten through. I have
asked them about this, but I fear the answer will be this is not
supported since they now don't have the service in-house anymore. I'm
waiting for a reply.

Today I tried sendinblue.com since they have a free plan, but they
insert <br> in the emails so that didn't work out. They also have some
kind of queue, after 1 hour 6 of 7 emails have gotten through.

Does anyone have an smtp provider to recommend that works with git
send-email and that sends out all the emails at once?

I have a patchset that I want to send out.

Noralf.

git send-email friendly smtp provider anyone?

[Noralf Trønnes]

Hi

A couple of years ago my email provider blocked me from using git
send-email with their smtp server. So I switched to the one my ISP
provides. Now my ISP have outsourced their email service so the first 3
emails gets through and the rest looks like it ends up in a tar pit or
something, 18 hours later and 5 of 7 emails have gotten through. I have
asked them about this, but I fear the answer will be this is not
supported since they now don't have the service in-house anymore. I'm
waiting for a reply.

Today I tried sendinblue.com since they have a free plan, but they
insert <br> in the emails so that didn't work out. They also have some
kind of queue, after 1 hour 6 of 7 emails have gotten through.

Does anyone have an smtp provider to recommend that works with git
send-email and that sends out all the emails at once?

I have a patchset that I want to send out.

Noralf.

Re: git send-email friendly smtp provider anyone?

[Simon Ser]

I think you can apply for a linux.dev mailbox [1].

[1]: https://korg.docs.kernel.org/linuxdev.html

Re: git send-email friendly smtp provider anyone?

[Simon Ser]

I think you can apply for a linux.dev mailbox [1].

[1]: https://korg.docs.kernel.org/linuxdev.html

Re: git send-email friendly smtp provider anyone?

[Noralf Trønnes]

Den 21.11.2022 14.33, skrev Simon Ser:
> I think you can apply for a linux.dev mailbox [1].
> 

Yeah you're right, I didn't know about that possibility.
But it depends on whether or not I can just use their smtp server and
keep my current email address. This looks like what's the problem with
my current ISP, I need to use the email account I have in their email
service (that I've never used) for sending through their smtp server,
but I want to send From: another email address.

Noralf.

> [1]: https://korg.docs.kernel.org/linuxdev.html

Re: git send-email friendly smtp provider anyone?

[Noralf Trønnes]

Den 21.11.2022 14.33, skrev Simon Ser:
> I think you can apply for a linux.dev mailbox [1].
> 

Yeah you're right, I didn't know about that possibility.
But it depends on whether or not I can just use their smtp server and
keep my current email address. This looks like what's the problem with
my current ISP, I need to use the email account I have in their email
service (that I've never used) for sending through their smtp server,
but I want to send From: another email address.

Noralf.

> [1]: https://korg.docs.kernel.org/linuxdev.html

Re: git send-email friendly smtp provider anyone?

[Simon Ser]

On Monday, November 21st, 2022 at 17:52, Noralf Trønnes <noralf@tronnes.org> wrote:

> Den 21.11.2022 14.33, skrev Simon Ser:
> 
> > I think you can apply for a linux.dev mailbox 1.
> 
> Yeah you're right, I didn't know about that possibility.
> But it depends on whether or not I can just use their smtp server and
> keep my current email address. This looks like what's the problem with
> my current ISP, I need to use the email account I have in their email
> service (that I've never used) for sending through their smtp server,
> but I want to send From: another email address.

That's not possible. It breaks DKIM, so your emails will end up in Spam
folders or be rejected. You need to use the SMTP server tied to your
email address.

Re: git send-email friendly smtp provider anyone?

[Simon Ser]

On Monday, November 21st, 2022 at 17:52, Noralf Trønnes <noralf@tronnes.org> wrote:

> Den 21.11.2022 14.33, skrev Simon Ser:
> 
> > I think you can apply for a linux.dev mailbox 1.
> 
> Yeah you're right, I didn't know about that possibility.
> But it depends on whether or not I can just use their smtp server and
> keep my current email address. This looks like what's the problem with
> my current ISP, I need to use the email account I have in their email
> service (that I've never used) for sending through their smtp server,
> but I want to send From: another email address.

That's not possible. It breaks DKIM, so your emails will end up in Spam
folders or be rejected. You need to use the SMTP server tied to your
email address.

Re: git send-email friendly smtp provider anyone?

[Simon Ser]

On Monday, November 21st, 2022 at 18:02, Simon Ser <contact@emersion.fr> wrote:

> On Monday, November 21st, 2022 at 17:52, Noralf Trønnes noralf@tronnes.org wrote:
> 
> > Den 21.11.2022 14.33, skrev Simon Ser:
> > 
> > > I think you can apply for a linux.dev mailbox 1.
> > 
> > Yeah you're right, I didn't know about that possibility.
> > But it depends on whether or not I can just use their smtp server and
> > keep my current email address. This looks like what's the problem with
> > my current ISP, I need to use the email account I have in their email
> > service (that I've never used) for sending through their smtp server,
> > but I want to send From: another email address.
> 
> That's not possible. It breaks DKIM, so your emails will end up in Spam
> folders or be rejected. You need to use the SMTP server tied to your
> email address.

That said, you can send patches from an email address different from
the one in your patches. IOW, you can send patches committed by
<noralf@tronnes.org> from any email account.

The From in the email header won't match the commit, but the very first
line of the patch will hold that information.

Re: git send-email friendly smtp provider anyone?

[Simon Ser]

On Monday, November 21st, 2022 at 18:02, Simon Ser <contact@emersion.fr> wrote:

> On Monday, November 21st, 2022 at 17:52, Noralf Trønnes noralf@tronnes.org wrote:
> 
> > Den 21.11.2022 14.33, skrev Simon Ser:
> > 
> > > I think you can apply for a linux.dev mailbox 1.
> > 
> > Yeah you're right, I didn't know about that possibility.
> > But it depends on whether or not I can just use their smtp server and
> > keep my current email address. This looks like what's the problem with
> > my current ISP, I need to use the email account I have in their email
> > service (that I've never used) for sending through their smtp server,
> > but I want to send From: another email address.
> 
> That's not possible. It breaks DKIM, so your emails will end up in Spam
> folders or be rejected. You need to use the SMTP server tied to your
> email address.

That said, you can send patches from an email address different from
the one in your patches. IOW, you can send patches committed by
<noralf@tronnes.org> from any email account.

The From in the email header won't match the commit, but the very first
line of the patch will hold that information.

Re: git send-email friendly smtp provider anyone?

[Noralf Trønnes]

Den 21.11.2022 18.06, skrev Simon Ser:
> On Monday, November 21st, 2022 at 18:02, Simon Ser <contact@emersion.fr> wrote:
> 
>> On Monday, November 21st, 2022 at 17:52, Noralf Trønnes noralf@tronnes.org wrote:
>>
>>> Den 21.11.2022 14.33, skrev Simon Ser:
>>>
>>>> I think you can apply for a linux.dev mailbox 1.
>>>
>>> Yeah you're right, I didn't know about that possibility.
>>> But it depends on whether or not I can just use their smtp server and
>>> keep my current email address. This looks like what's the problem with
>>> my current ISP, I need to use the email account I have in their email
>>> service (that I've never used) for sending through their smtp server,
>>> but I want to send From: another email address.
>>
>> That's not possible. It breaks DKIM, so your emails will end up in Spam
>> folders or be rejected. You need to use the SMTP server tied to your
>> email address.
> 
> That said, you can send patches from an email address different from
> the one in your patches. IOW, you can send patches committed by
> <noralf@tronnes.org> from any email account.
> 
> The From in the email header won't match the commit, but the very first
> line of the patch will hold that information.

Thanks that was useful information. I've seen the DKIM abbr. but haven't
looked into the meaning of it.

I tried:

git send-email --from=noralf.tronnes@altiboxmail.no
--reply=noralf@tronnes.org

and now I'm getting 'pass' in the Authentication-Results field, so
that's progress. I'm still not getting all the emails through, so I
still have that problem, I'll have to wait and see what the ISP can tell me.

But this means that a linux.dev mailbox is an option for me should my
ISP be a blocker.

Noralf.

Re: git send-email friendly smtp provider anyone?

[Noralf Trønnes]

Den 21.11.2022 18.06, skrev Simon Ser:
> On Monday, November 21st, 2022 at 18:02, Simon Ser <contact@emersion.fr> wrote:
> 
>> On Monday, November 21st, 2022 at 17:52, Noralf Trønnes noralf@tronnes.org wrote:
>>
>>> Den 21.11.2022 14.33, skrev Simon Ser:
>>>
>>>> I think you can apply for a linux.dev mailbox 1.
>>>
>>> Yeah you're right, I didn't know about that possibility.
>>> But it depends on whether or not I can just use their smtp server and
>>> keep my current email address. This looks like what's the problem with
>>> my current ISP, I need to use the email account I have in their email
>>> service (that I've never used) for sending through their smtp server,
>>> but I want to send From: another email address.
>>
>> That's not possible. It breaks DKIM, so your emails will end up in Spam
>> folders or be rejected. You need to use the SMTP server tied to your
>> email address.
> 
> That said, you can send patches from an email address different from
> the one in your patches. IOW, you can send patches committed by
> <noralf@tronnes.org> from any email account.
> 
> The From in the email header won't match the commit, but the very first
> line of the patch will hold that information.

Thanks that was useful information. I've seen the DKIM abbr. but haven't
looked into the meaning of it.

I tried:

git send-email --from=noralf.tronnes@altiboxmail.no
--reply=noralf@tronnes.org

and now I'm getting 'pass' in the Authentication-Results field, so
that's progress. I'm still not getting all the emails through, so I
still have that problem, I'll have to wait and see what the ISP can tell me.

But this means that a linux.dev mailbox is an option for me should my
ISP be a blocker.

Noralf.

Re: git send-email friendly smtp provider anyone?

[Maxime Ripard]

[-- Attachment #1: Type: text/plain, Size: 1153 bytes --]

On Mon, Nov 21, 2022 at 12:48:52PM +0100, Noralf Trønnes wrote:
> A couple of years ago my email provider blocked me from using git
> send-email with their smtp server. So I switched to the one my ISP
> provides. Now my ISP have outsourced their email service so the first 3
> emails gets through and the rest looks like it ends up in a tar pit or
> something, 18 hours later and 5 of 7 emails have gotten through. I have
> asked them about this, but I fear the answer will be this is not
> supported since they now don't have the service in-house anymore. I'm
> waiting for a reply.
> 
> Today I tried sendinblue.com since they have a free plan, but they
> insert <br> in the emails so that didn't work out. They also have some
> kind of queue, after 1 hour 6 of 7 emails have gotten through.
> 
> Does anyone have an smtp provider to recommend that works with git
> send-email and that sends out all the emails at once?

I'm using fastmail and am very happy about it so far.

Otherwise, you might consider using:
https://b4.docs.kernel.org/en/latest/contributor/send.html#authenticating-with-the-web-submission-endpoint

Maxime

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 228 bytes --]

Re: git send-email friendly smtp provider anyone?

[Maxime Ripard]

[-- Attachment #1: Type: text/plain, Size: 1153 bytes --]

On Mon, Nov 21, 2022 at 12:48:52PM +0100, Noralf Trønnes wrote:
> A couple of years ago my email provider blocked me from using git
> send-email with their smtp server. So I switched to the one my ISP
> provides. Now my ISP have outsourced their email service so the first 3
> emails gets through and the rest looks like it ends up in a tar pit or
> something, 18 hours later and 5 of 7 emails have gotten through. I have
> asked them about this, but I fear the answer will be this is not
> supported since they now don't have the service in-house anymore. I'm
> waiting for a reply.
> 
> Today I tried sendinblue.com since they have a free plan, but they
> insert <br> in the emails so that didn't work out. They also have some
> kind of queue, after 1 hour 6 of 7 emails have gotten through.
> 
> Does anyone have an smtp provider to recommend that works with git
> send-email and that sends out all the emails at once?

I'm using fastmail and am very happy about it so far.

Otherwise, you might consider using:
https://b4.docs.kernel.org/en/latest/contributor/send.html#authenticating-with-the-web-submission-endpoint

Maxime

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 228 bytes --]

Re: git send-email friendly smtp provider anyone?

[Noralf Trønnes]

Den 21.11.2022 16.19, skrev Maxime Ripard:
> On Mon, Nov 21, 2022 at 12:48:52PM +0100, Noralf Trønnes wrote:
>> A couple of years ago my email provider blocked me from using git
>> send-email with their smtp server. So I switched to the one my ISP
>> provides. Now my ISP have outsourced their email service so the first 3
>> emails gets through and the rest looks like it ends up in a tar pit or
>> something, 18 hours later and 5 of 7 emails have gotten through. I have
>> asked them about this, but I fear the answer will be this is not
>> supported since they now don't have the service in-house anymore. I'm
>> waiting for a reply.
>>
>> Today I tried sendinblue.com since they have a free plan, but they
>> insert <br> in the emails so that didn't work out. They also have some
>> kind of queue, after 1 hour 6 of 7 emails have gotten through.
>>
>> Does anyone have an smtp provider to recommend that works with git
>> send-email and that sends out all the emails at once?
> 
> I'm using fastmail and am very happy about it so far.
> 
> Otherwise, you might consider using:
> https://b4.docs.kernel.org/en/latest/contributor/send.html#authenticating-with-the-web-submission-endpoint
> 

That's an interesting option. I did briefly look at b4 a few months back
but it looked like it was under heavy development so I figured I'd wait
before trying it out. I think I'll give b4 a spin to see how it works, I
wonder how it handles patch changelogs.

Noralf.

Re: git send-email friendly smtp provider anyone?

[Noralf Trønnes]

Den 21.11.2022 16.19, skrev Maxime Ripard:
> On Mon, Nov 21, 2022 at 12:48:52PM +0100, Noralf Trønnes wrote:
>> A couple of years ago my email provider blocked me from using git
>> send-email with their smtp server. So I switched to the one my ISP
>> provides. Now my ISP have outsourced their email service so the first 3
>> emails gets through and the rest looks like it ends up in a tar pit or
>> something, 18 hours later and 5 of 7 emails have gotten through. I have
>> asked them about this, but I fear the answer will be this is not
>> supported since they now don't have the service in-house anymore. I'm
>> waiting for a reply.
>>
>> Today I tried sendinblue.com since they have a free plan, but they
>> insert <br> in the emails so that didn't work out. They also have some
>> kind of queue, after 1 hour 6 of 7 emails have gotten through.
>>
>> Does anyone have an smtp provider to recommend that works with git
>> send-email and that sends out all the emails at once?
> 
> I'm using fastmail and am very happy about it so far.
> 
> Otherwise, you might consider using:
> https://b4.docs.kernel.org/en/latest/contributor/send.html#authenticating-with-the-web-submission-endpoint
> 

That's an interesting option. I did briefly look at b4 a few months back
but it looked like it was under heavy development so I figured I'd wait
before trying it out. I think I'll give b4 a spin to see how it works, I
wonder how it handles patch changelogs.

Noralf.

Re: git send-email friendly smtp provider anyone?

[Konstantin Ryabitsev]

On Mon, Nov 21, 2022 at 07:13:28PM +0100, Noralf Trønnes wrote:
> > Otherwise, you might consider using:
> > https://b4.docs.kernel.org/en/latest/contributor/send.html#authenticating-with-the-web-submission-endpoint
> > 
> 
> That's an interesting option. I did briefly look at b4 a few months back
> but it looked like it was under heavy development so I figured I'd wait
> before trying it out. I think I'll give b4 a spin to see how it works, I
> wonder how it handles patch changelogs.

I'd be happy to help set this up for you -- to date, this service hasn't been
used beyond a few test posts.

-K

Re: git send-email friendly smtp provider anyone?

[Noralf Trønnes]

Den 22.11.2022 16.51, skrev Konstantin Ryabitsev:
> On Mon, Nov 21, 2022 at 07:13:28PM +0100, Noralf Trønnes wrote:
>>> Otherwise, you might consider using:
>>> https://b4.docs.kernel.org/en/latest/contributor/send.html#authenticating-with-the-web-submission-endpoint
>>>
>>
>> That's an interesting option. I did briefly look at b4 a few months back
>> but it looked like it was under heavy development so I figured I'd wait
>> before trying it out. I think I'll give b4 a spin to see how it works, I
>> wonder how it handles patch changelogs.
> 
> I'd be happy to help set this up for you -- to date, this service hasn't been
> used beyond a few test posts.
> 

Ok, I'll give it a try.

I have now prepared the patchset, generated a key and can now do:
b4 send -o

The first thing that strikes me is that everyone mentioned in one of the
patches get the entire patchset, even stable@vger.kernel.org (cc'ed in a
fixes patch). The first patch touches a core file and as a result a few
drivers, so I've cc'ed the driver maintainers in that patch, but now
they get the entire patchset where 5 of 6 patches is about a driver that
I maintain. So from their point of view, they see a patchset about a
driver they don't care about and a patch touching a core file, but from
the subject it's not apparent that it touches their driver. I'm afraid
that this might result in none of them looking at that patch. In this
particular case it's not that important, but in another case it might be.

As for the setting up the web endpoint, should I just follow the b4 docs
on that?

I use b4 version 0.10.1, is that recent enough?

Noralf.

Re: git send-email friendly smtp provider anyone?

[Noralf Trønnes]

Den 22.11.2022 16.51, skrev Konstantin Ryabitsev:
> On Mon, Nov 21, 2022 at 07:13:28PM +0100, Noralf Trønnes wrote:
>>> Otherwise, you might consider using:
>>> https://b4.docs.kernel.org/en/latest/contributor/send.html#authenticating-with-the-web-submission-endpoint
>>>
>>
>> That's an interesting option. I did briefly look at b4 a few months back
>> but it looked like it was under heavy development so I figured I'd wait
>> before trying it out. I think I'll give b4 a spin to see how it works, I
>> wonder how it handles patch changelogs.
> 
> I'd be happy to help set this up for you -- to date, this service hasn't been
> used beyond a few test posts.
> 

Ok, I'll give it a try.

I have now prepared the patchset, generated a key and can now do:
b4 send -o

The first thing that strikes me is that everyone mentioned in one of the
patches get the entire patchset, even stable@vger.kernel.org (cc'ed in a
fixes patch). The first patch touches a core file and as a result a few
drivers, so I've cc'ed the driver maintainers in that patch, but now
they get the entire patchset where 5 of 6 patches is about a driver that
I maintain. So from their point of view, they see a patchset about a
driver they don't care about and a patch touching a core file, but from
the subject it's not apparent that it touches their driver. I'm afraid
that this might result in none of them looking at that patch. In this
particular case it's not that important, but in another case it might be.

As for the setting up the web endpoint, should I just follow the b4 docs
on that?

I use b4 version 0.10.1, is that recent enough?

Noralf.

Re: git send-email friendly smtp provider anyone?

[Konstantin Ryabitsev]

On Tue, Nov 22, 2022 at 06:42:19PM +0100, Noralf Trønnes wrote:
> The first thing that strikes me is that everyone mentioned in one of the
> patches get the entire patchset, even stable@vger.kernel.org (cc'ed in a
> fixes patch). The first patch touches a core file and as a result a few
> drivers, so I've cc'ed the driver maintainers in that patch, but now
> they get the entire patchset where 5 of 6 patches is about a driver that
> I maintain. So from their point of view, they see a patchset about a
> driver they don't care about and a patch touching a core file, but from
> the subject it's not apparent that it touches their driver. I'm afraid
> that this might result in none of them looking at that patch. In this
> particular case it's not that important, but in another case it might be.

I did some (unscientific) polling among kernel maintainers and, by a vast
margin, they always prefer to receive the entire series instead of
cherry-picked patches -- having the entire series helps provide important
context for the change they are looking at.

So, this is deliberate and, for now at least, not configurable. Unless you're
sending 100+ patch series, I doubt anyone will have any problem with receiving
the whole series instead of individual patches.

> As for the setting up the web endpoint, should I just follow the b4 docs
> on that?
> 
> I use b4 version 0.10.1, is that recent enough?

Yes. There will be a 0.10.2 in the near future, but the incoming fixes
shouldn't make much difference for the b4 send code.

-K

Re: git send-email friendly smtp provider anyone?

[Noralf Trønnes]

Den 22.11.2022 19.50, skrev Konstantin Ryabitsev:
> On Tue, Nov 22, 2022 at 06:42:19PM +0100, Noralf Trønnes wrote:
>> The first thing that strikes me is that everyone mentioned in one of the
>> patches get the entire patchset, even stable@vger.kernel.org (cc'ed in a
>> fixes patch). The first patch touches a core file and as a result a few
>> drivers, so I've cc'ed the driver maintainers in that patch, but now
>> they get the entire patchset where 5 of 6 patches is about a driver that
>> I maintain. So from their point of view, they see a patchset about a
>> driver they don't care about and a patch touching a core file, but from
>> the subject it's not apparent that it touches their driver. I'm afraid
>> that this might result in none of them looking at that patch. In this
>> particular case it's not that important, but in another case it might be.
> 
> I did some (unscientific) polling among kernel maintainers and, by a vast
> margin, they always prefer to receive the entire series instead of
> cherry-picked patches -- having the entire series helps provide important
> context for the change they are looking at.
> 
> So, this is deliberate and, for now at least, not configurable. Unless you're
> sending 100+ patch series, I doubt anyone will have any problem with receiving
> the whole series instead of individual patches.
> 
>> As for the setting up the web endpoint, should I just follow the b4 docs
>> on that?
>>
>> I use b4 version 0.10.1, is that recent enough?
> 
> Yes. There will be a 0.10.2 in the near future, but the incoming fixes
> shouldn't make much difference for the b4 send code.
> 

This is what I got:

$ b4 send --web-auth-verify <challenge string from email>
Signing challenge
Submitting verification to https://lkml.kernel.org/_b4_submit
Traceback (most recent call last):
  File "/home/pi/.local/bin/b4", line 8, in <module>
    sys.exit(cmd())
  File "/home/pi/.local/lib/python3.10/site-packages/b4/command.py",
line 341, in cmd
    cmdargs.func(cmdargs)
  File "/home/pi/.local/lib/python3.10/site-packages/b4/command.py",
line 86, in cmd_send
    b4.ez.cmd_send(cmdargs)
  File "/home/pi/.local/lib/python3.10/site-packages/b4/ez.py", line
1102, in cmd_send
    auth_verify(cmdargs)
  File "/home/pi/.local/lib/python3.10/site-packages/b4/ez.py", line
188, in auth_verify
    res = ses.post(endpoint, json=req)
  File "/usr/lib/python3/dist-packages/requests/sessions.py", line 590,
in post
    return self.request('POST', url, data=data, json=json, **kwargs)
  File "/usr/lib/python3/dist-packages/requests/sessions.py", line 528,
in request
    prep = self.prepare_request(req)
  File "/usr/lib/python3/dist-packages/requests/sessions.py", line 456,
in prepare_request
    p.prepare(
  File "/usr/lib/python3/dist-packages/requests/models.py", line 319, in
prepare
    self.prepare_body(data, files, json)
  File "/usr/lib/python3/dist-packages/requests/models.py", line 469, in
prepare_body
    body = complexjson.dumps(json)
  File "/usr/lib/python3.10/json/__init__.py", line 231, in dumps
    return _default_encoder.encode(obj)
  File "/usr/lib/python3.10/json/encoder.py", line 199, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "/usr/lib/python3.10/json/encoder.py", line 257, in iterencode
    return _iterencode(o, 0)
  File "/usr/lib/python3.10/json/encoder.py", line 179, in default
    raise TypeError(f'Object of type {o.__class__.__name__} '
TypeError: Object of type bytes is not JSON serializable

$ python3 --version
Python 3.10.6

Turning on debug output didn't add much:

$ b4 -d send --web-auth-verify 7ad470b4-f531-4632-8093-738d4d3e5d88
Running git --no-pager rev-parse --show-toplevel
Running git --no-pager config -z --get-regexp b4\..*
Running git --no-pager config -z --get-regexp gpg\..*
Running git --no-pager config -z --get-regexp user\..*
Signing challenge
Submitting verification to https://lkml.kernel.org/_b4_submit
Traceback (most recent call last):
  File "/home/pi/.local/bin/b4", line 8, in <module>
    sys.exit(cmd())
  File "/home/pi/.local/lib/python3.10/site-packages/b4/command.py",
line 341, in cmd
    cmdargs.func(cmdargs)
  File "/home/pi/.local/lib/python3.10/site-packages/b4/command.py",
line 86, in cmd_send
    b4.ez.cmd_send(cmdargs)
  File "/home/pi/.local/lib/python3.10/site-packages/b4/ez.py", line
1102, in cmd_send
    auth_verify(cmdargs)
  File "/home/pi/.local/lib/python3.10/site-packages/b4/ez.py", line
188, in auth_verify
    res = ses.post(endpoint, json=req)
  File "/usr/lib/python3/dist-packages/requests/sessions.py", line 590,
in post
    return self.request('POST', url, data=data, json=json, **kwargs)
  File "/usr/lib/python3/dist-packages/requests/sessions.py", line 528,
in request
    prep = self.prepare_request(req)
  File "/usr/lib/python3/dist-packages/requests/sessions.py", line 456,
in prepare_request
    p.prepare(
  File "/usr/lib/python3/dist-packages/requests/models.py", line 319, in
prepare
    self.prepare_body(data, files, json)
  File "/usr/lib/python3/dist-packages/requests/models.py", line 469, in
prepare_body
    body = complexjson.dumps(json)
  File "/usr/lib/python3.10/json/__init__.py", line 231, in dumps
    return _default_encoder.encode(obj)
  File "/usr/lib/python3.10/json/encoder.py", line 199, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "/usr/lib/python3.10/json/encoder.py", line 257, in iterencode
    return _iterencode(o, 0)
  File "/usr/lib/python3.10/json/encoder.py", line 179, in default
    raise TypeError(f'Object of type {o.__class__.__name__} '
TypeError: Object of type bytes is not JSON serializable


Noralf.

Re: git send-email friendly smtp provider anyone?

[Noralf Trønnes]

Den 22.11.2022 19.50, skrev Konstantin Ryabitsev:
> On Tue, Nov 22, 2022 at 06:42:19PM +0100, Noralf Trønnes wrote:
>> The first thing that strikes me is that everyone mentioned in one of the
>> patches get the entire patchset, even stable@vger.kernel.org (cc'ed in a
>> fixes patch). The first patch touches a core file and as a result a few
>> drivers, so I've cc'ed the driver maintainers in that patch, but now
>> they get the entire patchset where 5 of 6 patches is about a driver that
>> I maintain. So from their point of view, they see a patchset about a
>> driver they don't care about and a patch touching a core file, but from
>> the subject it's not apparent that it touches their driver. I'm afraid
>> that this might result in none of them looking at that patch. In this
>> particular case it's not that important, but in another case it might be.
> 
> I did some (unscientific) polling among kernel maintainers and, by a vast
> margin, they always prefer to receive the entire series instead of
> cherry-picked patches -- having the entire series helps provide important
> context for the change they are looking at.
> 
> So, this is deliberate and, for now at least, not configurable. Unless you're
> sending 100+ patch series, I doubt anyone will have any problem with receiving
> the whole series instead of individual patches.
> 
>> As for the setting up the web endpoint, should I just follow the b4 docs
>> on that?
>>
>> I use b4 version 0.10.1, is that recent enough?
> 
> Yes. There will be a 0.10.2 in the near future, but the incoming fixes
> shouldn't make much difference for the b4 send code.
> 

This is what I got:

$ b4 send --web-auth-verify <challenge string from email>
Signing challenge
Submitting verification to https://lkml.kernel.org/_b4_submit
Traceback (most recent call last):
  File "/home/pi/.local/bin/b4", line 8, in <module>
    sys.exit(cmd())
  File "/home/pi/.local/lib/python3.10/site-packages/b4/command.py",
line 341, in cmd
    cmdargs.func(cmdargs)
  File "/home/pi/.local/lib/python3.10/site-packages/b4/command.py",
line 86, in cmd_send
    b4.ez.cmd_send(cmdargs)
  File "/home/pi/.local/lib/python3.10/site-packages/b4/ez.py", line
1102, in cmd_send
    auth_verify(cmdargs)
  File "/home/pi/.local/lib/python3.10/site-packages/b4/ez.py", line
188, in auth_verify
    res = ses.post(endpoint, json=req)
  File "/usr/lib/python3/dist-packages/requests/sessions.py", line 590,
in post
    return self.request('POST', url, data=data, json=json, **kwargs)
  File "/usr/lib/python3/dist-packages/requests/sessions.py", line 528,
in request
    prep = self.prepare_request(req)
  File "/usr/lib/python3/dist-packages/requests/sessions.py", line 456,
in prepare_request
    p.prepare(
  File "/usr/lib/python3/dist-packages/requests/models.py", line 319, in
prepare
    self.prepare_body(data, files, json)
  File "/usr/lib/python3/dist-packages/requests/models.py", line 469, in
prepare_body
    body = complexjson.dumps(json)
  File "/usr/lib/python3.10/json/__init__.py", line 231, in dumps
    return _default_encoder.encode(obj)
  File "/usr/lib/python3.10/json/encoder.py", line 199, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "/usr/lib/python3.10/json/encoder.py", line 257, in iterencode
    return _iterencode(o, 0)
  File "/usr/lib/python3.10/json/encoder.py", line 179, in default
    raise TypeError(f'Object of type {o.__class__.__name__} '
TypeError: Object of type bytes is not JSON serializable

$ python3 --version
Python 3.10.6

Turning on debug output didn't add much:

$ b4 -d send --web-auth-verify 7ad470b4-f531-4632-8093-738d4d3e5d88
Running git --no-pager rev-parse --show-toplevel
Running git --no-pager config -z --get-regexp b4\..*
Running git --no-pager config -z --get-regexp gpg\..*
Running git --no-pager config -z --get-regexp user\..*
Signing challenge
Submitting verification to https://lkml.kernel.org/_b4_submit
Traceback (most recent call last):
  File "/home/pi/.local/bin/b4", line 8, in <module>
    sys.exit(cmd())
  File "/home/pi/.local/lib/python3.10/site-packages/b4/command.py",
line 341, in cmd
    cmdargs.func(cmdargs)
  File "/home/pi/.local/lib/python3.10/site-packages/b4/command.py",
line 86, in cmd_send
    b4.ez.cmd_send(cmdargs)
  File "/home/pi/.local/lib/python3.10/site-packages/b4/ez.py", line
1102, in cmd_send
    auth_verify(cmdargs)
  File "/home/pi/.local/lib/python3.10/site-packages/b4/ez.py", line
188, in auth_verify
    res = ses.post(endpoint, json=req)
  File "/usr/lib/python3/dist-packages/requests/sessions.py", line 590,
in post
    return self.request('POST', url, data=data, json=json, **kwargs)
  File "/usr/lib/python3/dist-packages/requests/sessions.py", line 528,
in request
    prep = self.prepare_request(req)
  File "/usr/lib/python3/dist-packages/requests/sessions.py", line 456,
in prepare_request
    p.prepare(
  File "/usr/lib/python3/dist-packages/requests/models.py", line 319, in
prepare
    self.prepare_body(data, files, json)
  File "/usr/lib/python3/dist-packages/requests/models.py", line 469, in
prepare_body
    body = complexjson.dumps(json)
  File "/usr/lib/python3.10/json/__init__.py", line 231, in dumps
    return _default_encoder.encode(obj)
  File "/usr/lib/python3.10/json/encoder.py", line 199, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "/usr/lib/python3.10/json/encoder.py", line 257, in iterencode
    return _iterencode(o, 0)
  File "/usr/lib/python3.10/json/encoder.py", line 179, in default
    raise TypeError(f'Object of type {o.__class__.__name__} '
TypeError: Object of type bytes is not JSON serializable


Noralf.

Error using b4 web endpoint (was Re: git send-email friendly smtp provider anyone?)

[Konstantin Ryabitsev]

(Dropping everyone from cc's and adding the tools list.)

On Tue, Nov 22, 2022 at 08:22:22PM +0100, Noralf Trønnes wrote:
> > Yes. There will be a 0.10.2 in the near future, but the incoming fixes
> > shouldn't make much difference for the b4 send code.
> > 
> 
> This is what I got:
> 
> $ b4 send --web-auth-verify <challenge string from email>
> Signing challenge
> Submitting verification to https://lkml.kernel.org/_b4_submit
> Traceback (most recent call last):
>   File "/home/pi/.local/lib/python3.10/site-packages/b4/ez.py", line 188, in auth_verify
>     res = ses.post(endpoint, json=req)
> ...
> TypeError: Object of type bytes is not JSON serializable

This is very strange, because this is literally the code in question:

    req = {
        'action': 'auth-verify',
        'msg': bdata.encode(),
    }
    logger.info('Submitting verification to %s', endpoint)
    ses = b4.get_requests_session()
    res = ses.post(endpoint, json=req)

I don't see where "Object of type bytes" would come from. Can you possibly
insert the following above the `res = ses.post` line:

    logger.debug('endpoint=%s, req=%s', endpoint, req)

Regards,
-K

Re: Error using b4 web endpoint (was Re: git send-email friendly smtp provider anyone?)

[Noralf Trønnes]

Den 22.11.2022 20.44, skrev Konstantin Ryabitsev:
> (Dropping everyone from cc's and adding the tools list.)
> 
> On Tue, Nov 22, 2022 at 08:22:22PM +0100, Noralf Trønnes wrote:
>>> Yes. There will be a 0.10.2 in the near future, but the incoming fixes
>>> shouldn't make much difference for the b4 send code.
>>>
>>
>> This is what I got:
>>
>> $ b4 send --web-auth-verify <challenge string from email>
>> Signing challenge
>> Submitting verification to https://lkml.kernel.org/_b4_submit
>> Traceback (most recent call last):
>>   File "/home/pi/.local/lib/python3.10/site-packages/b4/ez.py", line 188, in auth_verify
>>     res = ses.post(endpoint, json=req)
>> ...
>> TypeError: Object of type bytes is not JSON serializable
> 
> This is very strange, because this is literally the code in question:
> 
>     req = {
>         'action': 'auth-verify',
>         'msg': bdata.encode(),
>     }
>     logger.info('Submitting verification to %s', endpoint)
>     ses = b4.get_requests_session()
>     res = ses.post(endpoint, json=req)
> 
> I don't see where "Object of type bytes" would come from. Can you possibly
> insert the following above the `res = ses.post` line:
> 
>     logger.debug('endpoint=%s, req=%s', endpoint, req)
> 

$ b4 -d send --web-auth-verify <challenge>
Running git --no-pager rev-parse --show-toplevel
Running git --no-pager config -z --get-regexp b4\..*
Running git --no-pager config -z --get-regexp gpg\..*
Running git --no-pager config -z --get-regexp user\..*
Signing challenge
Submitting verification to https://lkml.kernel.org/_b4_submit
endpoint=https://lkml.kernel.org/_b4_submit, req={'action':
'auth-verify', 'msg': b'From: noralf@tronnes.org\nSubject:
b4-send-verify\nX-Developer-Signature: v=1; a=ed25519-sha256;
t=1669146435; l=45;\n i=noralf@tronnes.org; s=20221122;
h=from:subject;\n bh=E4BgU7G2i7CeIkR8F1AfDTtOvDBQA7ld22s0U29/H84=;\n
b=eRcDEBQvE702NSqEWQDLRyp9emSRvBjdhIW1c8e65OPf9Z1h78HB8UVqwWpf21mBSBtO8ZsLzPkg\n
L5w0Y/ekCLtNWAVqAjRGRASBZKkcbWnQ/CWLAHmzbHskL1tiwgMv\nX-Developer-Key:
i=noralf@tronnes.org; a=ed25519;\n
pk=0o9is4iddvvlrY3yON5SVtAbgPnVs0LfQsjfqR2Hvz8=\n\nverify:7ad470b4-f531-4632-8093-738d4d3e5d88\n'}
Traceback (most recent call last):
  File "/home/pi/.local/bin/b4", line 8, in <module>
    sys.exit(cmd())
  File "/home/pi/.local/lib/python3.10/site-packages/b4/command.py",
line 341, in cmd
    cmdargs.func(cmdargs)
  File "/home/pi/.local/lib/python3.10/site-packages/b4/command.py",
line 86, in cmd_send
    b4.ez.cmd_send(cmdargs)
  File "/home/pi/.local/lib/python3.10/site-packages/b4/ez.py", line
1103, in cmd_send
    auth_verify(cmdargs)
  File "/home/pi/.local/lib/python3.10/site-packages/b4/ez.py", line
189, in auth_verify
    res = ses.post(endpoint, json=req)
  File "/usr/lib/python3/dist-packages/requests/sessions.py", line 590,
in post
    return self.request('POST', url, data=data, json=json, **kwargs)
  File "/usr/lib/python3/dist-packages/requests/sessions.py", line 528,
in request
    prep = self.prepare_request(req)
  File "/usr/lib/python3/dist-packages/requests/sessions.py", line 456,
in prepare_request
    p.prepare(
  File "/usr/lib/python3/dist-packages/requests/models.py", line 319, in
prepare
    self.prepare_body(data, files, json)
  File "/usr/lib/python3/dist-packages/requests/models.py", line 469, in
prepare_body
    body = complexjson.dumps(json)
  File "/usr/lib/python3.10/json/__init__.py", line 231, in dumps
    return _default_encoder.encode(obj)
  File "/usr/lib/python3.10/json/encoder.py", line 199, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "/usr/lib/python3.10/json/encoder.py", line 257, in iterencode
    return _iterencode(o, 0)
  File "/usr/lib/python3.10/json/encoder.py", line 179, in default
    raise TypeError(f'Object of type {o.__class__.__name__} '
TypeError: Object of type bytes is not JSON serializable

Noralf.

Re: Error using b4 web endpoint (was Re: git send-email friendly smtp provider anyone?)

[Konstantin Ryabitsev]

On Tue, Nov 22, 2022 at 08:48:27PM +0100, Noralf Trønnes wrote:
> > This is very strange, because this is literally the code in question:
> > 
> >     req = {
> >         'action': 'auth-verify',
> >         'msg': bdata.encode(),
> >     }
> >     logger.info('Submitting verification to %s', endpoint)
> >     ses = b4.get_requests_session()
> >     res = ses.post(endpoint, json=req)
> > 
> > I don't see where "Object of type bytes" would come from. Can you possibly
> > insert the following above the `res = ses.post` line:
> > 
> >     logger.debug('endpoint=%s, req=%s', endpoint, req)
> > 
> 
> $ b4 -d send --web-auth-verify <challenge>
> Running git --no-pager rev-parse --show-toplevel
> Running git --no-pager config -z --get-regexp b4\..*
> Running git --no-pager config -z --get-regexp gpg\..*
> Running git --no-pager config -z --get-regexp user\..*
> Signing challenge
> Submitting verification to https://lkml.kernel.org/_b4_submit
> endpoint=https://lkml.kernel.org/_b4_submit, req={'action':
> 'auth-verify', 'msg': b'From: noralf@tronnes.org\nSubject:
> b4-send-verify\nX-Developer-Signature: v=1; a=ed25519-sha256;
> t=1669146435; l=45;\n i=noralf@tronnes.org; s=20221122;
> h=from:subject;\n bh=E4BgU7G2i7CeIkR8F1AfDTtOvDBQA7ld22s0U29/H84=;\n
> b=eRcDEBQvE702NSqEWQDLRyp9emSRvBjdhIW1c8e65OPf9Z1h78HB8UVqwWpf21mBSBtO8ZsLzPkg\n
> L5w0Y/ekCLtNWAVqAjRGRASBZKkcbWnQ/CWLAHmzbHskL1tiwgMv\nX-Developer-Key:
> i=noralf@tronnes.org; a=ed25519;\n
> pk=0o9is4iddvvlrY3yON5SVtAbgPnVs0LfQsjfqR2Hvz8=\n\nverify:7ad470b4-f531-4632-8093-738d4d3e5d88\n'}

Heh, woops, I got turned around with encode() and decode() again -- we do put
bytes there, but it works for me and it doesn't work for you despite the same
Python version. Wonderful.

I'm guessing it's probably something python-requests handles behind the
scenes. What version is python3-requests for you?

-K

Re: Error using b4 web endpoint (was Re: git send-email friendly smtp provider anyone?)

[Noralf Trønnes]

Den 22.11.2022 21.00, skrev Konstantin Ryabitsev:
> On Tue, Nov 22, 2022 at 08:48:27PM +0100, Noralf Trønnes wrote:
>>> This is very strange, because this is literally the code in question:
>>>
>>>     req = {
>>>         'action': 'auth-verify',
>>>         'msg': bdata.encode(),
>>>     }
>>>     logger.info('Submitting verification to %s', endpoint)
>>>     ses = b4.get_requests_session()
>>>     res = ses.post(endpoint, json=req)
>>>
>>> I don't see where "Object of type bytes" would come from. Can you possibly
>>> insert the following above the `res = ses.post` line:
>>>
>>>     logger.debug('endpoint=%s, req=%s', endpoint, req)
>>>
>>
>> $ b4 -d send --web-auth-verify <challenge>
>> Running git --no-pager rev-parse --show-toplevel
>> Running git --no-pager config -z --get-regexp b4\..*
>> Running git --no-pager config -z --get-regexp gpg\..*
>> Running git --no-pager config -z --get-regexp user\..*
>> Signing challenge
>> Submitting verification to https://lkml.kernel.org/_b4_submit
>> endpoint=https://lkml.kernel.org/_b4_submit, req={'action':
>> 'auth-verify', 'msg': b'From: noralf@tronnes.org\nSubject:
>> b4-send-verify\nX-Developer-Signature: v=1; a=ed25519-sha256;
>> t=1669146435; l=45;\n i=noralf@tronnes.org; s=20221122;
>> h=from:subject;\n bh=E4BgU7G2i7CeIkR8F1AfDTtOvDBQA7ld22s0U29/H84=;\n
>> b=eRcDEBQvE702NSqEWQDLRyp9emSRvBjdhIW1c8e65OPf9Z1h78HB8UVqwWpf21mBSBtO8ZsLzPkg\n
>> L5w0Y/ekCLtNWAVqAjRGRASBZKkcbWnQ/CWLAHmzbHskL1tiwgMv\nX-Developer-Key:
>> i=noralf@tronnes.org; a=ed25519;\n
>> pk=0o9is4iddvvlrY3yON5SVtAbgPnVs0LfQsjfqR2Hvz8=\n\nverify:7ad470b4-f531-4632-8093-738d4d3e5d88\n'}
> 
> Heh, woops, I got turned around with encode() and decode() again -- we do put
> bytes there, but it works for me and it doesn't work for you despite the same
> Python version. Wonderful.
> 
> I'm guessing it's probably something python-requests handles behind the
> scenes. What version is python3-requests for you?
> 

$ apt show python3-requests
Package: python3-requests
Version: 2.25.1+dfsg-2
Priority: optional
Section: python
Source: requests
Origin: Ubuntu

Re: Error using b4 web endpoint (was Re: git send-email friendly smtp provider anyone?)

[Konstantin Ryabitsev]

On Tue, Nov 22, 2022 at 09:10:40PM +0100, Noralf Trønnes wrote:
> > I'm guessing it's probably something python-requests handles behind the
> > scenes. What version is python3-requests for you?
> > 
> 
> $ apt show python3-requests
> Package: python3-requests
> Version: 2.25.1+dfsg-2

OK, this didn't get us closer to the culprit. :( This is the code that works
for me on Fedora 36, 37, RHEL 7, RHEL 8, and Ubuntu 20.04:

    #!/usr/bin/env python3
    import requests
    print('version: %s' % requests.__version__)
    jdata = {'foo': b'bar', 'action': 'test'}
    res = requests.post('https://lkml.kernel.org/_b4_submit', json=jdata)
    print(res.text)

E.g. on Ubuntu-20.04:

    mricon@i7:~$ lsb_release -ir
    Distributor ID: Ubuntu
    Release:        20.04
    mricon@i7:~$ python3 /tmp/testme.py
    version: 2.25.1
    Unknown action: test

However, I see that it's failing on Debian 11 with the JSON error.
I'll poke some more now that I have a test env and follow up.

-K

Re: Error using b4 web endpoint (was Re: git send-email friendly smtp provider anyone?)

[Noralf Trønnes]

Den 22.11.2022 21.20, skrev Konstantin Ryabitsev:
> On Tue, Nov 22, 2022 at 09:10:40PM +0100, Noralf Trønnes wrote:
>>> I'm guessing it's probably something python-requests handles behind the
>>> scenes. What version is python3-requests for you?
>>>
>>
>> $ apt show python3-requests
>> Package: python3-requests
>> Version: 2.25.1+dfsg-2
> 
> OK, this didn't get us closer to the culprit. :( This is the code that works
> for me on Fedora 36, 37, RHEL 7, RHEL 8, and Ubuntu 20.04:
> 
>     #!/usr/bin/env python3
>     import requests
>     print('version: %s' % requests.__version__)
>     jdata = {'foo': b'bar', 'action': 'test'}
>     res = requests.post('https://lkml.kernel.org/_b4_submit', json=jdata)
>     print(res.text)
> 
> E.g. on Ubuntu-20.04:
> 
>     mricon@i7:~$ lsb_release -ir
>     Distributor ID: Ubuntu
>     Release:        20.04
>     mricon@i7:~$ python3 /tmp/testme.py
>     version: 2.25.1
>     Unknown action: test
> 
> However, I see that it's failing on Debian 11 with the JSON error.
> I'll poke some more now that I have a test env and follow up.
> 

Yeah that test snippet fails with the same error.

$ lsb_release -ir
Distributor ID: Ubuntu
Release:        22.04

Noralf.

Re: Error using b4 web endpoint (was Re: git send-email friendly smtp provider anyone?)

[Konstantin Ryabitsev]

On Tue, Nov 22, 2022 at 09:26:03PM +0100, Noralf Trønnes wrote:
> > However, I see that it's failing on Debian 11 with the JSON error.
> > I'll poke some more now that I have a test env and follow up.
> > 
> 
> Yeah that test snippet fails with the same error.
> 
> $ lsb_release -ir
> Distributor ID: Ubuntu
> Release:        22.04

Okay, this is not a real fix, but it will get things working for you while I
implement the proper solution:

apt install python3-simplejson

-K

Re: Error using b4 web endpoint (was Re: git send-email friendly smtp provider anyone?)

[Noralf Trønnes]

Den 22.11.2022 21.44, skrev Konstantin Ryabitsev:
> On Tue, Nov 22, 2022 at 09:26:03PM +0100, Noralf Trønnes wrote:
>>> However, I see that it's failing on Debian 11 with the JSON error.
>>> I'll poke some more now that I have a test env and follow up.
>>>
>>
>> Yeah that test snippet fails with the same error.
>>
>> $ lsb_release -ir
>> Distributor ID: Ubuntu
>> Release:        22.04
> 
> Okay, this is not a real fix, but it will get things working for you while I
> implement the proper solution:
> 
> apt install python3-simplejson
> 

Thanks, no errors this time.

Series:
https://lore.kernel.org/dri-devel/20221122-gud-shadow-plane-v1-0-9de3afa3383e@tronnes.org/

Noralf.

Re: Error using b4 web endpoint (was Re: git send-email friendly smtp provider anyone?)

[Konstantin Ryabitsev]

On Tue, Nov 22, 2022 at 10:03:55PM +0100, Noralf Trønnes wrote:
> Thanks, no errors this time.
> 
> Series:
> https://lore.kernel.org/dri-devel/20221122-gud-shadow-plane-v1-0-9de3afa3383e@tronnes.org/

Great! I also pushed a fix (which consists of merely removing the encode() in
auth_verify) -- I have no idea why I put it there in the first place, but it
was totally unnecessary and only really worked because simplejson
automatically decoded it back into unicode (and python's json just errors
out).

I see that your name got mangled in the From: line -- I don't think
public-inbox is happy with 8bit headers. I'll get it fixed, but it shouldn't
matter for git.

Thank you for being willing to try this out. It's still considered an
experimental feature and you're literally the first to ever use it. :)

-K

Re: Error using b4 web endpoint (was Re: git send-email friendly smtp provider anyone?)

[Konstantin Ryabitsev]

On Tue, Nov 22, 2022 at 10:03:55PM +0100, Noralf Trønnes wrote:
> > On Tue, Nov 22, 2022 at 09:26:03PM +0100, Noralf Trønnes wrote:
> >>> However, I see that it's failing on Debian 11 with the JSON error.
> >>> I'll poke some more now that I have a test env and follow up.
> >>>
> >>
> >> Yeah that test snippet fails with the same error.
> >>
> >> $ lsb_release -ir
> >> Distributor ID: Ubuntu
> >> Release:        22.04
> > 
> > Okay, this is not a real fix, but it will get things working for you while I
> > implement the proper solution:
> > 
> > apt install python3-simplejson
> > 
> 
> Thanks, no errors this time.
> 
> Series:
> https://lore.kernel.org/dri-devel/20221122-gud-shadow-plane-v1-0-9de3afa3383e@tronnes.org/

Sigh. I clearly need to fix the 8bit header issue (ugh, woe!, etc). I'll try
to get this done asap. I'm glad the very first use of the web endpoint we had
is the one where there's 8bit content in the From: header, and we have to
rewrite it with more 8bit content for DMARC purposes.

-K

Re: Error using b4 web endpoint (was Re: git send-email friendly smtp provider anyone?)

[Noralf Trønnes]

Den 22.11.2022 22.03, skrev Noralf Trønnes:
> 
> 
> Den 22.11.2022 21.44, skrev Konstantin Ryabitsev:
>> On Tue, Nov 22, 2022 at 09:26:03PM +0100, Noralf Trønnes wrote:
>>>> However, I see that it's failing on Debian 11 with the JSON error.
>>>> I'll poke some more now that I have a test env and follow up.
>>>>
>>>
>>> Yeah that test snippet fails with the same error.
>>>
>>> $ lsb_release -ir
>>> Distributor ID: Ubuntu
>>> Release:        22.04
>>
>> Okay, this is not a real fix, but it will get things working for you while I
>> implement the proper solution:
>>
>> apt install python3-simplejson
>>
> 
> Thanks, no errors this time.
> 
> Series:
> https://lore.kernel.org/dri-devel/20221122-gud-shadow-plane-v1-0-9de3afa3383e@tronnes.org/
> 

I'm ready to send version 2 of my patchset, but it doesn't work.

$ ~/projects/b4/b4.sh --version
0.11.0-dev-cc6f6

$ ~/projects/b4/b4.sh send
Converted the branch to 7 messages
Populating To/Cc addresses
Will send the following messages:
---
To: Maxime Ripard <mripard@kernel.org>
    Thomas Zimmermann <tzimmermann@suse.de>
    dri-devel@lists.freedesktop.org
    Noralf Trønnes <noralf@tronnes.org>
    stable@vger.kernel.org
    Javier Martinez Canillas <javierm@redhat.com>
---
  [PATCH v2 0/6] drm/gud: Use the shadow plane helper
  [PATCH v2 1/6] drm/gud: Fix UBSAN warning
  [PATCH v2 2/6] drm/gud: Don't retry a failed framebuffer flush
  [PATCH v2 3/6] drm/gud: Split up gud_flush_work()
  [PATCH v2 4/6] drm/gud: Prepare buffer for CPU access in gud_flush_work()
  [PATCH v2 5/6] drm/gud: Use the shadow plane helper
  [PATCH v2 6/6] drm/gud: Enable synchronous flushing by default
---
Press Enter to send or Ctrl-C to abort
---
Sending via web endpoint https://lkml.kernel.org/_b4_submit
---
CRITICAL: Was not able to send messages.


Printing some more info:

diff --git a/b4/__init__.py b/b4/__init__.py
index 64b36f6..eb62719 100644
--- a/b4/__init__.py
+++ b/b4/__init__.py
@@ -3294,9 +3294,12 @@ def send_mail(smtp: Union[smtplib.SMTP,
smtplib.SMTP_SSL, None], msgs: Sequence[
             'messages': [x[1].decode() for x in tosend],
         }
         ses = get_requests_session()
+        print('ses:', ses)
         res = ses.post(endpoint, json=req)
+        print('res', res)
         try:
             rdata = res.json()
+            print('rdata', rdata)
             if rdata.get('result') == 'success':
                 return len(tosend)
         except Exception as ex:  # noqa

Gives me this:

---
Sending via web endpoint https://lkml.kernel.org/_b4_submit
ses: <requests.sessions.Session object at 0x7f8da11e6920>
res <Response [500]>
rdata {'title': '500 Internal Server Error'}
---
CRITICAL: Was not able to send messages.

So looks like a server error and b4 is unable to pick up on that.

I went straight for the latest dev version this time, let me know if I
should try 0.10.1 which I used to send the previous patchset.

Noralf.

Re: Error using b4 web endpoint (was Re: git send-email friendly smtp provider anyone?)

[Konstantin Ryabitsev]

On Wed, Nov 30, 2022 at 05:17:44PM +0100, Noralf Trønnes wrote:
> I'm ready to send version 2 of my patchset, but it doesn't work.

Ah, sorry to hear that -- I did make changes to the endpoint, so it's not
really surprising, even though I tried to run a lot of tests to make sure it's
working.

Can you please do `b4 send -o /somedir` and then send me the tarball with the
output of that?

-K

Re: Error using b4 web endpoint (was Re: git send-email friendly smtp provider anyone?)

[Noralf Trønnes]

[-- Attachment #1: Type: text/plain, Size: 484 bytes --]



Den 30.11.2022 17.55, skrev Konstantin Ryabitsev:
> On Wed, Nov 30, 2022 at 05:17:44PM +0100, Noralf Trønnes wrote:
>> I'm ready to send version 2 of my patchset, but it doesn't work.
> 
> Ah, sorry to hear that -- I did make changes to the endpoint, so it's not
> really surprising, even though I tried to run a lot of tests to make sure it's
> working.
> 
> Can you please do `b4 send -o /somedir` and then send me the tarball with the
> output of that?
> 

Here you go.

Noralf.

[-- Attachment #2: b4-send-2.tar.gz --]
[-- Type: application/x-gzip, Size: 9499 bytes --]

Re: Error using b4 web endpoint (was Re: git send-email friendly smtp provider anyone?)

[Konstantin Ryabitsev]

On Wed, Nov 30, 2022 at 06:02:58PM +0100, Noralf Trønnes wrote:
> > Can you please do `b4 send -o /somedir` and then send me the tarball with the
> > output of that?
> > 
> 
> Here you go.

Thank you very much. Can you please try again?

-K

Re: Error using b4 web endpoint (was Re: git send-email friendly smtp provider anyone?)

[Noralf Trønnes]

Den 30.11.2022 20.17, skrev Konstantin Ryabitsev:
> On Wed, Nov 30, 2022 at 06:02:58PM +0100, Noralf Trønnes wrote:
>>> Can you please do `b4 send -o /somedir` and then send me the tarball with the
>>> output of that?
>>>
>>
>> Here you go.
> 
> Thank you very much. Can you please try again?
> 

Thanks, working now:

https://lore.kernel.org/dri-devel/20221122-gud-shadow-plane-v2-0-435037990a83@tronnes.org/T/#t

https://patchwork.freedesktop.org/series/111222/

Is there an easy way for me to use b4 diff before I send out a new
version so I can see the exact changes I've made so I can make sure I've
not messed things up or forgotten to fix up something?

Noralf.

Re: Error using b4 web endpoint (was Re: git send-email friendly smtp provider anyone?)

[Konstantin Ryabitsev]

On Wed, Nov 30, 2022 at 08:34:45PM +0100, Noralf Trønnes wrote:
> Thanks, working now:
> 
> https://lore.kernel.org/dri-devel/20221122-gud-shadow-plane-v2-0-435037990a83@tronnes.org/T/#t

Great. There's still some weird annoying stuff with encoding happening, but it
shouldn't really interfere with anything. You're a great testcase because you
have 8bit content in many headers of the message and it's tripping up a lot of
my assumptions about how Python would handle things internally.

Thank you for being patient!

> https://patchwork.freedesktop.org/series/111222/
> 
> Is there an easy way for me to use b4 diff before I send out a new
> version so I can see the exact changes I've made so I can make sure I've
> not messed things up or forgotten to fix up something?

Yes, if you're using the dev version of b4, you can run:

    b4 prep --compare-to v1

This will give you a range-diff of the current branch against any previously
sent version.

Cheers,
-K

Re: Error using b4 web endpoint (was Re: git send-email friendly smtp provider anyone?)

[Noralf Trønnes]

Den 30.11.2022 20.39, skrev Konstantin Ryabitsev:
> On Wed, Nov 30, 2022 at 08:34:45PM +0100, Noralf Trønnes wrote:
>> Thanks, working now:
>>
>> https://lore.kernel.org/dri-devel/20221122-gud-shadow-plane-v2-0-435037990a83@tronnes.org/T/#t
> 
> Great. There's still some weird annoying stuff with encoding happening, but it
> shouldn't really interfere with anything. You're a great testcase because you
> have 8bit content in many headers of the message and it's tripping up a lot of
> my assumptions about how Python would handle things internally.
> 
> Thank you for being patient!
> 

No problem, you fix this so fast. If it had taken many days, then I
would have had to find another way to send the patches.

It's also rewarding to help make this work, I guess more and more people
will have problem using git send-mail as email providers and ISP's are
tightening security.

>> https://patchwork.freedesktop.org/series/111222/
>>
>> Is there an easy way for me to use b4 diff before I send out a new
>> version so I can see the exact changes I've made so I can make sure I've
>> not messed things up or forgotten to fix up something?
> 
> Yes, if you're using the dev version of b4, you can run:
> 
>     b4 prep --compare-to v1
> 
> This will give you a range-diff of the current branch against any previously
> sent version.
> 

Excellent, just what I was looking for!

Noralf.

Re: Error using b4 web endpoint (was Re: git send-email friendly smtp provider anyone?)

[Konstantin Ryabitsev]

On Wed, 30 Nov 2022 at 14:39, Konstantin Ryabitsev
<konstantin@linuxfoundation.org> wrote:
>
> On Wed, Nov 30, 2022 at 08:34:45PM +0100, Noralf Trønnes wrote:
> > Thanks, working now:
> >
> > https://lore.kernel.org/dri-devel/20221122-gud-shadow-plane-v2-0-435037990a83@tronnes.org/T/#t
>
> Great. There's still some weird annoying stuff with encoding happening, but it
> shouldn't really interfere with anything. You're a great testcase because you
> have 8bit content in many headers of the message and it's tripping up a lot of
> my assumptions about how Python would handle things internally.

For the record, this helped identify a bug in Python:
https://github.com/python/cpython/issues/99927

I'll have to work around it in b4, unfortunately, as even if it's
fixed in 3.11, I'm not sure it'll be backported properly to earlier
versions.

-K

Re: git send-email friendly smtp provider anyone?

[Noralf Trønnes]

Den 22.11.2022 20.22, skrev Noralf Trønnes:
> 
> 
> Den 22.11.2022 19.50, skrev Konstantin Ryabitsev:
>> On Tue, Nov 22, 2022 at 06:42:19PM +0100, Noralf Trønnes wrote:
>>> The first thing that strikes me is that everyone mentioned in one of the
>>> patches get the entire patchset, even stable@vger.kernel.org (cc'ed in a
>>> fixes patch). The first patch touches a core file and as a result a few
>>> drivers, so I've cc'ed the driver maintainers in that patch, but now
>>> they get the entire patchset where 5 of 6 patches is about a driver that
>>> I maintain. So from their point of view, they see a patchset about a
>>> driver they don't care about and a patch touching a core file, but from
>>> the subject it's not apparent that it touches their driver. I'm afraid
>>> that this might result in none of them looking at that patch. In this
>>> particular case it's not that important, but in another case it might be.
>>
>> I did some (unscientific) polling among kernel maintainers and, by a vast
>> margin, they always prefer to receive the entire series instead of
>> cherry-picked patches -- having the entire series helps provide important
>> context for the change they are looking at.
>>
>> So, this is deliberate and, for now at least, not configurable. Unless you're
>> sending 100+ patch series, I doubt anyone will have any problem with receiving
>> the whole series instead of individual patches.
>>
>>> As for the setting up the web endpoint, should I just follow the b4 docs
>>> on that?
>>>
>>> I use b4 version 0.10.1, is that recent enough?
>>
>> Yes. There will be a 0.10.2 in the near future, but the incoming fixes
>> shouldn't make much difference for the b4 send code.
>>
> 
> This is what I got:
> 
> $ b4 send --web-auth-verify <challenge string from email>
> Signing challenge
> Submitting verification to https://lkml.kernel.org/_b4_submit
> Traceback (most recent call last):
>   File "/home/pi/.local/bin/b4", line 8, in <module>
>     sys.exit(cmd())
>   File "/home/pi/.local/lib/python3.10/site-packages/b4/command.py",
> line 341, in cmd
>     cmdargs.func(cmdargs)
>   File "/home/pi/.local/lib/python3.10/site-packages/b4/command.py",
> line 86, in cmd_send
>     b4.ez.cmd_send(cmdargs)
>   File "/home/pi/.local/lib/python3.10/site-packages/b4/ez.py", line
> 1102, in cmd_send
>     auth_verify(cmdargs)
>   File "/home/pi/.local/lib/python3.10/site-packages/b4/ez.py", line
> 188, in auth_verify
>     res = ses.post(endpoint, json=req)
>   File "/usr/lib/python3/dist-packages/requests/sessions.py", line 590,
> in post
>     return self.request('POST', url, data=data, json=json, **kwargs)
>   File "/usr/lib/python3/dist-packages/requests/sessions.py", line 528,
> in request
>     prep = self.prepare_request(req)
>   File "/usr/lib/python3/dist-packages/requests/sessions.py", line 456,
> in prepare_request
>     p.prepare(
>   File "/usr/lib/python3/dist-packages/requests/models.py", line 319, in
> prepare
>     self.prepare_body(data, files, json)
>   File "/usr/lib/python3/dist-packages/requests/models.py", line 469, in
> prepare_body
>     body = complexjson.dumps(json)
>   File "/usr/lib/python3.10/json/__init__.py", line 231, in dumps
>     return _default_encoder.encode(obj)
>   File "/usr/lib/python3.10/json/encoder.py", line 199, in encode
>     chunks = self.iterencode(o, _one_shot=True)
>   File "/usr/lib/python3.10/json/encoder.py", line 257, in iterencode
>     return _iterencode(o, 0)
>   File "/usr/lib/python3.10/json/encoder.py", line 179, in default
>     raise TypeError(f'Object of type {o.__class__.__name__} '
> TypeError: Object of type bytes is not JSON serializable
> 

Konstantin found a workaround, so I was able to push the patches.

Here's the result if anyone is interested in seeing the result of using
b4 and the web endpoint:
https://lore.kernel.org/dri-devel/20221122-gud-shadow-plane-v1-0-9de3afa3383e@tronnes.org/

Patchwork gave me a new submitter ID:
https://patchwork.freedesktop.org/series/111222/

Noralf.

Re: git send-email friendly smtp provider anyone?

[Noralf Trønnes]

Den 22.11.2022 20.22, skrev Noralf Trønnes:
> 
> 
> Den 22.11.2022 19.50, skrev Konstantin Ryabitsev:
>> On Tue, Nov 22, 2022 at 06:42:19PM +0100, Noralf Trønnes wrote:
>>> The first thing that strikes me is that everyone mentioned in one of the
>>> patches get the entire patchset, even stable@vger.kernel.org (cc'ed in a
>>> fixes patch). The first patch touches a core file and as a result a few
>>> drivers, so I've cc'ed the driver maintainers in that patch, but now
>>> they get the entire patchset where 5 of 6 patches is about a driver that
>>> I maintain. So from their point of view, they see a patchset about a
>>> driver they don't care about and a patch touching a core file, but from
>>> the subject it's not apparent that it touches their driver. I'm afraid
>>> that this might result in none of them looking at that patch. In this
>>> particular case it's not that important, but in another case it might be.
>>
>> I did some (unscientific) polling among kernel maintainers and, by a vast
>> margin, they always prefer to receive the entire series instead of
>> cherry-picked patches -- having the entire series helps provide important
>> context for the change they are looking at.
>>
>> So, this is deliberate and, for now at least, not configurable. Unless you're
>> sending 100+ patch series, I doubt anyone will have any problem with receiving
>> the whole series instead of individual patches.
>>
>>> As for the setting up the web endpoint, should I just follow the b4 docs
>>> on that?
>>>
>>> I use b4 version 0.10.1, is that recent enough?
>>
>> Yes. There will be a 0.10.2 in the near future, but the incoming fixes
>> shouldn't make much difference for the b4 send code.
>>
> 
> This is what I got:
> 
> $ b4 send --web-auth-verify <challenge string from email>
> Signing challenge
> Submitting verification to https://lkml.kernel.org/_b4_submit
> Traceback (most recent call last):
>   File "/home/pi/.local/bin/b4", line 8, in <module>
>     sys.exit(cmd())
>   File "/home/pi/.local/lib/python3.10/site-packages/b4/command.py",
> line 341, in cmd
>     cmdargs.func(cmdargs)
>   File "/home/pi/.local/lib/python3.10/site-packages/b4/command.py",
> line 86, in cmd_send
>     b4.ez.cmd_send(cmdargs)
>   File "/home/pi/.local/lib/python3.10/site-packages/b4/ez.py", line
> 1102, in cmd_send
>     auth_verify(cmdargs)
>   File "/home/pi/.local/lib/python3.10/site-packages/b4/ez.py", line
> 188, in auth_verify
>     res = ses.post(endpoint, json=req)
>   File "/usr/lib/python3/dist-packages/requests/sessions.py", line 590,
> in post
>     return self.request('POST', url, data=data, json=json, **kwargs)
>   File "/usr/lib/python3/dist-packages/requests/sessions.py", line 528,
> in request
>     prep = self.prepare_request(req)
>   File "/usr/lib/python3/dist-packages/requests/sessions.py", line 456,
> in prepare_request
>     p.prepare(
>   File "/usr/lib/python3/dist-packages/requests/models.py", line 319, in
> prepare
>     self.prepare_body(data, files, json)
>   File "/usr/lib/python3/dist-packages/requests/models.py", line 469, in
> prepare_body
>     body = complexjson.dumps(json)
>   File "/usr/lib/python3.10/json/__init__.py", line 231, in dumps
>     return _default_encoder.encode(obj)
>   File "/usr/lib/python3.10/json/encoder.py", line 199, in encode
>     chunks = self.iterencode(o, _one_shot=True)
>   File "/usr/lib/python3.10/json/encoder.py", line 257, in iterencode
>     return _iterencode(o, 0)
>   File "/usr/lib/python3.10/json/encoder.py", line 179, in default
>     raise TypeError(f'Object of type {o.__class__.__name__} '
> TypeError: Object of type bytes is not JSON serializable
> 

Konstantin found a workaround, so I was able to push the patches.

Here's the result if anyone is interested in seeing the result of using
b4 and the web endpoint:
https://lore.kernel.org/dri-devel/20221122-gud-shadow-plane-v1-0-9de3afa3383e@tronnes.org/

Patchwork gave me a new submitter ID:
https://patchwork.freedesktop.org/series/111222/

Noralf.

Re: git send-email friendly smtp provider anyone?

[Konstantin Ryabitsev]

On Tue, Nov 22, 2022 at 10:10:47PM +0100, Noralf Trønnes wrote:
> Konstantin found a workaround, so I was able to push the patches.

Yes, this uncovered quite a few bugs -- which is excellent for me, not so
excellent for you. :)

> Here's the result if anyone is interested in seeing the result of using
> b4 and the web endpoint:
> https://lore.kernel.org/dri-devel/20221122-gud-shadow-plane-v1-0-9de3afa3383e@tronnes.org/
> 
> Patchwork gave me a new submitter ID:
> https://patchwork.freedesktop.org/series/111222/

Oooh, I see that patchwork is still not doing the right thing with
X-Original-From. It will only do the substitution when the From email address
is the same as the email address of the list.

https://github.com/getpatchwork/patchwork/blob/main/patchwork/parser.py#L437

There's unfortunately no fix for this that I can do on my end. :(

-K

Re: git send-email friendly smtp provider anyone?

[Konstantin Ryabitsev]

On Tue, Nov 22, 2022 at 10:10:47PM +0100, Noralf Trønnes wrote:
> Konstantin found a workaround, so I was able to push the patches.

Yes, this uncovered quite a few bugs -- which is excellent for me, not so
excellent for you. :)

> Here's the result if anyone is interested in seeing the result of using
> b4 and the web endpoint:
> https://lore.kernel.org/dri-devel/20221122-gud-shadow-plane-v1-0-9de3afa3383e@tronnes.org/
> 
> Patchwork gave me a new submitter ID:
> https://patchwork.freedesktop.org/series/111222/

Oooh, I see that patchwork is still not doing the right thing with
X-Original-From. It will only do the substitution when the From email address
is the same as the email address of the list.

https://github.com/getpatchwork/patchwork/blob/main/patchwork/parser.py#L437

There's unfortunately no fix for this that I can do on my end. :(

-K