Re: b4: UTF-8 in replies

Re: b4: UTF-8 in replies

[Konstantin Ryabitsev]

On Thu, Oct 01, 2020 at 07:14:05PM +0200, Michał Mirosław wrote:
> I'm writing to you because I can't find a bug tracker for b4 script.

There isn't one. Just email tools@linux.kernel.org (cc'd).

> The bug is that when generating the reply, script doesn't include MIME
> headers but is using UTF-8 in the body (eg. when replying to an author
> with non-US name, like me).

Hm... is that causing you any problems? We make sure to set 
"Content-Transfer-Encoding: 8bit" in the headers, so unless you're using 
some extremely old SMTP software, UTF-8 in the body or headers shouldn't 
matter.

Regards,
-K

Re: [tools] b4: UTF-8 in replies

[Mark Brown]

[-- Attachment #1: Type: text/plain, Size: 610 bytes --]

On Thu, Oct 01, 2020 at 02:05:41PM -0400, Konstantin Ryabitsev wrote:
> On Thu, Oct 01, 2020 at 07:14:05PM +0200, Michał Mirosław wrote:

> > The bug is that when generating the reply, script doesn't include MIME
> > headers but is using UTF-8 in the body (eg. when replying to an author
> > with non-US name, like me).

> Hm... is that causing you any problems? We make sure to set 
> "Content-Transfer-Encoding: 8bit" in the headers, so unless you're using 
> some extremely old SMTP software, UTF-8 in the body or headers shouldn't 
> matter.

vger doesn't like 8 bit content in the headers.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 499 bytes --]

Re: b4: UTF-8 in replies

[Michał Mirosław]

On Thu, Oct 01, 2020 at 02:05:41PM -0400, Konstantin Ryabitsev wrote:
> On Thu, Oct 01, 2020 at 07:14:05PM +0200, Michał Mirosław wrote:
> > I'm writing to you because I can't find a bug tracker for b4 script.
> 
> There isn't one. Just email tools@linux.kernel.org (cc'd).
> 
> > The bug is that when generating the reply, script doesn't include MIME
> > headers but is using UTF-8 in the body (eg. when replying to an author
> > with non-US name, like me).
> 
> Hm... is that causing you any problems? We make sure to set 
> "Content-Transfer-Encoding: 8bit" in the headers, so unless you're using 
> some extremely old SMTP software, UTF-8 in the body or headers shouldn't 
> matter.

At least K9 Mail on Android requires MIME headers to properly decode it.
The additional headers are actually required by RFC 5322 to include
8-bit data in the message.

I guess the fix would be to just include Content-Type (and others) with
charset="utf8".

Best Regards,
Michał Mirosław

Re: b4: UTF-8 in replies

[Michał Mirosław]

On Thu, Oct 01, 2020 at 08:36:47PM +0200, Michał Mirosław wrote:
> On Thu, Oct 01, 2020 at 02:05:41PM -0400, Konstantin Ryabitsev wrote:
> > On Thu, Oct 01, 2020 at 07:14:05PM +0200, Michał Mirosław wrote:
> > > I'm writing to you because I can't find a bug tracker for b4 script.
> > 
> > There isn't one. Just email tools@linux.kernel.org (cc'd).
> > 
> > > The bug is that when generating the reply, script doesn't include MIME
> > > headers but is using UTF-8 in the body (eg. when replying to an author
> > > with non-US name, like me).
> > 
> > Hm... is that causing you any problems? We make sure to set 
> > "Content-Transfer-Encoding: 8bit" in the headers, so unless you're using 
> > some extremely old SMTP software, UTF-8 in the body or headers shouldn't 
> > matter.
> 
> At least K9 Mail on Android requires MIME headers to properly decode it.
> The additional headers are actually required by RFC 5322 to include
> 8-bit data in the message.
> 
> I guess the fix would be to just include Content-Type (and others) with
> charset="utf8".

According to RFC2045 those should be enough:

MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 8bit

Best Regards,
Michał Mirosław

Re: b4: UTF-8 in replies

[Konstantin Ryabitsev]

On Thu, Oct 01, 2020 at 08:41:09PM +0200, Michał Mirosław wrote:
> > At least K9 Mail on Android requires MIME headers to properly decode 
> > it.
> > The additional headers are actually required by RFC 5322 to include
> > 8-bit data in the message.
> > 
> > I guess the fix would be to just include Content-Type (and others) with
> > charset="utf8".
> 
> According to RFC2045 those should be enough:
> 
> MIME-Version: 1.0
> Content-Type: text/plain; charset=utf-8
> Content-Transfer-Encoding: 8bit

Hrmph, apparently my assumption on what headers will be generated by 
policy=8bit was pretty wrong. I pushed a commit to master and 
stable-0.5.y that should properly set the charset and generate the 
necessary headers.

I intend to cut 0.6.0 tomorrow that will incorporate this fix, among 
several others.

Thanks!
-K

Re: [tools] b4: UTF-8 in replies

[Mark Brown]

[-- Attachment #1: Type: text/plain, Size: 1502 bytes --]

On Thu, Oct 01, 2020 at 03:26:00PM -0400, Konstantin Ryabitsev wrote:

> Hrmph, apparently my assumption on what headers will be generated by 
> policy=8bit was pretty wrong. I pushed a commit to master and 
> stable-0.5.y that should properly set the charset and generate the 
> necessary headers.

This fix seems to have caused some issues for me - using msmtp with
kernel.org on 1594818110-786-1-git-send-email-brent.lu@intel.com I'm
getting headers with:

To: alsa-devel@alsa-project.org, Brent Lu <brent.lu@intel.com>
Cc: =?utf-8?q?linux-kernel=40vger=2Ekernel=2Eorg=2C_Kai_Vehmanen_=3Ckai=2Eve?=
 =?utf-8?q?hmanen=40linux=2Eintel=2Ecom=3E=2C_Pierre-Louis_Bossart_=3Cpierre?=
 =?utf-8?q?-louis=2Ebossart=40linux=2Eintel=2Ecom=3E=2C_Takashi_Iwai_=3Ctiwa?=
 =?utf-8?q?i=40suse=2Ecom=3E=2C_Guennadi_Liakhovetski_=3Cguennadi=2Eliakhove?=
 =?utf-8?q?tski=40linux=2Eintel=2Ecom=3E=2C_Liam_Girdwood_=3Clgirdwood=40gma?=
 =?utf-8?q?il=2Ecom=3E=2C_Amadeusz_S=C5=82awi=C5=84ski_=3Camadeuszx=2Eslawin?=
 =?utf-8?q?ski=40linux=2Eintel=2Ecom=3E?=

(in message ID 160167211500.21762.13369928255363041989.b4-ty@kernel.org
which I'll manage to send) which /usr/sbin/sendmail -t from both msmtp
and exim has a lot of trouble parsing and since b4 has generated the
thanks I now can't ask the database (though I don't immediately see a
way to extract it anyway).  I think we need a sidecar file with the
unencoded e-mail addresses in it?
ask b4 what addresses are buried in there even if 

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 499 bytes --]

Re: [tools] b4: UTF-8 in replies

[Michał Mirosław]

On Fri, Oct 02, 2020 at 10:09:50PM +0100, Mark Brown wrote:
> On Thu, Oct 01, 2020 at 03:26:00PM -0400, Konstantin Ryabitsev wrote:
> 
> > Hrmph, apparently my assumption on what headers will be generated by 
> > policy=8bit was pretty wrong. I pushed a commit to master and 
> > stable-0.5.y that should properly set the charset and generate the 
> > necessary headers.
> 
> This fix seems to have caused some issues for me - using msmtp with
> kernel.org on 1594818110-786-1-git-send-email-brent.lu@intel.com I'm
> getting headers with:
> 
> To: alsa-devel@alsa-project.org, Brent Lu <brent.lu@intel.com>
> Cc: =?utf-8?q?linux-kernel=40vger=2Ekernel=2Eorg=2C_Kai_Vehmanen_=3Ckai=2Eve?=
>  =?utf-8?q?hmanen=40linux=2Eintel=2Ecom=3E=2C_Pierre-Louis_Bossart_=3Cpierre?=
>  =?utf-8?q?-louis=2Ebossart=40linux=2Eintel=2Ecom=3E=2C_Takashi_Iwai_=3Ctiwa?=
>  =?utf-8?q?i=40suse=2Ecom=3E=2C_Guennadi_Liakhovetski_=3Cguennadi=2Eliakhove?=
>  =?utf-8?q?tski=40linux=2Eintel=2Ecom=3E=2C_Liam_Girdwood_=3Clgirdwood=40gma?=
>  =?utf-8?q?il=2Ecom=3E=2C_Amadeusz_S=C5=82awi=C5=84ski_=3Camadeuszx=2Eslawin?=
>  =?utf-8?q?ski=40linux=2Eintel=2Ecom=3E?=

RFC 2047 section 6.2. suggests that this encoding is invalid, because it
encodes structural characters (commas in this case) -- the MTA/MUA
might treat the whole 'Cc:' as one entry. Also, section 5 point 3 of the
RFC forbids encoding the addr-spec part (mail address if I understand
correctly).

Best Regards,
Michał Mirosław

Re: [tools] b4: UTF-8 in replies

[Michał Mirosław]

On Sat, Oct 03, 2020 at 12:13:45PM +0200, Michał Mirosław wrote:
> On Fri, Oct 02, 2020 at 10:09:50PM +0100, Mark Brown wrote:
> > On Thu, Oct 01, 2020 at 03:26:00PM -0400, Konstantin Ryabitsev wrote:
> > 
> > > Hrmph, apparently my assumption on what headers will be generated by 
> > > policy=8bit was pretty wrong. I pushed a commit to master and 
> > > stable-0.5.y that should properly set the charset and generate the 
> > > necessary headers.
> > 
> > This fix seems to have caused some issues for me - using msmtp with
> > kernel.org on 1594818110-786-1-git-send-email-brent.lu@intel.com I'm
> > getting headers with:
> > 
> > To: alsa-devel@alsa-project.org, Brent Lu <brent.lu@intel.com>
> > Cc: =?utf-8?q?linux-kernel=40vger=2Ekernel=2Eorg=2C_Kai_Vehmanen_=3Ckai=2Eve?=
> >  =?utf-8?q?hmanen=40linux=2Eintel=2Ecom=3E=2C_Pierre-Louis_Bossart_=3Cpierre?=
> >  =?utf-8?q?-louis=2Ebossart=40linux=2Eintel=2Ecom=3E=2C_Takashi_Iwai_=3Ctiwa?=
> >  =?utf-8?q?i=40suse=2Ecom=3E=2C_Guennadi_Liakhovetski_=3Cguennadi=2Eliakhove?=
> >  =?utf-8?q?tski=40linux=2Eintel=2Ecom=3E=2C_Liam_Girdwood_=3Clgirdwood=40gma?=
> >  =?utf-8?q?il=2Ecom=3E=2C_Amadeusz_S=C5=82awi=C5=84ski_=3Camadeuszx=2Eslawin?=
> >  =?utf-8?q?ski=40linux=2Eintel=2Ecom=3E?=
> 
> RFC 2047 section 6.2. suggests that this encoding is invalid, because it
> encodes structural characters (commas in this case) -- the MTA/MUA
> might treat the whole 'Cc:' as one entry. Also, section 5 point 3 of the
> RFC forbids encoding the addr-spec part (mail address if I understand
> correctly).

IOW, correct encoding might be at most:

Cc: linux-kernel@vger.kernel.org, Kai Vehmanen
	<kai.vehmanen@linux.intel.com>, Pierre-Louis Bossart
	<pierre-louis.bossart@linux.intel.com>, Takashi Iwai
	<tiwai@suse.com>, Guennadi Liakhovetski
	<guennadi.liakhovetski@linux.intel.com>, Liam Girdwood
	<lgirdwood@gmail.com>,
	=?utf-8?q?Amadeusz_S=C5=82awi=C5=84ski?=
	<amadeuszx.slawinski@linux.intel.com>

Best Regards,
Michał Mirosław

Re: [tools] b4: UTF-8 in replies

[Konstantin Ryabitsev]

On Sat, Oct 03, 2020 at 12:18:04PM +0200, Michał Mirosław wrote:
> > RFC 2047 section 6.2. suggests that this encoding is invalid, 
> > because it
> > encodes structural characters (commas in this case) -- the MTA/MUA
> > might treat the whole 'Cc:' as one entry. Also, section 5 point 3 of the
> > RFC forbids encoding the addr-spec part (mail address if I understand
> > correctly).
> 
> IOW, correct encoding might be at most:
> 
> Cc: linux-kernel@vger.kernel.org, Kai Vehmanen
> 	<kai.vehmanen@linux.intel.com>, Pierre-Louis Bossart
> 	<pierre-louis.bossart@linux.intel.com>, Takashi Iwai
> 	<tiwai@suse.com>, Guennadi Liakhovetski
> 	<guennadi.liakhovetski@linux.intel.com>, Liam Girdwood
> 	<lgirdwood@gmail.com>,
> 	=?utf-8?q?Amadeusz_S=C5=82awi=C5=84ski?=
> 	<amadeuszx.slawinski@linux.intel.com>

Please feel free to report it to Python upstream, as we're just 
deferring to whatever decisions they make. Anyway, we're back to 8bit 
headers with the latest change.

-K