From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-14.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id C12ACC47098 for ; Thu, 3 Jun 2021 17:18:20 +0000 (UTC) Received: by mail.kernel.org (Postfix) id 9DCA761242; Thu, 3 Jun 2021 17:18:20 +0000 (UTC) Received: from out2.migadu.com (out2.migadu.com [188.165.223.204]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 09E0D6108E for ; Thu, 3 Jun 2021 17:18:19 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 09E0D6108E Authentication-Results: mail.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=konstantin.ryabitsev@linux.dev X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1622740698; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=gLyUsLqWIJoto1XObpOLCpkIUnKmpzf+tk4SvNREbuI=; b=oH4jZO2PgIAFC4ZT1KTu4bEwKPuo91RkerfaaMfZp5KDZBSsCa71PSz+/PxrG9obAee/3q kvIScBh3Oyoh8D9GbbMXWUxT01OLfeRVFHXgjn7sJrj9ZjrWLiensoZJh4+rjQigp2wS8X Z/lYro3Sh0j5wGYHAETOx8M7K/1DuAU= From: Konstantin Ryabitsev List-Id: To: signatures@kernel.org Subject: [PATCH 3/5] Handle MIME encoded-word & other header manglings Date: Thu, 3 Jun 2021 13:18:13 -0400 Message-Id: <30840a6acae935ebc5332d08d61222cebe0b518b.1622740672.git.konstantin.ryabitsev@linux.dev> In-Reply-To: <7754d7d35d03b462109c4a93d625f0af21383312.1622740672.git.konstantin.ryabitsev@linux.dev> References: <7754d7d35d03b462109c4a93d625f0af21383312.1622740672.git.konstantin.ryabitsev@linux.dev> MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=4513; i=konstantin.ryabitsev@linux.dev; h=from:subject:message-id; bh=NUs0mDymXdHNFbuTlbCR+n7aZ2Wwdr7ePm6u3KdAJUA=; b=owGbwMvMwCG27YjM47CUmTmMp9WSGBJ28l13nVIq+3napisMAk61bu8cu3q0ZgpN8sqtushc/z8p isOlo5SFQYyDQVZMkaVsX+ymoMKHHnLpPaYwc1iZQIYwcHEKwET0dzIyPN5+sDLQu+i5WfvZrXGF/Q earruLTS6w+9xxXOoXV+pPdkaGezr8c/fYKx6xFfC7leNhdYNVznQDx4/UlYZ8Tz2q8935AA== X-Developer-Key: i=konstantin.ryabitsev@linux.dev; a=openpgp; fpr=DE0E66E32F1FDD0902666B96E63EDCA9329DD07E Content-Transfer-Encoding: 8bit X-Migadu-Flow: FLOW_OUT X-Migadu-Auth-User: konstantin.ryabitsev@linux.dev From: Paul Barker When testing patatt with patches sent to a sr.ht hosted mailing list, it was found that long header lines (such as the X-Developer-Signature line) were re-encoded using the MIME encoded-word syntax (RFC 2047) when an mbox archive is generated, causing patatt to choke on the resulting text which looks like this: X-Developer-Signature: v=1; a=openpgp-sha256; l=672; h=from:subject; bh=C40yOKgIfnNIUP+OW9WyPdBfljkZPpfUL1NepOODlx8=; =?utf-8?q?b=3DowGbwMvMwCF2?= =?utf-8?q?w7xIXuiX9CvG02pJDAmb67lTNi0+IeF97TL76vtKD7xjSjaluz0o/KfmZLX8rMi7_?= =?utf-8?q?l3M6O0pZGMQ4GGTFFFl2z951+fqDJVt7b0gHw8xhZQIZwsDFKQATydFhZJi+fFfvJ?= =?utf-8?q?8+0MF7GrfzWnP?= K7mAM/3n/r/UC+bprf6/g114QYGdbHcsaK7b1nanfA4IeZi1V0lL26cruXUWxgSEnNDP1FrAA= Avoiding this issue by neatly wrapping the X-Developer-Signature header before sending doesn't appear to be possible without making invasive changes to git-send-email and/or the Net::SMTP perl module. The header content generated by patatt is wrapped at 78 characters as can be seen here from a locally signed patch file: X-Developer-Signature: v=1; a=openpgp-sha256; l=672; h=from:subject; bh=C40yOKgIfnNIUP+OW9WyPdBfljkZPpfUL1NepOODlx8=; b=owGbwMvMwCF2w7xIXuiX9CvG02pJDAmbN1xO2bT4hIT3tcvsq+8rPfCOKdmU7vag8J+ak9XysyLv Xs7p7ChlYRDjYJAVU2TZPXvX5esPlmztvSEdDDOHlQlkCAMXpwBMpG0Dw/9Kpzgpc8UsQwOPK/taW6 dFnZyy5QlXPfNCC4WTc76ft9ZnZJjI37a17fP7sxvclKJ1tm36EhITcK62Pphje9KrmOxMJg4A Running `git send-email --smtp-debug=1 0001.patch` shows that this is joined into a single long line before the message is sent: Net::SMTP::_SSL=GLOB(0x5646fbdc3ac8)>>> X-Developer-Signature: v=1; a=openpgp-sha256; l=672; h=from:subject; bh=C40yOKgIfnNIUP+OW9WyPdBfljkZPpfUL1NepOODlx8=; b=owGbwMvMwCF2w7xIXuiX9CvG02pJDAmb571P2bT4hIT3tcvsq+8rPfCOKdmU7vag8J+ak9XysyLv Xs7p7ChlYRDjYJAVU2TZPXvX5esPlmztvSEdDDOHlQlkCAMXpwBM5JA3I8O5hP6Tqm7lJst0rldcux 1V7M4q8T5o1fPU6Zs+hxj+SjvN8D/DK3rn8b0m34/Xy388Yeu8jvFdJf/c6Y6LDU7Hulj01nAAAA== So we need to accept that the X-Developer-Signature line may be quite long and so may be re-encoded by a mail server or archiver. The Python email.header module provides the decode_header() and make_header() functions which can be used to handle MIME encoded-word syntax or other header manglings which may occur. The decode_header() function requires a str argument so we must decode our bytes before using this function. Thankfully, RFC 2822 makes life easy here as it says that all header content must be composed of US-ASCII characters (see section 2.2 of the RFC) so decoding is straightforward. The header content is re-encoded into bytes after un-mangling to avoid having to modify every other location in patatt where the header content is accessed. Signed-off-by: Paul Barker Signed-off-by: Konstantin Ryabitsev Link: https://lore.kernel.org/r/20210531140539.7630-1-paul@pbarker.dev --- patatt/__init__.py | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-) diff --git a/patatt/__init__.py b/patatt/__init__.py index 460d282..b4018ab 100644 --- a/patatt/__init__.py +++ b/patatt/__init__.py @@ -91,7 +91,7 @@ class DevsigHeader: def from_bytes(self, hval: bytes) -> None: self.hval = DevsigHeader._dkim_canonicalize_header(hval) - hval = re.sub(rb'\s*', b'', hval) + hval = re.sub(rb'\s*', b'', self.hval) for chunk in hval.split(b';'): parts = chunk.split(b'=', 1) if len(parts) < 2: @@ -392,6 +392,15 @@ class DevsigHeader: @staticmethod def _dkim_canonicalize_header(hval: bytes) -> bytes: + # Handle MIME encoded-word syntax or other types of header encoding if + # present. The decode_header() function requires a str argument (not + # bytes) so we must decode our bytes first, this is easy as RFC2822 (sec + # 2.2) says header fields must be composed of US-ASCII characters. The + # resulting string is re-encoded to allow further processing. + if b'?q?' in hval: + hval = hval.decode('ascii', errors='ignore') + hval = str(email.header.make_header(email.header.decode_header(hval))) + hval = hval.encode('utf-8') # We only do relaxed for headers # o Unfold all header field continuation lines as described in # [RFC5322]; in particular, lines with terminators embedded in -- 2.31.1