From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-10.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,HTML_MESSAGE,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 19CFCC433F5 for ; Thu, 16 Sep 2021 13:45:08 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 9E9C760F38 for ; Thu, 16 Sep 2021 13:45:07 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 9E9C760F38 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=nongnu.org Received: from localhost ([::1]:60990 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mQrhK-0007zo-KB for qemu-devel@archiver.kernel.org; Thu, 16 Sep 2021 09:45:06 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:35516) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mQrfG-0005gQ-OZ for qemu-devel@nongnu.org; Thu, 16 Sep 2021 09:42:58 -0400 Received: from us-smtp-delivery-124.mimecast.com ([216.205.24.124]:41712) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mQrfB-0000FN-Q7 for qemu-devel@nongnu.org; Thu, 16 Sep 2021 09:42:57 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1631799771; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=HPiUXY4+c3KhodmENffwvEWaABK/LRXjEo6QDhno+2Y=; b=Bn3OmPZOA/pUnp4m1ab+bh+MInlQr/KOLcl/Dkg9MuKo469FOCy212cuKso/pb5j7IQPTY egBDpJcpu5ANJgDnKhXB7+umJrM+7E55HxKkaD/t4sbdKxYX5HsD+ypFqyD1/YZ1nViMos Z/yZhOFK9e6r9Cxms361h+weinGOIc4= Received: from mail-ot1-f72.google.com (mail-ot1-f72.google.com [209.85.210.72]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-213-3un-lICHPvWPfhv9-m9hHA-1; Thu, 16 Sep 2021 09:42:42 -0400 X-MC-Unique: 3un-lICHPvWPfhv9-m9hHA-1 Received: by mail-ot1-f72.google.com with SMTP id r10-20020a056830448a00b0051b9c05a2a8so26720899otv.2 for ; Thu, 16 Sep 2021 06:42:42 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=HPiUXY4+c3KhodmENffwvEWaABK/LRXjEo6QDhno+2Y=; b=bIx27CuwcEVYryzBFqk12fyC9LjCKt9aWQFL0rVuuWzWHi1AgCgDQDK8H8sfXvnKeY HaN0yMUks/+hyc4QVNxlzgcMVHXHpmth8N4pGcJQU3ZowKELeh+SeJsGJe+LczhAP0fE g1bLt5yKKh4ouzQ5p3Mv230Ja1MZjCsFPZ/8cAr/wjbbqZWW4pGf8sj+wEe73H8Rx77i MmZwYG3Z6hSM0jYwplqNoQAooPY4DYdPus/cPUYiQ/0cUvOqCk3jnxuTma8XcgdAue1I 6rZZi4ahKOkl9CMnVPAG+Pt8fegSOBSk7wshvK6qtBmkndQLQTQ3zQFEy0t2WvCI1d8Z xMZQ== X-Gm-Message-State: AOAM532IsKRg5RpHrCrb8sNS6WSOpsTnzX6+WWL1Nv5cUi/j5LWyqk9N McBa/UtC+wa3teOxbwpfBEF43466OT8M5XUC7eO0oCJD/+trzN91zmiEAJ4CkwYoq0Rn9lwyYhO 1xVKUbAmdlAXY3VQ+dT3e9NCO3qzlp14= X-Received: by 2002:a05:6830:310c:: with SMTP id b12mr1297713ots.252.1631799762141; Thu, 16 Sep 2021 06:42:42 -0700 (PDT) X-Google-Smtp-Source: ABdhPJwOjz/15BhHeHfcoz8jUi2HvwBb7cRJXgMyHDWCQMap6MRj7cTdgJPrhdij69VZh1E8/TB/54ApJ4C+YNVIVqQ= X-Received: by 2002:a05:6830:310c:: with SMTP id b12mr1297694ots.252.1631799761884; Thu, 16 Sep 2021 06:42:41 -0700 (PDT) MIME-Version: 1.0 References: <20210915154031.321592-1-jsnow@redhat.com> <20210915154031.321592-2-jsnow@redhat.com> In-Reply-To: From: John Snow Date: Thu, 16 Sep 2021 09:42:30 -0400 Message-ID: Subject: Re: [PATCH v3 1/1] python: Update for pylint 2.10 To: =?UTF-8?Q?Daniel_P=2E_Berrang=C3=A9?= Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=jsnow@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Type: multipart/alternative; boundary="00000000000026e97a05cc1cfdf2" Received-SPF: pass client-ip=216.205.24.124; envelope-from=jsnow@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -31 X-Spam_score: -3.2 X-Spam_bar: --- X-Spam_report: (-3.2 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.392, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Peter Maydell , Eduardo Habkost , qemu-devel , G S Niteesh Babu , Cleber Rosa , Eric Blake Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" --00000000000026e97a05cc1cfdf2 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Thu, Sep 16, 2021 at 8:59 AM Daniel P. Berrang=C3=A9 wrote: > On Wed, Sep 15, 2021 at 11:40:31AM -0400, John Snow wrote: > > A few new annoyances. Of note is the new warning for an unspecified > > encoding when opening a text file, which actually does indicate a > > potentially real problem; see > > https://www.python.org/dev/peps/pep-0597/#motivation > > > > Use LC_CTYPE to determine an encoding to use for interpreting QEMU's > > terminal output. Note that Python states: "language code and encoding > > may be None if their values cannot be determined" -- use a platform > > default as a backup. > > > > Signed-off-by: John Snow > > --- > > python/qemu/machine/machine.py | 9 ++++++++- > > python/setup.cfg | 1 + > > 2 files changed, 9 insertions(+), 1 deletion(-) > > > > diff --git a/python/qemu/machine/machine.py > b/python/qemu/machine/machine.py > > index a7081b1845..51b6e79a13 100644 > > --- a/python/qemu/machine/machine.py > > +++ b/python/qemu/machine/machine.py > > @@ -19,6 +19,7 @@ > > > > import errno > > from itertools import chain > > +import locale > > import logging > > import os > > import shutil > > @@ -290,8 +291,14 @@ def get_pid(self) -> Optional[int]: > > return self._subp.pid > > > > def _load_io_log(self) -> None: > > + # Assume that the output encoding of QEMU's terminal output > > + # is defined by our locale. If indeterminate, use a platform > default. > > + _, encoding =3D locale.getlocale() > > + if encoding is None: > > + encoding =3D locale.getpreferredencoding(do_setlocale=3DFa= lse) > > Do we really need this getpreferredencoding ? IIUC, this is a sign > that the application is buggy by not calling > > locale.setlocale(locale.LC_ALL, '') > > during its main() method, which I think we can just delegate to the > code in question to fix. Missing setlocale will affect everything > they run, so doing workarounds in only 1 place is not worth it IMHO > > I genuinely don't know! (And, I try to keep the Python code free from assuming Linux as much as I can help it.) Python's getlocale documentation states: "language code and encoding may be None if their values cannot be determined." https://docs.python.org/3/library/locale.html#locale.getlocale But it is quiet as to the circumstances under which this may happen. Browsing the cpython source code, (3.9ish): ``` def getlocale(category=3DLC_CTYPE): localename =3D _setlocale(category) if category =3D=3D LC_ALL and ';' in localename: raise TypeError('category LC_ALL is not supported') return _parse_localename(localename) ``` _setlocale is ultimately a call to (I think) _localemodule.c's PyLocale_setlocale(PyObject *self, PyObject *args) C function. It calls `result =3D setlocale(category, locale)` where the category is goi= ng to be LC_CTYPE, so this should be equivalent to locale(3) (LC_CTYPE, NULL). locale(3) says that "The return value is NULL if the request cannot be honored." Python parses that string according to _parse_localename, which in turn calls normalize(localename). Normalization looks quite involved, but has a fallback of returning the string verbatim. If the normalized locale string is "C", we return the tuple (None, None)! So I figured there was a non-zero chance that we'd see a value of `None` here. Source code is in cpython/Lib/locale.py and cpython/Modules/_localemodule.c if you want to nose around yourself. I also have no idea how this will all shake out on Windows, so I decided to add the fallback here just in case. (Does the Python package work on Windows? I don't know, but I avoid assuming it won't EVER run there... Certainly, I have an interest in having the QMP packages I am building work on all platforms.) Thoughts? > if self._qemu_log_path is not None: > > - with open(self._qemu_log_path, "r") as iolog: > > + with open(self._qemu_log_path, "r", > > + encoding=3Dencoding) as iolog: > > self._iolog =3D iolog.read() > Regards, > Daniel > -- > |: https://berrange.com -o- > https://www.flickr.com/photos/dberrange :| > |: https://libvirt.org -o- > https://fstop138.berrange.com :| > |: https://entangle-photo.org -o- > https://www.instagram.com/dberrange :| > > --00000000000026e97a05cc1cfdf2 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable


=
On Thu, Sep 16, 2021 at 8:59 AM Danie= l P. Berrang=C3=A9 <berrange@redhat.com> wrote:
On Wed, Sep 15, 2021 at 11:40:31AM -0400, John Snow= wrote:
> A few new annoyances. Of note is the new warning for an unspecified > encoding when opening a text file, which actually does indicate a
> potentially real problem; see
> https://www.python.org/dev/peps/pep-0597/= #motivation
>
> Use LC_CTYPE to determine an encoding to use for interpreting QEMU'= ;s
> terminal output. Note that Python states: "language code and enco= ding
> may be None if their values cannot be determined" -- use a platfo= rm
> default as a backup.
>
> Signed-off-by: John Snow <jsnow@redhat.com>
> ---
>=C2=A0 python/qemu/machine/machine.py | 9 ++++++++-
>=C2=A0 python/setup.cfg=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0= =C2=A0| 1 +
>=C2=A0 2 files changed, 9 insertions(+), 1 deletion(-)
>
> diff --git a/python/qemu/machine/machine.py b/python/qemu/machine/mach= ine.py
> index a7081b1845..51b6e79a13 100644
> --- a/python/qemu/machine/machine.py
> +++ b/python/qemu/machine/machine.py
> @@ -19,6 +19,7 @@
>=C2=A0
>=C2=A0 import errno
>=C2=A0 from itertools import chain
> +import locale
>=C2=A0 import logging
>=C2=A0 import os
>=C2=A0 import shutil
> @@ -290,8 +291,14 @@ def get_pid(self) -> Optional[int]:
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 return self._subp.pid
>=C2=A0
>=C2=A0 =C2=A0 =C2=A0 def _load_io_log(self) -> None:
> +=C2=A0 =C2=A0 =C2=A0 =C2=A0 # Assume that the output encoding of QEMU= 's terminal output
> +=C2=A0 =C2=A0 =C2=A0 =C2=A0 # is defined by our locale. If indetermin= ate, use a platform default.
> +=C2=A0 =C2=A0 =C2=A0 =C2=A0 _, encoding =3D locale.getlocale()
> +=C2=A0 =C2=A0 =C2=A0 =C2=A0 if encoding is None:
> +=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 encoding =3D locale.getpref= erredencoding(do_setlocale=3DFalse)

Do we really need this getpreferredencoding ?=C2=A0 IIUC, this is a sign that the application is buggy by not calling

=C2=A0 locale.setlocale(locale.LC_ALL, '')

during its main() method, which I think we can just delegate to the
code in question to fix. Missing setlocale will affect everything
they run, so doing workarounds in only 1 place is not worth it IMHO


I genuinely don't know! (And, I tr= y to keep the Python code free from assuming Linux as much as I can help it= .)

Python's getlocale documentation states= : "language code and encoding may be None if their values cannot be de= termined."

But it is quiet as to the circum= stances under which this may happen. Browsing the cpython source code, (3.9= ish):

```
def getlocale(category=3DL= C_CTYPE):
=C2=A0 =C2=A0 localename =3D _setlocale(category)
=C2=A0 = =C2=A0 if category =3D=3D LC_ALL and ';' in localename:
=C2=A0 = =C2=A0 =C2=A0 =C2=A0 raise TypeError('category LC_ALL is not supported&= #39;)
=C2=A0 =C2=A0 return _parse_localename(localename)
```
_setlocale is ultimately a call to (I think) _localemodule.c's= PyLocale_setlocale(PyObject *self, PyObject *args) C function.
I= t calls `result =3D setlocale(category, locale)` where the category is goin= g to be LC_CTYPE, so this should be equivalent to locale(3) (LC_CTYPE, NULL= ).

locale(3) says that "The return value is N= ULL if the request cannot be honored."

Python= parses that string according to _parse_localename, which in turn calls nor= malize(localename).
Normalization looks quite involved, but has a= fallback of returning the string verbatim. If the normalized locale string= is "C", we return the tuple (None, None)!

So I figured there was a non-zero chance that we'd see a value of `N= one` here.

Source code is in cpython/Lib/locale.py= and cpython/Modules/_localemodule.c if you want to nose around yourself.

I also have no idea how this will all shake out on = Windows, so I decided to add the fallback here just in case. (Does the Pyth= on package work on Windows? I don't know, but I avoid assuming it won&#= 39;t EVER run there... Certainly, I have an interest in having the QMP pack= ages I am building work on all platforms.)
=C2=A0
T= houghts?

>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 if self._qemu_log_path is not None:<= br> > -=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 with open(self._qemu_log_pa= th, "r") as iolog:
> +=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 with open(self._qemu_log_pa= th, "r",
> +=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0= =C2=A0 encoding=3Dencoding) as iolog:
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 self._io= log =3D iolog.read()
Regards,
Daniel
--
|: ht= tps://berrange.com=C2=A0 =C2=A0 =C2=A0 -o-=C2=A0 =C2=A0 h= ttps://www.flickr.com/photos/dberrange :|
|: htt= ps://libvirt.org=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0-o-=C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 https://fstop138.berrange.com :|
|: https://entangle-photo.org=C2=A0 =C2=A0 -o-=C2=A0 =C2=A0 = https://www.instagram.com/dberrange :|

--00000000000026e97a05cc1cfdf2--