[Buildroot] [PATCH buildroot-test] scripts/autobuild-run: make it Python 3.x compatible

[Buildroot] [PATCH buildroot-test] scripts/autobuild-run: make it Python 3.x compatible

[Thomas Petazzoni]

With Python 3.7, the autobuild-run did not work due to the following
issues:

 - The urlparse module no longer exists, it's not urllib.parse

 - 0022 is no longer recognized as an octal value, we must use 0o022,
   which also works in Python 2.x

 - reading the CSV file with the list of branches through the CSV
   parser failed due to the lack of decoding, as urlopen_closing()
   returns a stream of bytes and not strings. So we need to call
   decode_bytes() on each element of the CSV array. Since the CSV file
   is typically 3 or 4 lines long, we don't really need to optimize
   things.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
---
 scripts/autobuild-run | 9 ++++++---
 1 file changed, 6 insertions(+), 3 deletions(-)

diff --git a/scripts/autobuild-run b/scripts/autobuild-run
index 5921edd..e475ea8 100755
--- a/scripts/autobuild-run
+++ b/scripts/autobuild-run
@@ -145,14 +145,15 @@ from distutils.version import StrictVersion
 import platform
 from threading import Thread, Event
 import datetime
-import urlparse
 
 if sys.hexversion >= 0x3000000:
     import configparser
     import urllib.request as _urllib
+    import urllib.parse as urlparse
 else:
     import ConfigParser as configparser
     import urllib2 as _urllib
+    import urlparse
 
 urlopen = _urllib.urlopen
 urlopen_closing = lambda uri: contextlib.closing(urlopen(uri))
@@ -307,8 +308,10 @@ class Builder:
         list. This way, branches with a higher weight are more likely to
         be selected.
         """
+        csv_branches = []
         with urlopen_closing(urlparse.urljoin(self.http_url, 'branches')) as r:
-            csv_branches = r.readlines()
+            for l in r.readlines():
+                csv_branches.append(decode_bytes(l))
         branches = []
         for branch in csv.reader(csv_branches):
             branches += [branch[0]] * int(branch[1])
@@ -830,7 +833,7 @@ def main():
     # Enforce the sanest umask here, to avoid buildroot doing it on its
     # own and causing a double-make call, thus adding extraneous lines
     # in case of failures.
-    os.umask(0022)
+    os.umask(0o022)
 
     def sigterm_handler(signum, frame):
         """Kill all children"""
-- 
2.23.0

[Buildroot] [PATCH buildroot-test] scripts/autobuild-run: make it Python 3.x compatible

[Baruch Siach]

Hi Thomas,

On Tue, Dec 03, 2019 at 05:54:57PM +0100, Thomas Petazzoni wrote:
> With Python 3.7, the autobuild-run did not work due to the following
> issues:
> 
>  - The urlparse module no longer exists, it's not urllib.parse

s/not/now/

> 
>  - 0022 is no longer recognized as an octal value, we must use 0o022,
>    which also works in Python 2.x
> 
>  - reading the CSV file with the list of branches through the CSV
>    parser failed due to the lack of decoding, as urlopen_closing()
>    returns a stream of bytes and not strings. So we need to call
>    decode_bytes() on each element of the CSV array. Since the CSV file
>    is typically 3 or 4 lines long, we don't really need to optimize
>    things.
> 
> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
> ---
>  scripts/autobuild-run | 9 ++++++---
>  1 file changed, 6 insertions(+), 3 deletions(-)
> 
> diff --git a/scripts/autobuild-run b/scripts/autobuild-run
> index 5921edd..e475ea8 100755
> --- a/scripts/autobuild-run
> +++ b/scripts/autobuild-run
> @@ -145,14 +145,15 @@ from distutils.version import StrictVersion
>  import platform
>  from threading import Thread, Event
>  import datetime
> -import urlparse
>  
>  if sys.hexversion >= 0x3000000:
>      import configparser
>      import urllib.request as _urllib
> +    import urllib.parse as urlparse

Would that work with any Python 3.x?

>  else:
>      import ConfigParser as configparser
>      import urllib2 as _urllib
> +    import urlparse
>  
>  urlopen = _urllib.urlopen
>  urlopen_closing = lambda uri: contextlib.closing(urlopen(uri))
> @@ -307,8 +308,10 @@ class Builder:
>          list. This way, branches with a higher weight are more likely to
>          be selected.
>          """
> +        csv_branches = []
>          with urlopen_closing(urlparse.urljoin(self.http_url, 'branches')) as r:
> -            csv_branches = r.readlines()
> +            for l in r.readlines():
> +                csv_branches.append(decode_bytes(l))
>          branches = []
>          for branch in csv.reader(csv_branches):
>              branches += [branch[0]] * int(branch[1])
> @@ -830,7 +833,7 @@ def main():
>      # Enforce the sanest umask here, to avoid buildroot doing it on its
>      # own and causing a double-make call, thus adding extraneous lines
>      # in case of failures.
> -    os.umask(0022)
> +    os.umask(0o022)
>  
>      def sigterm_handler(signum, frame):
>          """Kill all children"""

-- 
     http://baruch.siach.name/blog/                  ~. .~   Tk Open Systems
=}------------------------------------------------ooO--U--Ooo------------{=
   - baruch at tkos.co.il - tel: +972.2.679.5364, http://www.tkos.co.il -

[Buildroot] [PATCH buildroot-test] scripts/autobuild-run: make it Python 3.x compatible

[Thomas Petazzoni]

Hello,

On Tue, 3 Dec 2019 19:11:37 +0200
Baruch Siach <baruch@tkos.co.il> wrote:

> >  - The urlparse module no longer exists, it's not urllib.parse  
> 
> s/not/now/

Indeed.

> >  if sys.hexversion >= 0x3000000:
> >      import configparser
> >      import urllib.request as _urllib
> > +    import urllib.parse as urlparse  
> 
> Would that work with any Python 3.x?

https://docs.python.org/3/library/urllib.parse.html#module-urllib.parse
has some comments in the documentation of urllib.parse.urlparse() that
say:

"""
Changed in version 3.2: Added IPv6 URL parsing capabilities.
"""

So it means that urllib.parse.urlparse() was there at least before
Python 3.2, which means it was most likely there since Python 3.x.

Best regards,

Thomas
-- 
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

[Buildroot] [PATCH buildroot-test] scripts/autobuild-run: make it Python 3.x compatible

[Arnout Vandecappelle]

On 03/12/2019 21:00, Thomas Petazzoni wrote:
> Hello,
> 
> On Tue, 3 Dec 2019 19:11:37 +0200
> Baruch Siach <baruch@tkos.co.il> wrote:
> 
>>>  - The urlparse module no longer exists, it's not urllib.parse  
>>
>> s/not/now/
> 
> Indeed.
> 
>>>  if sys.hexversion >= 0x3000000:
>>>      import configparser
>>>      import urllib.request as _urllib
>>> +    import urllib.parse as urlparse  
>>
>> Would that work with any Python 3.x?
> 
> https://docs.python.org/3/library/urllib.parse.html#module-urllib.parse
> has some comments in the documentation of urllib.parse.urlparse() that
> say:
> 
> """
> Changed in version 3.2: Added IPv6 URL parsing capabilities.
> """
> 
> So it means that urllib.parse.urlparse() was there at least before
> Python 3.2, which means it was most likely there since Python 3.x.

 And anyway, I think we generally should assume Python >= 3.4. Debian
oldoldstable (jessie) has 3.4 already. Trying to support older versions will
make our life unnecessarily difficult.

 Regards,
 Arnout

[Buildroot] [PATCH buildroot-test] scripts/autobuild-run: make it Python 3.x compatible

[Thomas Petazzoni]

On Tue,  3 Dec 2019 17:54:57 +0100
Thomas Petazzoni <thomas.petazzoni@bootlin.com> wrote:

> With Python 3.7, the autobuild-run did not work due to the following
> issues:
> 
>  - The urlparse module no longer exists, it's not urllib.parse
> 
>  - 0022 is no longer recognized as an octal value, we must use 0o022,
>    which also works in Python 2.x
> 
>  - reading the CSV file with the list of branches through the CSV
>    parser failed due to the lack of decoding, as urlopen_closing()
>    returns a stream of bytes and not strings. So we need to call
>    decode_bytes() on each element of the CSV array. Since the CSV file
>    is typically 3 or 4 lines long, we don't really need to optimize
>    things.
> 
> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
> ---
>  scripts/autobuild-run | 9 ++++++---
>  1 file changed, 6 insertions(+), 3 deletions(-)

Applied to buildroot-test, thanks.

Thomas
-- 
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com