All of lore.kernel.org
 help / color / mirror / Atom feed
* [Buildroot] [PATCH] scanpypi: rework download_package error handling
@ 2018-06-13  7:37 yegorslists at googlemail.com
  2018-06-15 12:13 ` Peter Korsgaard
  0 siblings, 1 reply; 5+ messages in thread
From: yegorslists at googlemail.com @ 2018-06-13  7:37 UTC (permalink / raw)
  To: buildroot

From: Yegor Yefremov <yegorslists@googlemail.com>

Some packages don't provide source archive but only a wheel file. In
this case download variable is not defined. So define this variable at
the very beginning and check whether it is None after searching for
source archives in the metadata.

Bonus: fix PEP8 issue with wrong indentation.

Signed-off-by: Yegor Yefremov <yegorslists@googlemail.com>
---
 utils/scanpypi | 14 +++++++++-----
 1 file changed, 9 insertions(+), 5 deletions(-)

diff --git a/utils/scanpypi b/utils/scanpypi
index 8a2ae00434..12c96b842e 100755
--- a/utils/scanpypi
+++ b/utils/scanpypi
@@ -178,6 +178,7 @@ class BuildrootPackage():
         """
         Download a package using metadata from pypi
         """
+        download = None
         try:
             self.metadata['urls'][0]['filename']
         except IndexError:
@@ -201,7 +202,7 @@ class BuildrootPackage():
                 continue
             try:
                 print('Downloading package {pkg} from {url}...'.format(
-                      pkg=self.real_name, url=download_url['url']))
+                    pkg=self.real_name, url=download_url['url']))
                 download = six.moves.urllib.request.urlopen(download_url['url'])
             except six.moves.urllib.error.HTTPError as http_error:
                 download = http_error
@@ -213,11 +214,14 @@ class BuildrootPackage():
                 self.md5_sum = hashlib.md5(self.as_string).hexdigest()
                 if self.md5_sum == download_url['digests']['md5']:
                     break
-        else:
-            if download.__class__ == six.moves.urllib.error.HTTPError:
-                raise download
-            raise DownloadFailed('Failed to download package {pkg}'
+
+        if download is None:
+            raise DownloadFailed('Failed to download package {pkg}: '
+                                 'No source archive available'
                                  .format(pkg=self.real_name))
+        elif download.__class__ == six.moves.urllib.error.HTTPError:
+            raise download
+
         self.filename = self.used_url['filename']
         self.url = self.used_url['url']
 
-- 
2.17.0

^ permalink raw reply related	[flat|nested] 5+ messages in thread

* [Buildroot] [PATCH] scanpypi: rework download_package error handling
  2018-06-13  7:37 [Buildroot] [PATCH] scanpypi: rework download_package error handling yegorslists at googlemail.com
@ 2018-06-15 12:13 ` Peter Korsgaard
  2018-06-15 12:30   ` Yegor Yefremov
  2018-06-15 12:59   ` Jaap Crezee
  0 siblings, 2 replies; 5+ messages in thread
From: Peter Korsgaard @ 2018-06-15 12:13 UTC (permalink / raw)
  To: buildroot

>>>>> "yegorslists" == yegorslists  <yegorslists@googlemail.com> writes:

 > From: Yegor Yefremov <yegorslists@googlemail.com>
 > Some packages don't provide source archive but only a wheel file. In
 > this case download variable is not defined. So define this variable at
 > the very beginning and check whether it is None after searching for
 > source archives in the metadata.

 > Bonus: fix PEP8 issue with wrong indentation.

 > Signed-off-by: Yegor Yefremov <yegorslists@googlemail.com>

Committed, thanks.

Out of interest, are you using python 2.x or 3.x? I just tried scanpypi
for the validators module on my Debian (so python 2.7) laptop:

./utils/scanpypi validators
spdx_lookup module is not installed. This can lead to an inaccurate licence detection. Please install it via
pip install spdx_lookup
buildroot package name for validators: python-validators
Package: python-validators
Fetching package validators
Downloading package validators from https://files.pythonhosted.org/packages/45/7b/5b7b74208a3e0744d1a0efbfb1935fa46fa4cfe58d3d63f17c49c58c429c/validators-0.12.2.tar.gz...
Added packages 'six, decorator' as dependencies of python-validators
Checking if package ./package/python-validators already exists...
Error: Package ./package/python-validators already exists
Do you want to delete existing package ? [y/N]y
Creating ./package/python-validators/python-validators.mk...
Creating ./package/python-validators/python-validators.hash...
Creating ./package/python-validators/Config.in...
Traceback (most recent call last):
  File "./utils/scanpypi", line 720, in <module>
    main()
  File "./utils/scanpypi", line 712, in main
    package.create_config_in()
  File "./utils/scanpypi", line 606, in create_config_in
    config_file.writelines(lines)
TypeError: writelines() argument must be a sequence of strings

Printing out 'lines', it seems to be a mix of normal strings and unicode:

lines: ['config BR2_PACKAGE_PYTHON_VALIDATORS\n', '\tbool "python-validators"\n', '\tselect BR2_PACKAGE_PYTHON_SIX # runtime\n', '\tselect BR2_PACKAGE_PYTHON_DECORATOR # runtime\n', '\thelp\n', u'\t  Python Data Validation for Humans\u2122.\n', '\n', u'\t  https://github.com/kvesteri/validators\n']

-- 
Bye, Peter Korsgaard

^ permalink raw reply	[flat|nested] 5+ messages in thread

* [Buildroot] [PATCH] scanpypi: rework download_package error handling
  2018-06-15 12:13 ` Peter Korsgaard
@ 2018-06-15 12:30   ` Yegor Yefremov
  2018-06-15 13:05     ` Peter Korsgaard
  2018-06-15 12:59   ` Jaap Crezee
  1 sibling, 1 reply; 5+ messages in thread
From: Yegor Yefremov @ 2018-06-15 12:30 UTC (permalink / raw)
  To: buildroot

On Fri, Jun 15, 2018 at 2:13 PM, Peter Korsgaard <peter@korsgaard.com> wrote:
>>>>>> "yegorslists" == yegorslists  <yegorslists@googlemail.com> writes:
>
>  > From: Yegor Yefremov <yegorslists@googlemail.com>
>  > Some packages don't provide source archive but only a wheel file. In
>  > this case download variable is not defined. So define this variable at
>  > the very beginning and check whether it is None after searching for
>  > source archives in the metadata.
>
>  > Bonus: fix PEP8 issue with wrong indentation.
>
>  > Signed-off-by: Yegor Yefremov <yegorslists@googlemail.com>
>
> Committed, thanks.
>
> Out of interest, are you using python 2.x or 3.x? I just tried scanpypi
> for the validators module on my Debian (so python 2.7) laptop:
>
> ./utils/scanpypi validators
> spdx_lookup module is not installed. This can lead to an inaccurate licence detection. Please install it via
> pip install spdx_lookup
> buildroot package name for validators: python-validators
> Package: python-validators
> Fetching package validators
> Downloading package validators from https://files.pythonhosted.org/packages/45/7b/5b7b74208a3e0744d1a0efbfb1935fa46fa4cfe58d3d63f17c49c58c429c/validators-0.12.2.tar.gz...
> Added packages 'six, decorator' as dependencies of python-validators
> Checking if package ./package/python-validators already exists...
> Error: Package ./package/python-validators already exists
> Do you want to delete existing package ? [y/N]y
> Creating ./package/python-validators/python-validators.mk...
> Creating ./package/python-validators/python-validators.hash...
> Creating ./package/python-validators/Config.in...
> Traceback (most recent call last):
>   File "./utils/scanpypi", line 720, in <module>
>     main()
>   File "./utils/scanpypi", line 712, in main
>     package.create_config_in()
>   File "./utils/scanpypi", line 606, in create_config_in
>     config_file.writelines(lines)
> TypeError: writelines() argument must be a sequence of strings
>
> Printing out 'lines', it seems to be a mix of normal strings and unicode:
>
> lines: ['config BR2_PACKAGE_PYTHON_VALIDATORS\n', '\tbool "python-validators"\n', '\tselect BR2_PACKAGE_PYTHON_SIX # runtime\n', '\tselect BR2_PACKAGE_PYTHON_DECORATOR # runtime\n', '\thelp\n', u'\t  Python Data Validation for Humans\u2122.\n', '\n', u'\t  https://github.com/kvesteri/validators\n']

I'm still using Python 2 by default. In this case I think trademark
symbol is the problem. That's why it was important to use scanpypi
with Python 3.

Yegor

^ permalink raw reply	[flat|nested] 5+ messages in thread

* [Buildroot] [PATCH] scanpypi: rework download_package error handling
  2018-06-15 12:13 ` Peter Korsgaard
  2018-06-15 12:30   ` Yegor Yefremov
@ 2018-06-15 12:59   ` Jaap Crezee
  1 sibling, 0 replies; 5+ messages in thread
From: Jaap Crezee @ 2018-06-15 12:59 UTC (permalink / raw)
  To: buildroot

On 06/15/18 14:13, Peter Korsgaard wrote:
> Out of interest, are you using python 2.x or 3.x? I just tried scanpypi
> for the validators module on my Debian (so python 2.7) laptop:

I use python 3(.6) as default python interpreter. The error you posted "sounds" vaguely familliar 
however.

regards,


Jaap

^ permalink raw reply	[flat|nested] 5+ messages in thread

* [Buildroot] [PATCH] scanpypi: rework download_package error handling
  2018-06-15 12:30   ` Yegor Yefremov
@ 2018-06-15 13:05     ` Peter Korsgaard
  0 siblings, 0 replies; 5+ messages in thread
From: Peter Korsgaard @ 2018-06-15 13:05 UTC (permalink / raw)
  To: buildroot

>>>>> "Yegor" == Yegor Yefremov <yegorslists@googlemail.com> writes:

Hi,

 >> Printing out 'lines', it seems to be a mix of normal strings and unicode:
 >> 
 >> lines: ['config BR2_PACKAGE_PYTHON_VALIDATORS\n', '\tbool "python-validators"\n', '\tselect BR2_PACKAGE_PYTHON_SIX # runtime\n', '\tselect BR2_PACKAGE_PYTHON_DECORATOR # runtime\n', '\thelp\n', u'\t  Python Data Validation for Humans\u2122.\n', '\n', u'\t  https://github.com/kvesteri/validators\n']

 > I'm still using Python 2 by default. In this case I think trademark
 > symbol is the problem.

Yes. Perhaps we should filter out these unicode characters? We probably
don't really want a TM character in the help text anyway.

-- 
Bye, Peter Korsgaard

^ permalink raw reply	[flat|nested] 5+ messages in thread

end of thread, other threads:[~2018-06-15 13:05 UTC | newest]

Thread overview: 5+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2018-06-13  7:37 [Buildroot] [PATCH] scanpypi: rework download_package error handling yegorslists at googlemail.com
2018-06-15 12:13 ` Peter Korsgaard
2018-06-15 12:30   ` Yegor Yefremov
2018-06-15 13:05     ` Peter Korsgaard
2018-06-15 12:59   ` Jaap Crezee

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.