* [Buildroot] [PATCH] scanpypi: add support for the new PyPI infrastructure
@ 2018-04-18 9:55 yegorslists at googlemail.com
2018-04-18 15:14 ` Thomas Petazzoni
` (2 more replies)
0 siblings, 3 replies; 6+ messages in thread
From: yegorslists at googlemail.com @ 2018-04-18 9:55 UTC (permalink / raw)
To: buildroot
From: Yegor Yefremov <yegorslists@googlemail.com>
https://pypi.python.org URL has been changed to https://pypi.org.
Package's JSON object now contains sha256 checksum, so use it
instead of locally computed one. Change comments in the hash
file accordingly.
Signed-off-by: Yegor Yefremov <yegorslists@googlemail.com>
---
utils/scanpypi | 29 +++++++++++++++--------------
1 file changed, 15 insertions(+), 14 deletions(-)
diff --git a/utils/scanpypi b/utils/scanpypi
index f03ad0bb64..8a2ae00434 100755
--- a/utils/scanpypi
+++ b/utils/scanpypi
@@ -153,7 +153,7 @@ class BuildrootPackage():
"""
Fetch a package's metadata from the python package index
"""
- self.metadata_url = 'https://pypi.python.org/pypi/{pkg}/json'.format(
+ self.metadata_url = 'https://pypi.org/pypi/{pkg}/json'.format(
pkg=self.real_name)
try:
pkg_json = six.moves.urllib.request.urlopen(self.metadata_url).read().decode()
@@ -187,7 +187,7 @@ class BuildrootPackage():
self.metadata['urls'] = [{
'packagetype': 'sdist',
'url': self.metadata['info']['download_url'],
- 'md5_digest': None}]
+ 'digests': None}]
# In this case, we can't get the name of the downloaded file
# from the pypi api, so we need to find it, this should work
urlpath = six.moves.urllib.parse.urlparse(
@@ -208,10 +208,10 @@ class BuildrootPackage():
else:
self.used_url = download_url
self.as_string = download.read()
- if not download_url['md5_digest']:
+ if not download_url['digests']['md5']:
break
self.md5_sum = hashlib.md5(self.as_string).hexdigest()
- if self.md5_sum == download_url['md5_digest']:
+ if self.md5_sum == download_url['digests']['md5']:
break
else:
if download.__class__ == six.moves.urllib.error.HTTPError:
@@ -529,22 +529,23 @@ class BuildrootPackage():
path_to_hash = os.path.join(self.pkg_dir, pkg_hash)
print('Creating {filename}...'.format(filename=path_to_hash))
lines = []
- if self.used_url['md5_digest']:
- md5_comment = '# md5 from {url}, sha256 locally computed\n'.format(
+ if self.used_url['digests']['md5'] and self.used_url['digests']['sha256']:
+ hash_header = '# md5, sha256 from {url}\n'.format(
url=self.metadata_url)
- lines.append(md5_comment)
+ lines.append(hash_header)
hash_line = '{method}\t{digest} {filename}\n'.format(
method='md5',
- digest=self.used_url['md5_digest'],
+ digest=self.used_url['digests']['md5'],
+ filename=self.filename)
+ lines.append(hash_line)
+ hash_line = '{method}\t{digest} {filename}\n'.format(
+ method='sha256',
+ digest=self.used_url['digests']['sha256'],
filename=self.filename)
lines.append(hash_line)
- digest = hashlib.sha256(self.as_string).hexdigest()
- hash_line = '{method}\t{digest} {filename}\n'.format(
- method='sha256',
- digest=digest,
- filename=self.filename)
- lines.append(hash_line)
+ if self.license_files:
+ lines.append('# Locally computed sha256 checksums\n')
for license_file in self.license_files:
sha256 = hashlib.sha256()
with open(license_file, 'rb') as lic_f:
--
2.17.0
^ permalink raw reply related [flat|nested] 6+ messages in thread
* [Buildroot] [PATCH] scanpypi: add support for the new PyPI infrastructure
2018-04-18 9:55 [Buildroot] [PATCH] scanpypi: add support for the new PyPI infrastructure yegorslists at googlemail.com
@ 2018-04-18 15:14 ` Thomas Petazzoni
2018-04-18 21:22 ` Arnout Vandecappelle
2018-05-01 6:53 ` Peter Korsgaard
2 siblings, 0 replies; 6+ messages in thread
From: Thomas Petazzoni @ 2018-04-18 15:14 UTC (permalink / raw)
To: buildroot
Hello,
On Wed, 18 Apr 2018 11:55:42 +0200, yegorslists at googlemail.com wrote:
> From: Yegor Yefremov <yegorslists@googlemail.com>
>
> https://pypi.python.org URL has been changed to https://pypi.org.
>
> Package's JSON object now contains sha256 checksum, so use it
> instead of locally computed one. Change comments in the hash
> file accordingly.
>
> Signed-off-by: Yegor Yefremov <yegorslists@googlemail.com>
> ---
> utils/scanpypi | 29 +++++++++++++++--------------
> 1 file changed, 15 insertions(+), 14 deletions(-)
Applied to master, thanks.
Thomas
--
Thomas Petazzoni, CTO, Bootlin (formerly Free Electrons)
Embedded Linux and Kernel engineering
https://bootlin.com
^ permalink raw reply [flat|nested] 6+ messages in thread
* [Buildroot] [PATCH] scanpypi: add support for the new PyPI infrastructure
2018-04-18 9:55 [Buildroot] [PATCH] scanpypi: add support for the new PyPI infrastructure yegorslists at googlemail.com
2018-04-18 15:14 ` Thomas Petazzoni
@ 2018-04-18 21:22 ` Arnout Vandecappelle
2018-04-19 6:57 ` Yegor Yefremov
2018-05-01 6:53 ` Peter Korsgaard
2 siblings, 1 reply; 6+ messages in thread
From: Arnout Vandecappelle @ 2018-04-18 21:22 UTC (permalink / raw)
To: buildroot
On 18-04-18 11:55, yegorslists at googlemail.com wrote:
> From: Yegor Yefremov <yegorslists@googlemail.com>
>
> https://pypi.python.org URL has been changed to https://pypi.org.
>
> Package's JSON object now contains sha256 checksum, so use it
> instead of locally computed one. Change comments in the hash
> file accordingly.
>
> Signed-off-by: Yegor Yefremov <yegorslists@googlemail.com>
> ---
> utils/scanpypi | 29 +++++++++++++++--------------
> 1 file changed, 15 insertions(+), 14 deletions(-)
>
> diff --git a/utils/scanpypi b/utils/scanpypi
> index f03ad0bb64..8a2ae00434 100755
> --- a/utils/scanpypi
> +++ b/utils/scanpypi
> @@ -153,7 +153,7 @@ class BuildrootPackage():
> """
> Fetch a package's metadata from the python package index
> """
> - self.metadata_url = 'https://pypi.python.org/pypi/{pkg}/json'.format(
> + self.metadata_url = 'https://pypi.org/pypi/{pkg}/json'.format(
> pkg=self.real_name)
> try:
> pkg_json = six.moves.urllib.request.urlopen(self.metadata_url).read().decode()
> @@ -187,7 +187,7 @@ class BuildrootPackage():
> self.metadata['urls'] = [{
> 'packagetype': 'sdist',
> 'url': self.metadata['info']['download_url'],
> - 'md5_digest': None}]
> + 'digests': None}]
> # In this case, we can't get the name of the downloaded file
> # from the pypi api, so we need to find it, this should work
> urlpath = six.moves.urllib.parse.urlparse(
> @@ -208,10 +208,10 @@ class BuildrootPackage():
> else:
> self.used_url = download_url
> self.as_string = download.read()
> - if not download_url['md5_digest']:
> + if not download_url['digests']['md5']:
> break
> self.md5_sum = hashlib.md5(self.as_string).hexdigest()
> - if self.md5_sum == download_url['md5_digest']:
> + if self.md5_sum == download_url['digests']['md5']:
> break
> else:
> if download.__class__ == six.moves.urllib.error.HTTPError:
> @@ -529,22 +529,23 @@ class BuildrootPackage():
> path_to_hash = os.path.join(self.pkg_dir, pkg_hash)
> print('Creating {filename}...'.format(filename=path_to_hash))
> lines = []
> - if self.used_url['md5_digest']:
> - md5_comment = '# md5 from {url}, sha256 locally computed\n'.format(
> + if self.used_url['digests']['md5'] and self.used_url['digests']['sha256']:
If there is a sha256, there is no point adding the md5.
If there is no sha256 for whatever reason (can this happen?), we don't get any
hash at all...
So I think this should be (pseudocode):
if self.used_url['digests']['sha256']:
hash_header = '# sha256 from ...'
...
else if self.used_url['digests']['md5']:
hash_header = '# md5 from {url}, sha256 locally computed\n'
# original code
Regards,
Arnout
> + hash_header = '# md5, sha256 from {url}\n'.format(
> url=self.metadata_url)
> - lines.append(md5_comment)
> + lines.append(hash_header)
> hash_line = '{method}\t{digest} {filename}\n'.format(
> method='md5',
> - digest=self.used_url['md5_digest'],
> + digest=self.used_url['digests']['md5'],
> + filename=self.filename)
> + lines.append(hash_line)
> + hash_line = '{method}\t{digest} {filename}\n'.format(
> + method='sha256',
> + digest=self.used_url['digests']['sha256'],
> filename=self.filename)
> lines.append(hash_line)
> - digest = hashlib.sha256(self.as_string).hexdigest()
> - hash_line = '{method}\t{digest} {filename}\n'.format(
> - method='sha256',
> - digest=digest,
> - filename=self.filename)
> - lines.append(hash_line)
>
> + if self.license_files:
> + lines.append('# Locally computed sha256 checksums\n')
> for license_file in self.license_files:
> sha256 = hashlib.sha256()
> with open(license_file, 'rb') as lic_f:
>
--
Arnout Vandecappelle arnout at mind be
Senior Embedded Software Architect +32-16-286500
Essensium/Mind http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint: 7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF
^ permalink raw reply [flat|nested] 6+ messages in thread
* [Buildroot] [PATCH] scanpypi: add support for the new PyPI infrastructure
2018-04-18 21:22 ` Arnout Vandecappelle
@ 2018-04-19 6:57 ` Yegor Yefremov
2018-04-23 7:45 ` Arnout Vandecappelle
0 siblings, 1 reply; 6+ messages in thread
From: Yegor Yefremov @ 2018-04-19 6:57 UTC (permalink / raw)
To: buildroot
Hi Arnout,
On Wed, Apr 18, 2018 at 11:22 PM, Arnout Vandecappelle <arnout@mind.be> wrote:
>
>
> On 18-04-18 11:55, yegorslists at googlemail.com wrote:
>> From: Yegor Yefremov <yegorslists@googlemail.com>
>>
>> https://pypi.python.org URL has been changed to https://pypi.org.
>>
>> Package's JSON object now contains sha256 checksum, so use it
>> instead of locally computed one. Change comments in the hash
>> file accordingly.
>>
>> Signed-off-by: Yegor Yefremov <yegorslists@googlemail.com>
>> ---
>> utils/scanpypi | 29 +++++++++++++++--------------
>> 1 file changed, 15 insertions(+), 14 deletions(-)
>>
>> diff --git a/utils/scanpypi b/utils/scanpypi
>> index f03ad0bb64..8a2ae00434 100755
>> --- a/utils/scanpypi
>> +++ b/utils/scanpypi
>> @@ -153,7 +153,7 @@ class BuildrootPackage():
>> """
>> Fetch a package's metadata from the python package index
>> """
>> - self.metadata_url = 'https://pypi.python.org/pypi/{pkg}/json'.format(
>> + self.metadata_url = 'https://pypi.org/pypi/{pkg}/json'.format(
>> pkg=self.real_name)
>> try:
>> pkg_json = six.moves.urllib.request.urlopen(self.metadata_url).read().decode()
>> @@ -187,7 +187,7 @@ class BuildrootPackage():
>> self.metadata['urls'] = [{
>> 'packagetype': 'sdist',
>> 'url': self.metadata['info']['download_url'],
>> - 'md5_digest': None}]
>> + 'digests': None}]
>> # In this case, we can't get the name of the downloaded file
>> # from the pypi api, so we need to find it, this should work
>> urlpath = six.moves.urllib.parse.urlparse(
>> @@ -208,10 +208,10 @@ class BuildrootPackage():
>> else:
>> self.used_url = download_url
>> self.as_string = download.read()
>> - if not download_url['md5_digest']:
>> + if not download_url['digests']['md5']:
>> break
>> self.md5_sum = hashlib.md5(self.as_string).hexdigest()
>> - if self.md5_sum == download_url['md5_digest']:
>> + if self.md5_sum == download_url['digests']['md5']:
>> break
>> else:
>> if download.__class__ == six.moves.urllib.error.HTTPError:
>> @@ -529,22 +529,23 @@ class BuildrootPackage():
>> path_to_hash = os.path.join(self.pkg_dir, pkg_hash)
>> print('Creating {filename}...'.format(filename=path_to_hash))
>> lines = []
>> - if self.used_url['md5_digest']:
>> - md5_comment = '# md5 from {url}, sha256 locally computed\n'.format(
>> + if self.used_url['digests']['md5'] and self.used_url['digests']['sha256']:
>
> If there is a sha256, there is no point adding the md5.
>
> If there is no sha256 for whatever reason (can this happen?), we don't get any
> hash at all...
>
> So I think this should be (pseudocode):
>
> if self.used_url['digests']['sha256']:
> hash_header = '# sha256 from ...'
> ...
> else if self.used_url['digests']['md5']:
> hash_header = '# md5 from {url}, sha256 locally computed\n'
> # original code
You're right. We don't need md5 at all. And I doubt there are any
packages without sha256 as those checksums will be calculated and put
into JSON automatically (at least I believe it is so). I've seen
sha256 checksum for older package version so it seems to be
consistent.
What do you think?
Yegor
^ permalink raw reply [flat|nested] 6+ messages in thread
* [Buildroot] [PATCH] scanpypi: add support for the new PyPI infrastructure
2018-04-19 6:57 ` Yegor Yefremov
@ 2018-04-23 7:45 ` Arnout Vandecappelle
0 siblings, 0 replies; 6+ messages in thread
From: Arnout Vandecappelle @ 2018-04-23 7:45 UTC (permalink / raw)
To: buildroot
On 19-04-18 08:57, Yegor Yefremov wrote:
> Hi Arnout,
>
> On Wed, Apr 18, 2018 at 11:22 PM, Arnout Vandecappelle <arnout@mind.be> wrote:
>>
>>
>> On 18-04-18 11:55, yegorslists at googlemail.com wrote:
>>> From: Yegor Yefremov <yegorslists@googlemail.com>
[snip]
>>> @@ -529,22 +529,23 @@ class BuildrootPackage():
>>> path_to_hash = os.path.join(self.pkg_dir, pkg_hash)
>>> print('Creating {filename}...'.format(filename=path_to_hash))
>>> lines = []
>>> - if self.used_url['md5_digest']:
>>> - md5_comment = '# md5 from {url}, sha256 locally computed\n'.format(
>>> + if self.used_url['digests']['md5'] and self.used_url['digests']['sha256']:
>>
>> If there is a sha256, there is no point adding the md5.
>>
>> If there is no sha256 for whatever reason (can this happen?), we don't get any
>> hash at all...
>>
>> So I think this should be (pseudocode):
>>
>> if self.used_url['digests']['sha256']:
>> hash_header = '# sha256 from ...'
>> ...
>> else if self.used_url['digests']['md5']:
>> hash_header = '# md5 from {url}, sha256 locally computed\n'
>> # original code
>
> You're right. We don't need md5 at all. And I doubt there are any
> packages without sha256 as those checksums will be calculated and put
> into JSON automatically (at least I believe it is so). I've seen
> sha256 checksum for older package version so it seems to be
> consistent.
>
> What do you think?
If usually there is a sha256, just support the sha256. If it is missing then
the md5 can be added manually.
Regards,
Arnout
--
Arnout Vandecappelle arnout at mind be
Senior Embedded Software Architect +32-16-286500
Essensium/Mind http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint: 7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF
^ permalink raw reply [flat|nested] 6+ messages in thread
* [Buildroot] [PATCH] scanpypi: add support for the new PyPI infrastructure
2018-04-18 9:55 [Buildroot] [PATCH] scanpypi: add support for the new PyPI infrastructure yegorslists at googlemail.com
2018-04-18 15:14 ` Thomas Petazzoni
2018-04-18 21:22 ` Arnout Vandecappelle
@ 2018-05-01 6:53 ` Peter Korsgaard
2 siblings, 0 replies; 6+ messages in thread
From: Peter Korsgaard @ 2018-05-01 6:53 UTC (permalink / raw)
To: buildroot
>>>>> "yegorslists" == yegorslists <yegorslists@googlemail.com> writes:
> From: Yegor Yefremov <yegorslists@googlemail.com>
> https://pypi.python.org URL has been changed to https://pypi.org.
> Package's JSON object now contains sha256 checksum, so use it
> instead of locally computed one. Change comments in the hash
> file accordingly.
> Signed-off-by: Yegor Yefremov <yegorslists@googlemail.com>
Committed to 2018.02.x, thanks.
--
Bye, Peter Korsgaard
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2018-05-01 6:53 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2018-04-18 9:55 [Buildroot] [PATCH] scanpypi: add support for the new PyPI infrastructure yegorslists at googlemail.com
2018-04-18 15:14 ` Thomas Petazzoni
2018-04-18 21:22 ` Arnout Vandecappelle
2018-04-19 6:57 ` Yegor Yefremov
2018-04-23 7:45 ` Arnout Vandecappelle
2018-05-01 6:53 ` Peter Korsgaard
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.