qemu-devel.nongnu.org archive mirror
 help / color / mirror / Atom feed
* Re: [Qemu-devel] [PATCH 2/5] tests/vm: Port basevm to Python 3
       [not found] ` <20190329210804.22121-3-wainersm@redhat.com>
@ 2019-04-01 11:34   ` Philippe Mathieu-Daudé
  0 siblings, 0 replies; 9+ messages in thread
From: Philippe Mathieu-Daudé @ 2019-04-01 11:34 UTC (permalink / raw)
  To: Wainer dos Santos Moschetta, qemu-devel
  Cc: fam, alex.bennee, berrange, lersek, pbonzini, peter.maydell

On 3/29/19 10:08 PM, Wainer dos Santos Moschetta wrote:
> Fixed tests/vm/basevm.py to run with Python 3:
>  - hashlib.sha1() requires an binary encoded object.
>  - uses floor division ("//") (PEP 238).
>  - decode bytes to unicode when needed.
> 
> Signed-off-by: Wainer dos Santos Moschetta <wainersm@redhat.com>

Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>

> ---
>  tests/vm/basevm.py | 8 ++++----
>  1 file changed, 4 insertions(+), 4 deletions(-)
> 
> diff --git a/tests/vm/basevm.py b/tests/vm/basevm.py
> index 0556bdcf9e..083befce9f 100755
> --- a/tests/vm/basevm.py
> +++ b/tests/vm/basevm.py
> @@ -85,12 +85,12 @@ class BaseVM(object):
>              if not sha256sum:
>                  return True
>              checksum = subprocess.check_output(["sha256sum", fname]).split()[0]
> -            return sha256sum == checksum
> +            return sha256sum == checksum.decode()
>  
>          cache_dir = os.path.expanduser("~/.cache/qemu-vm/download")
>          if not os.path.exists(cache_dir):
>              os.makedirs(cache_dir)
> -        fname = os.path.join(cache_dir, hashlib.sha1(url).hexdigest())
> +        fname = os.path.join(cache_dir, hashlib.sha1(url.encode()).hexdigest())
>          if os.path.exists(fname) and check_sha256sum(fname):
>              return fname
>          logging.debug("Downloading %s to %s...", url, fname)
> @@ -134,7 +134,7 @@ class BaseVM(object):
>          raise NotImplementedError
>  
>      def add_source_dir(self, src_dir):
> -        name = "data-" + hashlib.sha1(src_dir).hexdigest()[:5]
> +        name = "data-" + hashlib.sha1(src_dir.encode()).hexdigest()[:5]
>          tarfile = os.path.join(self._tmpdir, name + ".tar")
>          logging.debug("Creating archive %s for src_dir dir: %s", tarfile, src_dir)
>          subprocess.check_call(["./scripts/archive-source.sh", tarfile],
> @@ -204,7 +204,7 @@ def parse_args(vmcls):
>  
>      def get_default_jobs():
>          if kvm_available(vmcls.arch):
> -            return multiprocessing.cpu_count() / 2
> +            return multiprocessing.cpu_count() // 2
>          else:
>              return 1
>  
> 

^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: [Qemu-devel] [PATCH 0/5] tests/vm: Python 3, improve image caching, and misc
       [not found] <20190329210804.22121-1-wainersm@redhat.com>
       [not found] ` <20190329210804.22121-3-wainersm@redhat.com>
@ 2019-04-23 19:28 ` Wainer dos Santos Moschetta
  2019-04-23 19:28   ` Wainer dos Santos Moschetta
                     ` (2 more replies)
       [not found] ` <20190329210804.22121-4-wainersm@redhat.com>
  2 siblings, 3 replies; 9+ messages in thread
From: Wainer dos Santos Moschetta @ 2019-04-23 19:28 UTC (permalink / raw)
  To: qemu-devel; +Cc: fam, peter.maydell, philmd, lersek, pbonzini, alex.bennee

Ping. More reviews needed.

I've already got Philippe's reviewed-by, thanks!

- Wainer


On 03/29/2019 06:07 PM, Wainer dos Santos Moschetta wrote:
> Hi all!
>
> This series bundles the support to Python3, improvement to image caching, and miscellaneous changes for the vm-test (`make vm-build-*`).
>
> Git tree: http://github.com/wainersm/qemu
> Branch: vm_test_python3_and_misc
> Travis: https://travis-ci.org/wainersm/qemu/builds/513220300
>
> Below you can find some general comments.
>
> Patch 01:
> Uses python configured at build, although vm-test can be executed from the code tree (see docs/devel/testing.rst) too. In this case $(PYTHON) will be empty and it picks python pointed by the script's shebang, as a result failing on Python 3 only Linux distros.
>
> Patch 02:
> Ported based.py to work with Python 3 (kept Python 2 compatibility). Fixed the follow errors:
>    - Raising "TypeError: Unicode-objects must be encoded before hashing" exception by hashlib.sha1().
>    - get_default_jobs() is used to set the vm cpus, but it was returning an invalid float number.
>    - On check_sha256sum(), subprocess.check_output() returns bytes so that string comparison was failing.
>
> Patch 03:
> Implemented a simple mechanism to detect the image file changed on http://download.patchew.org, as discussed in https://www.mail-archive.com/qemu-devel@nongnu.org/msg607839.html. I didn't implement the '--force' option that Paolo suggested, because it would require a major refactoring (technically the vm implementation decides whether use the cache or not). My intention is to keep the script as simple as possible, so IMO, if the image file changed on the server then it should equally be updated the checksum on QEMU's side.
>
> Patch 04:
> Default network (NETWORK=1) backend is enough to have the docker-based tests run properly. Note: `make docker-test-block@centos7` is broken inside the CentOS VM and apparently on my host (Fedora 29 x86_64) too.
>
> Wainer dos Santos Moschetta (5):
>    tests/vm: Use python configured on build
>    tests/vm: Port basevm to Python 3
>    tests/vm: Detect the image changed on server
>    tests/vm: Fix build-centos docker-based tests run
>    tests/vm: Add missing variables on help
>
>   tests/vm/Makefile.include | 12 ++++++++----
>   tests/vm/basevm.py        | 39 ++++++++++++++++++++++++++++++++++-----
>   tests/vm/centos           |  6 +++---
>   3 files changed, 45 insertions(+), 12 deletions(-)
>

^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: [Qemu-devel] [PATCH 0/5] tests/vm: Python 3, improve image caching, and misc
  2019-04-23 19:28 ` [Qemu-devel] [PATCH 0/5] tests/vm: Python 3, improve image caching, and misc Wainer dos Santos Moschetta
@ 2019-04-23 19:28   ` Wainer dos Santos Moschetta
  2019-04-24  9:07   ` Laszlo Ersek
  2019-05-29 16:50   ` Alex Bennée
  2 siblings, 0 replies; 9+ messages in thread
From: Wainer dos Santos Moschetta @ 2019-04-23 19:28 UTC (permalink / raw)
  To: qemu-devel; +Cc: fam, peter.maydell, philmd, lersek, pbonzini, alex.bennee

Ping. More reviews needed.

I've already got Philippe's reviewed-by, thanks!

- Wainer


On 03/29/2019 06:07 PM, Wainer dos Santos Moschetta wrote:
> Hi all!
>
> This series bundles the support to Python3, improvement to image caching, and miscellaneous changes for the vm-test (`make vm-build-*`).
>
> Git tree: http://github.com/wainersm/qemu
> Branch: vm_test_python3_and_misc
> Travis: https://travis-ci.org/wainersm/qemu/builds/513220300
>
> Below you can find some general comments.
>
> Patch 01:
> Uses python configured at build, although vm-test can be executed from the code tree (see docs/devel/testing.rst) too. In this case $(PYTHON) will be empty and it picks python pointed by the script's shebang, as a result failing on Python 3 only Linux distros.
>
> Patch 02:
> Ported based.py to work with Python 3 (kept Python 2 compatibility). Fixed the follow errors:
>    - Raising "TypeError: Unicode-objects must be encoded before hashing" exception by hashlib.sha1().
>    - get_default_jobs() is used to set the vm cpus, but it was returning an invalid float number.
>    - On check_sha256sum(), subprocess.check_output() returns bytes so that string comparison was failing.
>
> Patch 03:
> Implemented a simple mechanism to detect the image file changed on http://download.patchew.org, as discussed in https://www.mail-archive.com/qemu-devel@nongnu.org/msg607839.html. I didn't implement the '--force' option that Paolo suggested, because it would require a major refactoring (technically the vm implementation decides whether use the cache or not). My intention is to keep the script as simple as possible, so IMO, if the image file changed on the server then it should equally be updated the checksum on QEMU's side.
>
> Patch 04:
> Default network (NETWORK=1) backend is enough to have the docker-based tests run properly. Note: `make docker-test-block@centos7` is broken inside the CentOS VM and apparently on my host (Fedora 29 x86_64) too.
>
> Wainer dos Santos Moschetta (5):
>    tests/vm: Use python configured on build
>    tests/vm: Port basevm to Python 3
>    tests/vm: Detect the image changed on server
>    tests/vm: Fix build-centos docker-based tests run
>    tests/vm: Add missing variables on help
>
>   tests/vm/Makefile.include | 12 ++++++++----
>   tests/vm/basevm.py        | 39 ++++++++++++++++++++++++++++++++++-----
>   tests/vm/centos           |  6 +++---
>   3 files changed, 45 insertions(+), 12 deletions(-)
>



^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: [Qemu-devel] [PATCH 0/5] tests/vm: Python 3, improve image caching, and misc
  2019-04-23 19:28 ` [Qemu-devel] [PATCH 0/5] tests/vm: Python 3, improve image caching, and misc Wainer dos Santos Moschetta
  2019-04-23 19:28   ` Wainer dos Santos Moschetta
@ 2019-04-24  9:07   ` Laszlo Ersek
  2019-04-24  9:07     ` Laszlo Ersek
  2019-04-25 18:02     ` Wainer dos Santos Moschetta
  2019-05-29 16:50   ` Alex Bennée
  2 siblings, 2 replies; 9+ messages in thread
From: Laszlo Ersek @ 2019-04-24  9:07 UTC (permalink / raw)
  To: Wainer dos Santos Moschetta, qemu-devel
  Cc: fam, peter.maydell, philmd, pbonzini, alex.bennee

Hello Wainer,

(answering because I dislike ignoring emails without giving any feedback:)

On 04/23/19 21:28, Wainer dos Santos Moschetta wrote:
> Ping. More reviews needed.
> 
> I've already got Philippe's reviewed-by, thanks!

I'm going to skip this one. According to "scripts/get_maintainer.pl", an
ACK from either Alex or Fam should suffice. (They could also decide to
queue your set with Phil's R-b, who's a designated reviewer on this topic.)

Thanks
Laszlo

> On 03/29/2019 06:07 PM, Wainer dos Santos Moschetta wrote:
>> Hi all!
>>
>> This series bundles the support to Python3, improvement to image
>> caching, and miscellaneous changes for the vm-test (`make vm-build-*`).
>>
>> Git tree: http://github.com/wainersm/qemu
>> Branch: vm_test_python3_and_misc
>> Travis: https://travis-ci.org/wainersm/qemu/builds/513220300
>>
>> Below you can find some general comments.
>>
>> Patch 01:
>> Uses python configured at build, although vm-test can be executed from
>> the code tree (see docs/devel/testing.rst) too. In this case $(PYTHON)
>> will be empty and it picks python pointed by the script's shebang, as
>> a result failing on Python 3 only Linux distros.
>>
>> Patch 02:
>> Ported based.py to work with Python 3 (kept Python 2 compatibility).
>> Fixed the follow errors:
>>    - Raising "TypeError: Unicode-objects must be encoded before
>> hashing" exception by hashlib.sha1().
>>    - get_default_jobs() is used to set the vm cpus, but it was
>> returning an invalid float number.
>>    - On check_sha256sum(), subprocess.check_output() returns bytes so
>> that string comparison was failing.
>>
>> Patch 03:
>> Implemented a simple mechanism to detect the image file changed on
>> http://download.patchew.org, as discussed in
>> https://www.mail-archive.com/qemu-devel@nongnu.org/msg607839.html. I
>> didn't implement the '--force' option that Paolo suggested, because it
>> would require a major refactoring (technically the vm implementation
>> decides whether use the cache or not). My intention is to keep the
>> script as simple as possible, so IMO, if the image file changed on the
>> server then it should equally be updated the checksum on QEMU's side.
>>
>> Patch 04:
>> Default network (NETWORK=1) backend is enough to have the docker-based
>> tests run properly. Note: `make docker-test-block@centos7` is broken
>> inside the CentOS VM and apparently on my host (Fedora 29 x86_64) too.
>>
>> Wainer dos Santos Moschetta (5):
>>    tests/vm: Use python configured on build
>>    tests/vm: Port basevm to Python 3
>>    tests/vm: Detect the image changed on server
>>    tests/vm: Fix build-centos docker-based tests run
>>    tests/vm: Add missing variables on help
>>
>>   tests/vm/Makefile.include | 12 ++++++++----
>>   tests/vm/basevm.py        | 39 ++++++++++++++++++++++++++++++++++-----
>>   tests/vm/centos           |  6 +++---
>>   3 files changed, 45 insertions(+), 12 deletions(-)
>>
> 

^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: [Qemu-devel] [PATCH 0/5] tests/vm: Python 3, improve image caching, and misc
  2019-04-24  9:07   ` Laszlo Ersek
@ 2019-04-24  9:07     ` Laszlo Ersek
  2019-04-25 18:02     ` Wainer dos Santos Moschetta
  1 sibling, 0 replies; 9+ messages in thread
From: Laszlo Ersek @ 2019-04-24  9:07 UTC (permalink / raw)
  To: Wainer dos Santos Moschetta, qemu-devel
  Cc: fam, peter.maydell, alex.bennee, philmd, pbonzini

Hello Wainer,

(answering because I dislike ignoring emails without giving any feedback:)

On 04/23/19 21:28, Wainer dos Santos Moschetta wrote:
> Ping. More reviews needed.
> 
> I've already got Philippe's reviewed-by, thanks!

I'm going to skip this one. According to "scripts/get_maintainer.pl", an
ACK from either Alex or Fam should suffice. (They could also decide to
queue your set with Phil's R-b, who's a designated reviewer on this topic.)

Thanks
Laszlo

> On 03/29/2019 06:07 PM, Wainer dos Santos Moschetta wrote:
>> Hi all!
>>
>> This series bundles the support to Python3, improvement to image
>> caching, and miscellaneous changes for the vm-test (`make vm-build-*`).
>>
>> Git tree: http://github.com/wainersm/qemu
>> Branch: vm_test_python3_and_misc
>> Travis: https://travis-ci.org/wainersm/qemu/builds/513220300
>>
>> Below you can find some general comments.
>>
>> Patch 01:
>> Uses python configured at build, although vm-test can be executed from
>> the code tree (see docs/devel/testing.rst) too. In this case $(PYTHON)
>> will be empty and it picks python pointed by the script's shebang, as
>> a result failing on Python 3 only Linux distros.
>>
>> Patch 02:
>> Ported based.py to work with Python 3 (kept Python 2 compatibility).
>> Fixed the follow errors:
>>    - Raising "TypeError: Unicode-objects must be encoded before
>> hashing" exception by hashlib.sha1().
>>    - get_default_jobs() is used to set the vm cpus, but it was
>> returning an invalid float number.
>>    - On check_sha256sum(), subprocess.check_output() returns bytes so
>> that string comparison was failing.
>>
>> Patch 03:
>> Implemented a simple mechanism to detect the image file changed on
>> http://download.patchew.org, as discussed in
>> https://www.mail-archive.com/qemu-devel@nongnu.org/msg607839.html. I
>> didn't implement the '--force' option that Paolo suggested, because it
>> would require a major refactoring (technically the vm implementation
>> decides whether use the cache or not). My intention is to keep the
>> script as simple as possible, so IMO, if the image file changed on the
>> server then it should equally be updated the checksum on QEMU's side.
>>
>> Patch 04:
>> Default network (NETWORK=1) backend is enough to have the docker-based
>> tests run properly. Note: `make docker-test-block@centos7` is broken
>> inside the CentOS VM and apparently on my host (Fedora 29 x86_64) too.
>>
>> Wainer dos Santos Moschetta (5):
>>    tests/vm: Use python configured on build
>>    tests/vm: Port basevm to Python 3
>>    tests/vm: Detect the image changed on server
>>    tests/vm: Fix build-centos docker-based tests run
>>    tests/vm: Add missing variables on help
>>
>>   tests/vm/Makefile.include | 12 ++++++++----
>>   tests/vm/basevm.py        | 39 ++++++++++++++++++++++++++++++++++-----
>>   tests/vm/centos           |  6 +++---
>>   3 files changed, 45 insertions(+), 12 deletions(-)
>>
> 



^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: [Qemu-devel] [PATCH 0/5] tests/vm: Python 3, improve image caching, and misc
  2019-04-24  9:07   ` Laszlo Ersek
  2019-04-24  9:07     ` Laszlo Ersek
@ 2019-04-25 18:02     ` Wainer dos Santos Moschetta
  2019-04-25 18:02       ` Wainer dos Santos Moschetta
  1 sibling, 1 reply; 9+ messages in thread
From: Wainer dos Santos Moschetta @ 2019-04-25 18:02 UTC (permalink / raw)
  To: Laszlo Ersek, qemu-devel
  Cc: fam, peter.maydell, philmd, pbonzini, alex.bennee

Hi Laszlo,


On 04/24/2019 06:07 AM, Laszlo Ersek wrote:
> Hello Wainer,
>
> (answering because I dislike ignoring emails without giving any feedback:)

I appreciated that! thanks!

>
> On 04/23/19 21:28, Wainer dos Santos Moschetta wrote:
>> Ping. More reviews needed.
>>
>> I've already got Philippe's reviewed-by, thanks!
> I'm going to skip this one. According to "scripts/get_maintainer.pl", an
> ACK from either Alex or Fam should suffice. (They could also decide to
> queue your set with Phil's R-b, who's a designated reviewer on this topic.)

Yes, that's ok.

I'm CC'ing you because the fix on patch 3/5 has origin on a discussion 
on your series ...

[PATCH for-4.1 v3 00/12] bundle edk2 platform firmware with QEMU

... see at https://www.mail-archive.com/qemu-devel@nongnu.org/msg606796.html

Sorry, failed to explain the context when I sent this series.

Thanks!

- Wainer

>
> Thanks
> Laszlo
>
>> On 03/29/2019 06:07 PM, Wainer dos Santos Moschetta wrote:
>>> Hi all!
>>>
>>> This series bundles the support to Python3, improvement to image
>>> caching, and miscellaneous changes for the vm-test (`make vm-build-*`).
>>>
>>> Git tree: http://github.com/wainersm/qemu
>>> Branch: vm_test_python3_and_misc
>>> Travis: https://travis-ci.org/wainersm/qemu/builds/513220300
>>>
>>> Below you can find some general comments.
>>>
>>> Patch 01:
>>> Uses python configured at build, although vm-test can be executed from
>>> the code tree (see docs/devel/testing.rst) too. In this case $(PYTHON)
>>> will be empty and it picks python pointed by the script's shebang, as
>>> a result failing on Python 3 only Linux distros.
>>>
>>> Patch 02:
>>> Ported based.py to work with Python 3 (kept Python 2 compatibility).
>>> Fixed the follow errors:
>>>     - Raising "TypeError: Unicode-objects must be encoded before
>>> hashing" exception by hashlib.sha1().
>>>     - get_default_jobs() is used to set the vm cpus, but it was
>>> returning an invalid float number.
>>>     - On check_sha256sum(), subprocess.check_output() returns bytes so
>>> that string comparison was failing.
>>>
>>> Patch 03:
>>> Implemented a simple mechanism to detect the image file changed on
>>> http://download.patchew.org, as discussed in
>>> https://www.mail-archive.com/qemu-devel@nongnu.org/msg607839.html. I
>>> didn't implement the '--force' option that Paolo suggested, because it
>>> would require a major refactoring (technically the vm implementation
>>> decides whether use the cache or not). My intention is to keep the
>>> script as simple as possible, so IMO, if the image file changed on the
>>> server then it should equally be updated the checksum on QEMU's side.
>>>
>>> Patch 04:
>>> Default network (NETWORK=1) backend is enough to have the docker-based
>>> tests run properly. Note: `make docker-test-block@centos7` is broken
>>> inside the CentOS VM and apparently on my host (Fedora 29 x86_64) too.
>>>
>>> Wainer dos Santos Moschetta (5):
>>>     tests/vm: Use python configured on build
>>>     tests/vm: Port basevm to Python 3
>>>     tests/vm: Detect the image changed on server
>>>     tests/vm: Fix build-centos docker-based tests run
>>>     tests/vm: Add missing variables on help
>>>
>>>    tests/vm/Makefile.include | 12 ++++++++----
>>>    tests/vm/basevm.py        | 39 ++++++++++++++++++++++++++++++++++-----
>>>    tests/vm/centos           |  6 +++---
>>>    3 files changed, 45 insertions(+), 12 deletions(-)
>>>

^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: [Qemu-devel] [PATCH 0/5] tests/vm: Python 3, improve image caching, and misc
  2019-04-25 18:02     ` Wainer dos Santos Moschetta
@ 2019-04-25 18:02       ` Wainer dos Santos Moschetta
  0 siblings, 0 replies; 9+ messages in thread
From: Wainer dos Santos Moschetta @ 2019-04-25 18:02 UTC (permalink / raw)
  To: Laszlo Ersek, qemu-devel
  Cc: fam, peter.maydell, alex.bennee, philmd, pbonzini

Hi Laszlo,


On 04/24/2019 06:07 AM, Laszlo Ersek wrote:
> Hello Wainer,
>
> (answering because I dislike ignoring emails without giving any feedback:)

I appreciated that! thanks!

>
> On 04/23/19 21:28, Wainer dos Santos Moschetta wrote:
>> Ping. More reviews needed.
>>
>> I've already got Philippe's reviewed-by, thanks!
> I'm going to skip this one. According to "scripts/get_maintainer.pl", an
> ACK from either Alex or Fam should suffice. (They could also decide to
> queue your set with Phil's R-b, who's a designated reviewer on this topic.)

Yes, that's ok.

I'm CC'ing you because the fix on patch 3/5 has origin on a discussion 
on your series ...

[PATCH for-4.1 v3 00/12] bundle edk2 platform firmware with QEMU

... see at https://www.mail-archive.com/qemu-devel@nongnu.org/msg606796.html

Sorry, failed to explain the context when I sent this series.

Thanks!

- Wainer

>
> Thanks
> Laszlo
>
>> On 03/29/2019 06:07 PM, Wainer dos Santos Moschetta wrote:
>>> Hi all!
>>>
>>> This series bundles the support to Python3, improvement to image
>>> caching, and miscellaneous changes for the vm-test (`make vm-build-*`).
>>>
>>> Git tree: http://github.com/wainersm/qemu
>>> Branch: vm_test_python3_and_misc
>>> Travis: https://travis-ci.org/wainersm/qemu/builds/513220300
>>>
>>> Below you can find some general comments.
>>>
>>> Patch 01:
>>> Uses python configured at build, although vm-test can be executed from
>>> the code tree (see docs/devel/testing.rst) too. In this case $(PYTHON)
>>> will be empty and it picks python pointed by the script's shebang, as
>>> a result failing on Python 3 only Linux distros.
>>>
>>> Patch 02:
>>> Ported based.py to work with Python 3 (kept Python 2 compatibility).
>>> Fixed the follow errors:
>>>     - Raising "TypeError: Unicode-objects must be encoded before
>>> hashing" exception by hashlib.sha1().
>>>     - get_default_jobs() is used to set the vm cpus, but it was
>>> returning an invalid float number.
>>>     - On check_sha256sum(), subprocess.check_output() returns bytes so
>>> that string comparison was failing.
>>>
>>> Patch 03:
>>> Implemented a simple mechanism to detect the image file changed on
>>> http://download.patchew.org, as discussed in
>>> https://www.mail-archive.com/qemu-devel@nongnu.org/msg607839.html. I
>>> didn't implement the '--force' option that Paolo suggested, because it
>>> would require a major refactoring (technically the vm implementation
>>> decides whether use the cache or not). My intention is to keep the
>>> script as simple as possible, so IMO, if the image file changed on the
>>> server then it should equally be updated the checksum on QEMU's side.
>>>
>>> Patch 04:
>>> Default network (NETWORK=1) backend is enough to have the docker-based
>>> tests run properly. Note: `make docker-test-block@centos7` is broken
>>> inside the CentOS VM and apparently on my host (Fedora 29 x86_64) too.
>>>
>>> Wainer dos Santos Moschetta (5):
>>>     tests/vm: Use python configured on build
>>>     tests/vm: Port basevm to Python 3
>>>     tests/vm: Detect the image changed on server
>>>     tests/vm: Fix build-centos docker-based tests run
>>>     tests/vm: Add missing variables on help
>>>
>>>    tests/vm/Makefile.include | 12 ++++++++----
>>>    tests/vm/basevm.py        | 39 ++++++++++++++++++++++++++++++++++-----
>>>    tests/vm/centos           |  6 +++---
>>>    3 files changed, 45 insertions(+), 12 deletions(-)
>>>



^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: [Qemu-devel] [PATCH 0/5] tests/vm: Python 3, improve image caching, and misc
  2019-04-23 19:28 ` [Qemu-devel] [PATCH 0/5] tests/vm: Python 3, improve image caching, and misc Wainer dos Santos Moschetta
  2019-04-23 19:28   ` Wainer dos Santos Moschetta
  2019-04-24  9:07   ` Laszlo Ersek
@ 2019-05-29 16:50   ` Alex Bennée
  2 siblings, 0 replies; 9+ messages in thread
From: Alex Bennée @ 2019-05-29 16:50 UTC (permalink / raw)
  To: Wainer dos Santos Moschetta
  Cc: fam, peter.maydell, lersek, qemu-devel, pbonzini, philmd


Wainer dos Santos Moschetta <wainersm@redhat.com> writes:

> Ping. More reviews needed.
>
> I've already got Philippe's reviewed-by, thanks!

I've queued patches 1,2,4,5 to testing/next, thanks.

I'm going to see is 3 is obviated by Gerd's VM rebuild work but I've
added it for now.

Sorry about the delay.

--
Alex Bennée


^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: [Qemu-devel] [PATCH 3/5] tests/vm: Detect the image changed on server
       [not found] ` <20190329210804.22121-4-wainersm@redhat.com>
@ 2019-05-29 19:05   ` Alex Bennée
  0 siblings, 0 replies; 9+ messages in thread
From: Alex Bennée @ 2019-05-29 19:05 UTC (permalink / raw)
  To: Wainer dos Santos Moschetta
  Cc: fam, peter.maydell, berrange, lersek, qemu-devel, pbonzini, philmd


Wainer dos Santos Moschetta <wainersm@redhat.com> writes:

> The current implementation of basevm does not check if the image
> file to be downloaded has changed on server side before honouring
> the cache. So any change on server-side file can go unnoticed,
> keeping the cached image.
>
> This change implements a simple mechanism to detect the image
> file changed by using the sha256sum file stored on server. It
> compares with the expected checksum and then abort the execution
> on mismatch.
>
> Signed-off-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
> ---
>  tests/vm/basevm.py | 31 ++++++++++++++++++++++++++++++-
>  1 file changed, 30 insertions(+), 1 deletion(-)
>
> diff --git a/tests/vm/basevm.py b/tests/vm/basevm.py
> index 083befce9f..4dfad2dc9b 100755
> --- a/tests/vm/basevm.py
> +++ b/tests/vm/basevm.py
> @@ -27,6 +27,7 @@ import tempfile
>  import shutil
>  import multiprocessing
>  import traceback
> +import urllib.request
>
>  SSH_KEY = open(os.path.join(os.path.dirname(__file__),
>                 "..", "keys", "id_rsa")).read()
> @@ -81,6 +82,18 @@ class BaseVM(object):
>          self._data_args = []
>
>      def _download_with_cache(self, url, sha256sum=None):
> +
> +        def fetch_image_hash(url):
> +            fetch_url = "%s.sha256sum" % url

OK this fails with the FreeBSD code as they use the form:

  https://download.freebsd.org/ftp/releases/ISO-IMAGES/12.0/CHECKSUM.SHA256-FreeBSD-12.0-RELEASE-amd64

I guess we need to have a method that can be overridden for this.

> +            try:
> +                with urllib.request.urlopen(fetch_url) as response:
> +                    content = response.read()
> +            except  urllib.error.URLError as error:
> +                logging.error("Failed to fetch image checksum file: %s",
> +                        fetch_url)
> +                raise error
> +            return content.decode().strip()
> +
>          def check_sha256sum(fname):
>              if not sha256sum:
>                  return True
> @@ -91,8 +104,24 @@ class BaseVM(object):
>          if not os.path.exists(cache_dir):
>              os.makedirs(cache_dir)
>          fname = os.path.join(cache_dir, hashlib.sha1(url.encode()).hexdigest())
> -        if os.path.exists(fname) and check_sha256sum(fname):
> +
> +        if os.path.exists(fname) and sha256sum is None:
>              return fname
> +
> +        if sha256sum:
> +            image_checksum = fetch_image_hash(url)
> +            # Check the url points to a known image file.
> +            if image_checksum != sha256sum:
> +                logging.error("Image %s checksum (%s) does not match " +
> +                        "expected (%s).", url, image_checksum, sha256sum)
> +                raise Exception("Image checksum failed.")
> +            # Check the cached image is up to date.
> +            if os.path.exists(fname):
> +                if check_sha256sum(fname):
> +                    return fname
> +                logging.warning("Invalid cached image. Attempt to download " +
> +                        "the updated one.")
> +
>          logging.debug("Downloading %s to %s...", url, fname)
>          subprocess.check_call(["wget", "-c", url, "-O", fname + ".download"],
>                                stdout=self._stdout, stderr=self._stderr)


--
Alex Bennée


^ permalink raw reply	[flat|nested] 9+ messages in thread

end of thread, other threads:[~2019-05-29 19:06 UTC | newest]

Thread overview: 9+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
     [not found] <20190329210804.22121-1-wainersm@redhat.com>
     [not found] ` <20190329210804.22121-3-wainersm@redhat.com>
2019-04-01 11:34   ` [Qemu-devel] [PATCH 2/5] tests/vm: Port basevm to Python 3 Philippe Mathieu-Daudé
2019-04-23 19:28 ` [Qemu-devel] [PATCH 0/5] tests/vm: Python 3, improve image caching, and misc Wainer dos Santos Moschetta
2019-04-23 19:28   ` Wainer dos Santos Moschetta
2019-04-24  9:07   ` Laszlo Ersek
2019-04-24  9:07     ` Laszlo Ersek
2019-04-25 18:02     ` Wainer dos Santos Moschetta
2019-04-25 18:02       ` Wainer dos Santos Moschetta
2019-05-29 16:50   ` Alex Bennée
     [not found] ` <20190329210804.22121-4-wainersm@redhat.com>
2019-05-29 19:05   ` [Qemu-devel] [PATCH 3/5] tests/vm: Detect the image changed on server Alex Bennée

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).