[Qemu-devel] Cryptic errors from PIP install if missing openssl-devel

[Qemu-devel] Cryptic errors from PIP install if missing openssl-devel

[David Gibson]

[-- Attachment #1: Type: text/plain, Size: 1761 bytes --]

If I attempt to run "make check-acceptance" on my POWER9, RHEL8.1
machine when the openssl-devel package isn't installed, I get the
following very cryptic error:

  VENV    /home/dwg/qemu/build/rhel8/tests/venv
  PIP     /home/dwg/qemu/tests/requirements.txt
Command "/home/dwg/qemu/build/rhel8/tests/venv/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-la4el5r5/cryptography/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-1efs22iz-record/install-record.txt --single-version-externally-managed --compile --install-headers /home/dwg/qemu/build/rhel8/tests/venv/include/site/python3.6/cryptography" failed with error code 1 in /tmp/pip-build-la4el5r5/cryptography/

Using V=1 doesn't give any more useful information, and it's not
(easily) possible to manually re-run the given command since it relies
on things in /tmp that are removed once the attempt finishes.

I only figured out it was openssl-devel being missing that was the
problem by (mostly) guesswork.  It would be really great if we could
generate a more helpful error here.

In addition, if I rerun "make check-acceptance" it no longer even
attempts the PIP install, since tests/venv already exists in my build
environment.  It then sort of works, but I think it might be hitting
other errors because of the missing python packages.  Sorry that's a
bit vague - I also seem to be getting unrelated errors that I'm still
trying to figure out.

-- 
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [Qemu-devel] Cryptic errors from PIP install if missing openssl-devel

[David Gibson]

[-- Attachment #1: Type: text/plain, Size: 2593 bytes --]

On Thu, Aug 29, 2019 at 11:31:25AM +1000, David Gibson wrote:
> If I attempt to run "make check-acceptance" on my POWER9, RHEL8.1
> machine when the openssl-devel package isn't installed, I get the
> following very cryptic error:
> 
>   VENV    /home/dwg/qemu/build/rhel8/tests/venv
>   PIP     /home/dwg/qemu/tests/requirements.txt
> Command "/home/dwg/qemu/build/rhel8/tests/venv/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-la4el5r5/cryptography/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-1efs22iz-record/install-record.txt --single-version-externally-managed --compile --install-headers /home/dwg/qemu/build/rhel8/tests/venv/include/site/python3.6/cryptography" failed with error code 1 in /tmp/pip-build-la4el5r5/cryptography/
> 
> Using V=1 doesn't give any more useful information, and it's not
> (easily) possible to manually re-run the given command since it relies
> on things in /tmp that are removed once the attempt finishes.
> 
> I only figured out it was openssl-devel being missing that was the
> problem by (mostly) guesswork.  It would be really great if we could
> generate a more helpful error here.
> 
> In addition, if I rerun "make check-acceptance" it no longer even
> attempts the PIP install, since tests/venv already exists in my build
> environment.  It then sort of works, but I think it might be hitting
> other errors because of the missing python packages.  Sorry that's a
> bit vague - I also seem to be getting unrelated errors that I'm still
> trying to figure out.

Fwiw, I also get an equally cryptic error that I haven't figured out
the cause for on my 32-bit Fedora container environment:

  VENV    /home/dwg/src/qemu/build/i386/tests/venv
  PIP     /home/dwg/src/qemu/tests/requirements.txt
  Failed building wheel for bcrypt
Could not build wheels for bcrypt which use PEP 517 and cannot be installed directly
You are using pip version 19.0.3, however version 19.2.3 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.

In this case the check definitely doesn't work - it doesn't appear to
have installed avocado in the venv.

  AVOCADO tests/acceptance
/home/dwg/src/qemu/build/i386/tests/venv/bin/python: No module named avocado

-- 
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [Qemu-devel] Cryptic errors from PIP install if missing openssl-devel

[Cleber Rosa]

On Thu, Aug 29, 2019 at 11:51:17AM +1000, David Gibson wrote:
> On Thu, Aug 29, 2019 at 11:31:25AM +1000, David Gibson wrote:
> > If I attempt to run "make check-acceptance" on my POWER9, RHEL8.1
> > machine when the openssl-devel package isn't installed, I get the
> > following very cryptic error:
> > 
> >   VENV    /home/dwg/qemu/build/rhel8/tests/venv
> >   PIP     /home/dwg/qemu/tests/requirements.txt
> > Command "/home/dwg/qemu/build/rhel8/tests/venv/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-la4el5r5/cryptography/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-1efs22iz-record/install-record.txt --single-version-externally-managed --compile --install-headers /home/dwg/qemu/build/rhel8/tests/venv/include/site/python3.6/cryptography" failed with error code 1 in /tmp/pip-build-la4el5r5/cryptography/
> > 
> > Using V=1 doesn't give any more useful information, and it's not
> > (easily) possible to manually re-run the given command since it relies
> > on things in /tmp that are removed once the attempt finishes.
> > 
> > I only figured out it was openssl-devel being missing that was the
> > problem by (mostly) guesswork.  It would be really great if we could
> > generate a more helpful error here.
> > 
> > In addition, if I rerun "make check-acceptance" it no longer even
> > attempts the PIP install, since tests/venv already exists in my build
> > environment.  It then sort of works, but I think it might be hitting
> > other errors because of the missing python packages.  Sorry that's a
> > bit vague - I also seem to be getting unrelated errors that I'm still
> > trying to figure out.
> 
> Fwiw, I also get an equally cryptic error that I haven't figured out
> the cause for on my 32-bit Fedora container environment:
> 
>   VENV    /home/dwg/src/qemu/build/i386/tests/venv
>   PIP     /home/dwg/src/qemu/tests/requirements.txt
>   Failed building wheel for bcrypt
> Could not build wheels for bcrypt which use PEP 517 and cannot be installed directly
> You are using pip version 19.0.3, however version 19.2.3 is available.
> You should consider upgrading via the 'pip install --upgrade pip' command.
>

This is certainly caused by pip not being able to install paramiko on
those systems.  I have dealt with paramiko (and its many dependencies)
before on the avocado remote runner plugin (which is not being used
here) and it was not fun.

My personal goal was to rely on the ssh binary as an ssh client, which
should be more ubiquitous, and with that I added a simple wrapper to
Avocado:

  https://avocado-framework.readthedocs.io/en/71.0/api/utils/avocado.utils.html#module-avocado.utils.ssh

I guess we should consider changing the (few) tests that require
paramiko to use that module instead.

> In this case the check definitely doesn't work - it doesn't appear to
> have installed avocado in the venv.
> 
>   AVOCADO tests/acceptance
> /home/dwg/src/qemu/build/i386/tests/venv/bin/python: No module named avocado
>

My hope is that with pure Python modules in requirements.txt, failures
will be much harder to come by.  Either way, it'd be nice to improve
the venv creation, at the very least add a cleanup if it fails to
complete successfully.

As a workaround I'd suggest two things:

 1) remove paramiko from requirements.txt
 2) set the env var CONTINUOUS_INTEGRATION=1 before running `make
    check-acceptace` (the tests in linux_ssh_mips_malta.py check for
    that and don't run if it's set).

But, we'll need to address those failures definitely ASAP.  Anyway,
thanks for reporting it.

- Cleber.

> -- 
> David Gibson			| I'll have my music baroque, and my code
> david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
> 				| _way_ _around_!
> http://www.ozlabs.org/~dgibson

Re: [Qemu-devel] Cryptic errors from PIP install if missing openssl-devel

[Philippe Mathieu-Daudé]

On 8/29/19 5:27 AM, Cleber Rosa wrote:
> On Thu, Aug 29, 2019 at 11:51:17AM +1000, David Gibson wrote:
>> On Thu, Aug 29, 2019 at 11:31:25AM +1000, David Gibson wrote:
>>> If I attempt to run "make check-acceptance" on my POWER9, RHEL8.1
>>> machine when the openssl-devel package isn't installed, I get the
>>> following very cryptic error:
>>>
>>>   VENV    /home/dwg/qemu/build/rhel8/tests/venv
>>>   PIP     /home/dwg/qemu/tests/requirements.txt
>>> Command "/home/dwg/qemu/build/rhel8/tests/venv/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-la4el5r5/cryptography/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-1efs22iz-record/install-record.txt --single-version-externally-managed --compile --install-headers /home/dwg/qemu/build/rhel8/tests/venv/include/site/python3.6/cryptography" failed with error code 1 in /tmp/pip-build-la4el5r5/cryptography/
>>>
>>> Using V=1 doesn't give any more useful information, and it's not
>>> (easily) possible to manually re-run the given command since it relies
>>> on things in /tmp that are removed once the attempt finishes.
>>>
>>> I only figured out it was openssl-devel being missing that was the
>>> problem by (mostly) guesswork.  It would be really great if we could
>>> generate a more helpful error here.
>>>
>>> In addition, if I rerun "make check-acceptance" it no longer even
>>> attempts the PIP install, since tests/venv already exists in my build
>>> environment.  It then sort of works, but I think it might be hitting
>>> other errors because of the missing python packages.  Sorry that's a
>>> bit vague - I also seem to be getting unrelated errors that I'm still
>>> trying to figure out.
>>
>> Fwiw, I also get an equally cryptic error that I haven't figured out
>> the cause for on my 32-bit Fedora container environment:
>>
>>   VENV    /home/dwg/src/qemu/build/i386/tests/venv
>>   PIP     /home/dwg/src/qemu/tests/requirements.txt
>>   Failed building wheel for bcrypt
>> Could not build wheels for bcrypt which use PEP 517 and cannot be installed directly
>> You are using pip version 19.0.3, however version 19.2.3 is available.
>> You should consider upgrading via the 'pip install --upgrade pip' command.
>>
> 
> This is certainly caused by pip not being able to install paramiko on
> those systems.  I have dealt with paramiko (and its many dependencies)
> before on the avocado remote runner plugin (which is not being used
> here) and it was not fun.
> 
> My personal goal was to rely on the ssh binary as an ssh client, which
> should be more ubiquitous, and with that I added a simple wrapper to
> Avocado:
> 
>   https://avocado-framework.readthedocs.io/en/71.0/api/utils/avocado.utils.html#module-avocado.utils.ssh

I did not know this module.

> I guess we should consider changing the (few) tests that require
> paramiko to use that module instead.

Clean way to resolve this issue.

>> In this case the check definitely doesn't work - it doesn't appear to
>> have installed avocado in the venv.
>>
>>   AVOCADO tests/acceptance
>> /home/dwg/src/qemu/build/i386/tests/venv/bin/python: No module named avocado
>>
> 
> My hope is that with pure Python modules in requirements.txt, failures
> will be much harder to come by.  Either way, it'd be nice to improve

Is there a way to check for this before submitting/merging patches?

Some checkpatch.py lines that report:

  The 'paramiko' module is not pure Python and can
  not be added in requirements.txt.

> the venv creation, at the very least add a cleanup if it fails to
> complete successfully.
> 
> As a workaround I'd suggest two things:
> 
>  1) remove paramiko from requirements.txt
>  2) set the env var CONTINUOUS_INTEGRATION=1 before running `make
>     check-acceptace` (the tests in linux_ssh_mips_malta.py check for
>     that and don't run if it's set).
> 
> But, we'll need to address those failures definitely ASAP.  Anyway,
> thanks for reporting it.
> 
> - Cleber.
> 
>> -- 
>> David Gibson			| I'll have my music baroque, and my code
>> david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
>> 				| _way_ _around_!
>> http://www.ozlabs.org/~dgibson
> 
> 
> 

Re: [Qemu-devel] Cryptic errors from PIP install if missing openssl-devel

[Philippe Mathieu-Daudé]

Hi Cleber, Lukáš,

On 8/29/19 11:24 AM, Philippe Mathieu-Daudé wrote:
> On 8/29/19 5:27 AM, Cleber Rosa wrote:
>> On Thu, Aug 29, 2019 at 11:51:17AM +1000, David Gibson wrote:
[...]
>>> Fwiw, I also get an equally cryptic error that I haven't figured out
>>> the cause for on my 32-bit Fedora container environment:
>>>
>>>   VENV    /home/dwg/src/qemu/build/i386/tests/venv
>>>   PIP     /home/dwg/src/qemu/tests/requirements.txt
>>>   Failed building wheel for bcrypt
>>> Could not build wheels for bcrypt which use PEP 517 and cannot be installed directly
>>> You are using pip version 19.0.3, however version 19.2.3 is available.
>>> You should consider upgrading via the 'pip install --upgrade pip' command.
>>>
>>
>> This is certainly caused by pip not being able to install paramiko on
>> those systems.  I have dealt with paramiko (and its many dependencies)
>> before on the avocado remote runner plugin (which is not being used
>> here) and it was not fun.
>>
>> My personal goal was to rely on the ssh binary as an ssh client, which
>> should be more ubiquitous, and with that I added a simple wrapper to
>> Avocado:
>>
>>   https://avocado-framework.readthedocs.io/en/71.0/api/utils/avocado.utils.html#module-avocado.utils.ssh
> 
> I did not know this module.

class avocado.utils.ssh.Session(address, credentials)

  Parameters:	

    credentials (tuple)
         username and path to a key for authentication purposes

The current test uses username + password.
Can we use this credentials with the Avocado module?
(The image used is prebuilt).

>> I guess we should consider changing the (few) tests that require
>> paramiko to use that module instead.
> 
> Clean way to resolve this issue.

Re: [Qemu-devel] Cryptic errors from PIP install if missing openssl-devel

[Eduardo Habkost]

On Thu, Aug 29, 2019 at 11:31:25AM +1000, David Gibson wrote:
> If I attempt to run "make check-acceptance" on my POWER9, RHEL8.1
> machine when the openssl-devel package isn't installed, I get the
> following very cryptic error:
> 
>   VENV    /home/dwg/qemu/build/rhel8/tests/venv
>   PIP     /home/dwg/qemu/tests/requirements.txt
> Command "/home/dwg/qemu/build/rhel8/tests/venv/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-la4el5r5/cryptography/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-1efs22iz-record/install-record.txt --single-version-externally-managed --compile --install-headers /home/dwg/qemu/build/rhel8/tests/venv/include/site/python3.6/cryptography" failed with error code 1 in /tmp/pip-build-la4el5r5/cryptography/
> 
> Using V=1 doesn't give any more useful information, and it's not
> (easily) possible to manually re-run the given command since it relies
> on things in /tmp that are removed once the attempt finishes.

V=1 is supposed to show the actual pip command being run.  I see
it here:

  $ make check-venv V=1
  [...]
  python3 -B -m venv --system-site-packages /home/ehabkost/rh/proj/virt/qemu/tests/venv
* /home/ehabkost/rh/proj/virt/qemu/tests/venv/bin/python -m pip -q install -r /home/ehabkost/rh/proj/virt/qemu/tests/requirements.txt
  You are using pip version 19.0.3, however version 19.2.3 is available.
  You should consider upgrading via the 'pip install --upgrade pip' command.
  touch /home/ehabkost/rh/proj/virt/qemu/tests/venv

Can you reproduce the problem if you re-run the pip command?


> 
> I only figured out it was openssl-devel being missing that was the
> problem by (mostly) guesswork.  It would be really great if we could
> generate a more helpful error here.

I don't think there's much we can do about it, as the error is generated by a
package being built by pip.

There's one thing we can do to make it slightly better: not passing `-q` to pip
if running with V=1, so people can see which packages are being installed and
which one is generating the error.

> 
> In addition, if I rerun "make check-acceptance" it no longer even
> attempts the PIP install, since tests/venv already exists in my build
> environment.  It then sort of works, but I think it might be hitting
> other errors because of the missing python packages.  Sorry that's a
> bit vague - I also seem to be getting unrelated errors that I'm still
> trying to figure out.

This sounds like a bug and we need to fix it.

-- 
Eduardo

Re: [Qemu-devel] Cryptic errors from PIP install if missing openssl-devel

[David Gibson]

[-- Attachment #1: Type: text/plain, Size: 3118 bytes --]

On Fri, Aug 30, 2019 at 02:56:48PM -0300, Eduardo Habkost wrote:
> On Thu, Aug 29, 2019 at 11:31:25AM +1000, David Gibson wrote:
> > If I attempt to run "make check-acceptance" on my POWER9, RHEL8.1
> > machine when the openssl-devel package isn't installed, I get the
> > following very cryptic error:
> > 
> >   VENV    /home/dwg/qemu/build/rhel8/tests/venv
> >   PIP     /home/dwg/qemu/tests/requirements.txt
> > Command "/home/dwg/qemu/build/rhel8/tests/venv/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-la4el5r5/cryptography/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-1efs22iz-record/install-record.txt --single-version-externally-managed --compile --install-headers /home/dwg/qemu/build/rhel8/tests/venv/include/site/python3.6/cryptography" failed with error code 1 in /tmp/pip-build-la4el5r5/cryptography/
> > 
> > Using V=1 doesn't give any more useful information, and it's not
> > (easily) possible to manually re-run the given command since it relies
> > on things in /tmp that are removed once the attempt finishes.
> 
> V=1 is supposed to show the actual pip command being run.  I see
> it here:
> 
>   $ make check-venv V=1
>   [...]
>   python3 -B -m venv --system-site-packages /home/ehabkost/rh/proj/virt/qemu/tests/venv
> * /home/ehabkost/rh/proj/virt/qemu/tests/venv/bin/python -m pip -q install -r /home/ehabkost/rh/proj/virt/qemu/tests/requirements.txt
>   You are using pip version 19.0.3, however version 19.2.3 is available.
>   You should consider upgrading via the 'pip install --upgrade pip' command.
>   touch /home/ehabkost/rh/proj/virt/qemu/tests/venv
> 
> Can you reproduce the problem if you re-run the pip command?

Yes, I can, but it's basically just the same error without any extra
information.

> > I only figured out it was openssl-devel being missing that was the
> > problem by (mostly) guesswork.  It would be really great if we could
> > generate a more helpful error here.
> 
> I don't think there's much we can do about it, as the error is generated by a
> package being built by pip.
> 
> There's one thing we can do to make it slightly better: not passing `-q` to pip
> if running with V=1, so people can see which packages are being installed and
> which one is generating the error.

That sounds like a good start to me.

> > In addition, if I rerun "make check-acceptance" it no longer even
> > attempts the PIP install, since tests/venv already exists in my build
> > environment.  It then sort of works, but I think it might be hitting
> > other errors because of the missing python packages.  Sorry that's a
> > bit vague - I also seem to be getting unrelated errors that I'm still
> > trying to figure out.
> 
> This sounds like a bug and we need to fix it.

I agree.

-- 
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [Qemu-devel] Cryptic errors from PIP install if missing openssl-devel

[Cleber Rosa]

On Thu, Aug 29, 2019 at 11:46:15AM +0200, Philippe Mathieu-Daudé wrote:
> 
> class avocado.utils.ssh.Session(address, credentials)
> 
>   Parameters:	
> 
>     credentials (tuple)
>          username and path to a key for authentication purposes
> 
> The current test uses username + password.
> Can we use this credentials with the Avocado module?
> (The image used is prebuilt).
> 

I'm working on adding password based authentication.  To keep the API
the same, I'm thinking of checking if the second credential item is an
existing file, if it is, assume one containing a key.  If not, assume
it's a password.

This should make the use simple in the case of keys:

  with Session(('hostname', port),
               ('username', '/path/to/key')) as session:
      session.cmd('cmd')

And passwords:

  with Session(('hostname', port),
               ('username', 'p@ssw0rD')) as session:
      session.cmd('cmd')

It's being tracked here:
  https://trello.com/c/uetpIgML/1517-avocadoutilssh-implement-password-based-auth

I'll try to have it in Avocado's 72.0 release due next week.

Let me know how that sounds, and thanks for the feedback.
- Cleber.

Re: [Qemu-devel] Cryptic errors from PIP install if missing openssl-devel

[Cleber Rosa]

On Thu, Aug 29, 2019 at 11:24:30AM +0200, Philippe Mathieu-Daudé wrote:
> Is there a way to check for this before submitting/merging patches?
> 
> Some checkpatch.py lines that report:
> 
>   The 'paramiko' module is not pure Python and can
>   not be added in requirements.txt.
>

I dont think there is... at least, I can't think of one.  None of the
PyPI classifiers[1] seems to cover the "pure Python .vs. compiled
code" installation requirements.  AFAIK, this is usually buried in the
setup.py code of each package, so it's not easy to inspect it.

But, we can define "only Pure Python modules" as a policy, document it
and hopefully flag it on code review.

[1] https://pypi.org/classifiers/

- Cleber.

Re: [Qemu-devel] Cryptic errors from PIP install if missing openssl-devel

[Philippe Mathieu-Daudé]

On 9/3/19 5:08 PM, Cleber Rosa wrote:
> On Thu, Aug 29, 2019 at 11:46:15AM +0200, Philippe Mathieu-Daudé wrote:
>>
>> class avocado.utils.ssh.Session(address, credentials)
>>
>>   Parameters:	
>>
>>     credentials (tuple)
>>          username and path to a key for authentication purposes
>>
>> The current test uses username + password.
>> Can we use this credentials with the Avocado module?
>> (The image used is prebuilt).
>>
> 
> I'm working on adding password based authentication.  To keep the API
> the same, I'm thinking of checking if the second credential item is an
> existing file, if it is, assume one containing a key.  If not, assume
> it's a password.

Why not use a dictionary? Keys would be explicit.

> This should make the use simple in the case of keys:
> 
>   with Session(('hostname', port),
>                ('username', '/path/to/key')) as session:
>       session.cmd('cmd')
> 
> And passwords:
> 
>   with Session(('hostname', port),
>                ('username', 'p@ssw0rD')) as session:
>       session.cmd('cmd')
> 
> It's being tracked here:
>   https://trello.com/c/uetpIgML/1517-avocadoutilssh-implement-password-based-auth
> 
> I'll try to have it in Avocado's 72.0 release due next week.
> 
> Let me know how that sounds, and thanks for the feedback.
> - Cleber.
> 

Re: [Qemu-devel] Cryptic errors from PIP install if missing openssl-devel

[Cleber Rosa]

On Sat, Aug 31, 2019 at 11:48:34AM +1000, David Gibson wrote:
> On Fri, Aug 30, 2019 at 02:56:48PM -0300, Eduardo Habkost wrote:
> > On Thu, Aug 29, 2019 at 11:31:25AM +1000, David Gibson wrote:
> > > If I attempt to run "make check-acceptance" on my POWER9, RHEL8.1
> > > machine when the openssl-devel package isn't installed, I get the
> > > following very cryptic error:
> > > 
> > >   VENV    /home/dwg/qemu/build/rhel8/tests/venv
> > >   PIP     /home/dwg/qemu/tests/requirements.txt
> > > Command "/home/dwg/qemu/build/rhel8/tests/venv/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-la4el5r5/cryptography/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-1efs22iz-record/install-record.txt --single-version-externally-managed --compile --install-headers /home/dwg/qemu/build/rhel8/tests/venv/include/site/python3.6/cryptography" failed with error code 1 in /tmp/pip-build-la4el5r5/cryptography/
> > > 
> > > Using V=1 doesn't give any more useful information, and it's not
> > > (easily) possible to manually re-run the given command since it relies
> > > on things in /tmp that are removed once the attempt finishes.
> > 
> > V=1 is supposed to show the actual pip command being run.  I see
> > it here:
> > 
> >   $ make check-venv V=1
> >   [...]
> >   python3 -B -m venv --system-site-packages /home/ehabkost/rh/proj/virt/qemu/tests/venv
> > * /home/ehabkost/rh/proj/virt/qemu/tests/venv/bin/python -m pip -q install -r /home/ehabkost/rh/proj/virt/qemu/tests/requirements.txt
> >   You are using pip version 19.0.3, however version 19.2.3 is available.
> >   You should consider upgrading via the 'pip install --upgrade pip' command.
> >   touch /home/ehabkost/rh/proj/virt/qemu/tests/venv
> > 
> > Can you reproduce the problem if you re-run the pip command?
> 
> Yes, I can, but it's basically just the same error without any extra
> information.
> 
> > > I only figured out it was openssl-devel being missing that was the
> > > problem by (mostly) guesswork.  It would be really great if we could
> > > generate a more helpful error here.
> > 
> > I don't think there's much we can do about it, as the error is generated by a
> > package being built by pip.
> > 
> > There's one thing we can do to make it slightly better: not passing `-q` to pip
> > if running with V=1, so people can see which packages are being installed and
> > which one is generating the error.

+1

> 
> That sounds like a good start to me.
> 
> > > In addition, if I rerun "make check-acceptance" it no longer even
> > > attempts the PIP install, since tests/venv already exists in my build
> > > environment.  It then sort of works, but I think it might be hitting
> > > other errors because of the missing python packages.  Sorry that's a
> > > bit vague - I also seem to be getting unrelated errors that I'm still
> > > trying to figure out.
> >

David,

I'm getting myself a Power machine to try to reproduce your experience.
I'll keep you posted.

> > This sounds like a bug and we need to fix it.
> 
> I agree.
>

Yep, definitely.

- Cleber.

> -- 
> David Gibson			| I'll have my music baroque, and my code
> david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
> 				| _way_ _around_!
> http://www.ozlabs.org/~dgibson

Re: [Qemu-devel] Cryptic errors from PIP install if missing openssl-devel

[Cleber Rosa]

On Tue, Sep 03, 2019 at 05:22:48PM +0200, Philippe Mathieu-Daudé wrote:
> On 9/3/19 5:08 PM, Cleber Rosa wrote:
> > On Thu, Aug 29, 2019 at 11:46:15AM +0200, Philippe Mathieu-Daudé wrote:
> >>
> >> class avocado.utils.ssh.Session(address, credentials)
> >>
> >>   Parameters:	
> >>
> >>     credentials (tuple)
> >>          username and path to a key for authentication purposes
> >>
> >> The current test uses username + password.
> >> Can we use this credentials with the Avocado module?
> >> (The image used is prebuilt).
> >>
> > 
> > I'm working on adding password based authentication.  To keep the API
> > the same, I'm thinking of checking if the second credential item is an
> > existing file, if it is, assume one containing a key.  If not, assume
> > it's a password.
> 
> Why not use a dictionary? Keys would be explicit.
>

Now it's clear that a dict would've been the best option from the
beginning, but I was wrongly optimistic, and biased by the
"avocado.utils.vmimage" + "avocado.utils.cloudinit" combination, that
we'd only have to deal with key based auth.

So the question now is really how to evolve the API, either breaking
the current version or not.  At this time, I'd try to keep the API
unchanged, given that it still feels logical that the tuple is about
credentials, just that the second item can be either a path to a key
or password.

Thoughts?
- Cleber.

> > This should make the use simple in the case of keys:
> > 
> >   with Session(('hostname', port),
> >                ('username', '/path/to/key')) as session:
> >       session.cmd('cmd')
> > 
> > And passwords:
> > 
> >   with Session(('hostname', port),
> >                ('username', 'p@ssw0rD')) as session:
> >       session.cmd('cmd')
> > 
> > It's being tracked here:
> >   https://trello.com/c/uetpIgML/1517-avocadoutilssh-implement-password-based-auth
> > 
> > I'll try to have it in Avocado's 72.0 release due next week.
> > 
> > Let me know how that sounds, and thanks for the feedback.
> > - Cleber.
> > 

Re: [Qemu-devel] Cryptic errors from PIP install if missing openssl-devel

[Lukáš Doktor]

[-- Attachment #1.1: Type: text/plain, Size: 2490 bytes --]

Dne 03. 09. 19 v 17:47 Cleber Rosa napsal(a):
> On Tue, Sep 03, 2019 at 05:22:48PM +0200, Philippe Mathieu-Daudé wrote:
>> On 9/3/19 5:08 PM, Cleber Rosa wrote:
>>> On Thu, Aug 29, 2019 at 11:46:15AM +0200, Philippe Mathieu-Daudé wrote:
>>>>
>>>> class avocado.utils.ssh.Session(address, credentials)
>>>>
>>>>   Parameters:	
>>>>
>>>>     credentials (tuple)
>>>>          username and path to a key for authentication purposes
>>>>
>>>> The current test uses username + password.
>>>> Can we use this credentials with the Avocado module?
>>>> (The image used is prebuilt).
>>>>
>>>
>>> I'm working on adding password based authentication.  To keep the API
>>> the same, I'm thinking of checking if the second credential item is an
>>> existing file, if it is, assume one containing a key.  If not, assume
>>> it's a password.
>>
>> Why not use a dictionary? Keys would be explicit.
>>
> 
> Now it's clear that a dict would've been the best option from the
> beginning, but I was wrongly optimistic, and biased by the
> "avocado.utils.vmimage" + "avocado.utils.cloudinit" combination, that
> we'd only have to deal with key based auth.
> 
> So the question now is really how to evolve the API, either breaking
> the current version or not.  At this time, I'd try to keep the API
> unchanged, given that it still feels logical that the tuple is about
> credentials, just that the second item can be either a path to a key
> or password.
> 
> Thoughts?
> - Cleber.
> 

How old is this feature? I guess it's not yet widely used so it's probably better to change it now than suffer the consequences when hundreds of people rely on it...

Regards,
Lukáš

PS: My favorited solution would be:

    with Session(addr=(hostname, port), username="user", key="/path/to/key", password="pass"):
        ...


>>> This should make the use simple in the case of keys:
>>>
>>>   with Session(('hostname', port),
>>>                ('username', '/path/to/key')) as session:
>>>       session.cmd('cmd')
>>>
>>> And passwords:
>>>
>>>   with Session(('hostname', port),
>>>                ('username', 'p@ssw0rD')) as session:
>>>       session.cmd('cmd')
>>>
>>> It's being tracked here:
>>>   https://trello.com/c/uetpIgML/1517-avocadoutilssh-implement-password-based-auth
>>>
>>> I'll try to have it in Avocado's 72.0 release due next week.
>>>
>>> Let me know how that sounds, and thanks for the feedback.
>>> - Cleber.
>>>



[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [Qemu-devel] Cryptic errors from PIP install if missing openssl-devel

[Lukáš Doktor]

[-- Attachment #1.1: Type: text/plain, Size: 2849 bytes --]

Dne 03. 09. 19 v 17:56 Lukáš Doktor napsal(a):
> Dne 03. 09. 19 v 17:47 Cleber Rosa napsal(a):
>> On Tue, Sep 03, 2019 at 05:22:48PM +0200, Philippe Mathieu-Daudé wrote:
>>> On 9/3/19 5:08 PM, Cleber Rosa wrote:
>>>> On Thu, Aug 29, 2019 at 11:46:15AM +0200, Philippe Mathieu-Daudé wrote:
>>>>>
>>>>> class avocado.utils.ssh.Session(address, credentials)
>>>>>
>>>>>   Parameters:	
>>>>>
>>>>>     credentials (tuple)
>>>>>          username and path to a key for authentication purposes
>>>>>
>>>>> The current test uses username + password.
>>>>> Can we use this credentials with the Avocado module?
>>>>> (The image used is prebuilt).
>>>>>
>>>>
>>>> I'm working on adding password based authentication.  To keep the API
>>>> the same, I'm thinking of checking if the second credential item is an
>>>> existing file, if it is, assume one containing a key.  If not, assume
>>>> it's a password.
>>>
>>> Why not use a dictionary? Keys would be explicit.
>>>
>>
>> Now it's clear that a dict would've been the best option from the
>> beginning, but I was wrongly optimistic, and biased by the
>> "avocado.utils.vmimage" + "avocado.utils.cloudinit" combination, that
>> we'd only have to deal with key based auth.
>>
>> So the question now is really how to evolve the API, either breaking
>> the current version or not.  At this time, I'd try to keep the API
>> unchanged, given that it still feels logical that the tuple is about
>> credentials, just that the second item can be either a path to a key
>> or password.
>>
>> Thoughts?
>> - Cleber.
>>
> 
> How old is this feature? I guess it's not yet widely used so it's probably better to change it now than suffer the consequences when hundreds of people rely on it...
> 
> Regards,
> Lukáš
> 
> PS: My favorited solution would be:
> 
>     with Session(addr=(hostname, port), username="user", key="/path/to/key", password="pass"):
>         ...

... actually I'd even avoid the use of tuple and simply add `addr=hostname, port=None, username...`. (basically the way it's in Avocado-vt, it works well there with some additional options ;-) )

Regards,
Lukáš

> 
> 
>>>> This should make the use simple in the case of keys:
>>>>
>>>>   with Session(('hostname', port),
>>>>                ('username', '/path/to/key')) as session:
>>>>       session.cmd('cmd')
>>>>
>>>> And passwords:
>>>>
>>>>   with Session(('hostname', port),
>>>>                ('username', 'p@ssw0rD')) as session:
>>>>       session.cmd('cmd')
>>>>
>>>> It's being tracked here:
>>>>   https://trello.com/c/uetpIgML/1517-avocadoutilssh-implement-password-based-auth
>>>>
>>>> I'll try to have it in Avocado's 72.0 release due next week.
>>>>
>>>> Let me know how that sounds, and thanks for the feedback.
>>>> - Cleber.
>>>>
> 
> 



[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [Qemu-devel] Cryptic errors from PIP install if missing openssl-devel

[Eduardo Habkost]

On Tue, Sep 03, 2019 at 11:08:24AM -0400, Cleber Rosa wrote:
> On Thu, Aug 29, 2019 at 11:46:15AM +0200, Philippe Mathieu-Daudé wrote:
> > 
> > class avocado.utils.ssh.Session(address, credentials)
> > 
> >   Parameters:	
> > 
> >     credentials (tuple)
> >          username and path to a key for authentication purposes
> > 
> > The current test uses username + password.
> > Can we use this credentials with the Avocado module?
> > (The image used is prebuilt).
> > 
> 
> I'm working on adding password based authentication.  To keep the API
> the same, I'm thinking of checking if the second credential item is an
> existing file, if it is, assume one containing a key.  If not, assume
> it's a password.
> 
> This should make the use simple in the case of keys:
> 
>   with Session(('hostname', port),
>                ('username', '/path/to/key')) as session:
>       session.cmd('cmd')
> 
> And passwords:
> 
>   with Session(('hostname', port),
>                ('username', 'p@ssw0rD')) as session:
>       session.cmd('cmd')

This would stop working as soon as a file named 'p@ssw0rD' gets
created.  Can the API be more explicit somehow?

Is there anything that prevents us from using keyword arguments?
e.g.:

  Session(..., username='root', password='p@ssw0rD')

> 
> It's being tracked here:
>   https://trello.com/c/uetpIgML/1517-avocadoutilssh-implement-password-based-auth
> 
> I'll try to have it in Avocado's 72.0 release due next week.
> 
> Let me know how that sounds, and thanks for the feedback.
> - Cleber.

-- 
Eduardo

Re: [Qemu-devel] Cryptic errors from PIP install if missing openssl-devel

[Cleber Rosa]

On Tue, Sep 03, 2019 at 02:27:58PM -0300, Eduardo Habkost wrote:
> This would stop working as soon as a file named 'p@ssw0rD' gets
> created.  Can the API be more explicit somehow?
> 
> Is there anything that prevents us from using keyword arguments?
> e.g.:
> 
>   Session(..., username='root', password='p@ssw0rD')
>

OK, we have enough fair criticism of the approach that would keep the
API compatible, so let's implement a better, not non-backwards
compatible version, like suggested here.

Thanks for the feedback!
- Cleber.

Re: [Qemu-devel] Cryptic errors from PIP install if missing openssl-devel

[Cleber Rosa]

On Sat, Aug 31, 2019 at 11:48:34AM +1000, David Gibson wrote:
> On Fri, Aug 30, 2019 at 02:56:48PM -0300, Eduardo Habkost wrote:
> > On Thu, Aug 29, 2019 at 11:31:25AM +1000, David Gibson wrote:
> > > If I attempt to run "make check-acceptance" on my POWER9, RHEL8.1
> > > machine when the openssl-devel package isn't installed, I get the
> > > following very cryptic error:
> > > 
> > >   VENV    /home/dwg/qemu/build/rhel8/tests/venv
> > >   PIP     /home/dwg/qemu/tests/requirements.txt
> > > Command "/home/dwg/qemu/build/rhel8/tests/venv/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-la4el5r5/cryptography/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-1efs22iz-record/install-record.txt --single-version-externally-managed --compile --install-headers /home/dwg/qemu/build/rhel8/tests/venv/include/site/python3.6/cryptography" failed with error code 1 in /tmp/pip-build-la4el5r5/cryptography/
> > > 
> > > Using V=1 doesn't give any more useful information, and it's not
> > > (easily) possible to manually re-run the given command since it relies
> > > on things in /tmp that are removed once the attempt finishes.
> > 
> > V=1 is supposed to show the actual pip command being run.  I see
> > it here:
> > 
> >   $ make check-venv V=1
> >   [...]
> >   python3 -B -m venv --system-site-packages /home/ehabkost/rh/proj/virt/qemu/tests/venv
> > * /home/ehabkost/rh/proj/virt/qemu/tests/venv/bin/python -m pip -q install -r /home/ehabkost/rh/proj/virt/qemu/tests/requirements.txt
> >   You are using pip version 19.0.3, however version 19.2.3 is available.
> >   You should consider upgrading via the 'pip install --upgrade pip' command.
> >   touch /home/ehabkost/rh/proj/virt/qemu/tests/venv
> > 
> > Can you reproduce the problem if you re-run the pip command?
> 
> Yes, I can, but it's basically just the same error without any extra
> information.
> 
> > > I only figured out it was openssl-devel being missing that was the
> > > problem by (mostly) guesswork.  It would be really great if we could
> > > generate a more helpful error here.
> > 
> > I don't think there's much we can do about it, as the error is generated by a
> > package being built by pip.
> > 
> > There's one thing we can do to make it slightly better: not passing `-q` to pip
> > if running with V=1, so people can see which packages are being installed and
> > which one is generating the error.
> 
> That sounds like a good start to me.
> 
> > > In addition, if I rerun "make check-acceptance" it no longer even
> > > attempts the PIP install, since tests/venv already exists in my build
> > > environment.  It then sort of works, but I think it might be hitting
> > > other errors because of the missing python packages.  Sorry that's a
> > > bit vague - I also seem to be getting unrelated errors that I'm still
> > > trying to figure out.
> > 
> > This sounds like a bug and we need to fix it.
> 
> I agree.

Hi David,

I've pushed a branch here (most of the commits have already been sent
separately):

   https://github.com/clebergnu/qemu/tree/ppc64

I've tested on a RHEL 8 ppc64le Power 9, and it seems to work for me.
My steps for building QEMU:

  - Configured with: '/home/cleber/src/qemu/configure' '--enable-slirp=git' '--python=/usr/bin/python3' '--target-list=x86_64-softmmu,ppc64-softmmu,arm-softmmu,aarch64-softmmu,mips-softmmu,mipsel-softmmu,mips64-softmmu,mips64el-softmmu,sh4-softmmu,s390x-softmmu,alpha-softmmu,m68k-softmmu,riscv64-softmmu'
  - make
  - make check-acceptance

Would you be able to test if that branch works smoothly for you?

Thanks,
- Cleber.

> 
> -- 
> David Gibson			| I'll have my music baroque, and my code
> david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
> 				| _way_ _around_!
> http://www.ozlabs.org/~dgibson

Re: [Qemu-devel] Cryptic errors from PIP install if missing openssl-devel

[David Gibson]

[-- Attachment #1: Type: text/plain, Size: 5337 bytes --]

On Wed, Sep 04, 2019 at 03:57:17PM -0400, Cleber Rosa wrote:
> On Sat, Aug 31, 2019 at 11:48:34AM +1000, David Gibson wrote:
> > On Fri, Aug 30, 2019 at 02:56:48PM -0300, Eduardo Habkost wrote:
> > > On Thu, Aug 29, 2019 at 11:31:25AM +1000, David Gibson wrote:
> > > > If I attempt to run "make check-acceptance" on my POWER9, RHEL8.1
> > > > machine when the openssl-devel package isn't installed, I get the
> > > > following very cryptic error:
> > > > 
> > > >   VENV    /home/dwg/qemu/build/rhel8/tests/venv
> > > >   PIP     /home/dwg/qemu/tests/requirements.txt
> > > > Command "/home/dwg/qemu/build/rhel8/tests/venv/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-la4el5r5/cryptography/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-1efs22iz-record/install-record.txt --single-version-externally-managed --compile --install-headers /home/dwg/qemu/build/rhel8/tests/venv/include/site/python3.6/cryptography" failed with error code 1 in /tmp/pip-build-la4el5r5/cryptography/
> > > > 
> > > > Using V=1 doesn't give any more useful information, and it's not
> > > > (easily) possible to manually re-run the given command since it relies
> > > > on things in /tmp that are removed once the attempt finishes.
> > > 
> > > V=1 is supposed to show the actual pip command being run.  I see
> > > it here:
> > > 
> > >   $ make check-venv V=1
> > >   [...]
> > >   python3 -B -m venv --system-site-packages /home/ehabkost/rh/proj/virt/qemu/tests/venv
> > > * /home/ehabkost/rh/proj/virt/qemu/tests/venv/bin/python -m pip -q install -r /home/ehabkost/rh/proj/virt/qemu/tests/requirements.txt
> > >   You are using pip version 19.0.3, however version 19.2.3 is available.
> > >   You should consider upgrading via the 'pip install --upgrade pip' command.
> > >   touch /home/ehabkost/rh/proj/virt/qemu/tests/venv
> > > 
> > > Can you reproduce the problem if you re-run the pip command?
> > 
> > Yes, I can, but it's basically just the same error without any extra
> > information.
> > 
> > > > I only figured out it was openssl-devel being missing that was the
> > > > problem by (mostly) guesswork.  It would be really great if we could
> > > > generate a more helpful error here.
> > > 
> > > I don't think there's much we can do about it, as the error is generated by a
> > > package being built by pip.
> > > 
> > > There's one thing we can do to make it slightly better: not passing `-q` to pip
> > > if running with V=1, so people can see which packages are being installed and
> > > which one is generating the error.
> > 
> > That sounds like a good start to me.
> > 
> > > > In addition, if I rerun "make check-acceptance" it no longer even
> > > > attempts the PIP install, since tests/venv already exists in my build
> > > > environment.  It then sort of works, but I think it might be hitting
> > > > other errors because of the missing python packages.  Sorry that's a
> > > > bit vague - I also seem to be getting unrelated errors that I'm still
> > > > trying to figure out.
> > > 
> > > This sounds like a bug and we need to fix it.
> > 
> > I agree.
> 
> Hi David,
> 
> I've pushed a branch here (most of the commits have already been sent
> separately):
> 
>    https://github.com/clebergnu/qemu/tree/ppc64
> 
> I've tested on a RHEL 8 ppc64le Power 9, and it seems to work for me.
> My steps for building QEMU:
> 
>   - Configured with: '/home/cleber/src/qemu/configure' '--enable-slirp=git' '--python=/usr/bin/python3' '--target-list=x86_64-softmmu,ppc64-softmmu,arm-softmmu,aarch64-softmmu,mips-softmmu,mipsel-softmmu,mips64-softmmu,mips64el-softmmu,sh4-softmmu,s390x-softmmu,alpha-softmmu,m68k-softmmu,riscv64-softmmu'
>   - make
>   - make check-acceptance
> 
> Would you be able to test if that branch works smoothly for you?

So, with this tree I'm no longer getting problems if openssl-devel is
not installed, so that much looks good.

I am getting some different errors - I was seeing this before (with
openssl-devel installed) sometimes, but only sometimes.  I haven't yet
worked out a pattern for when they appeared.  They also don't appear
to be fatal, the rest of the tests seem to be running ok.  Any ideas?

 VENV    /home/dwg/qemu/build/rhel8/tests/venv
  PIP     /home/dwg/qemu/tests/requirements.txt
  AVOCADO tests/acceptance
Error running method "run" of plugin "html": 'Namespace' object has no attribute 'get'
Error running method "run" of plugin "varianter_cit": 'Namespace' object has no attribute 'get'
JOB ID     : 6808bb316632af2bf7652586b3fbb22ac497e090
JOB LOG    : /home/dwg/qemu/build/rhel8/tests/results/job-2019-09-04T22.06-6808bb3/job.log
Error running method "initialize" of plugin "varianter_cit": 'Namespace' object has no attribute 'get'
Error running method "initialize" of plugin "yaml_to_mux": 'Namespace' object has no attribute 'get'
 (01/34) /home/dwg/qemu/tests/acceptance/boot_linux_console.py:BootLinuxConsole.test_x86_64_pc: PASS (4.45 s)

...

-- 
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [Qemu-devel] Cryptic errors from PIP install if missing openssl-devel

[Cleber Rosa]

On Thu, Sep 05, 2019 at 12:09:24PM +1000, David Gibson wrote:
> On Wed, Sep 04, 2019 at 03:57:17PM -0400, Cleber Rosa wrote:
> > 
> > Hi David,
> > 
> > I've pushed a branch here (most of the commits have already been sent
> > separately):
> > 
> >    https://github.com/clebergnu/qemu/tree/ppc64
> > 
> > I've tested on a RHEL 8 ppc64le Power 9, and it seems to work for me.
> > My steps for building QEMU:
> > 
> >   - Configured with: '/home/cleber/src/qemu/configure' '--enable-slirp=git' '--python=/usr/bin/python3' '--target-list=x86_64-softmmu,ppc64-softmmu,arm-softmmu,aarch64-softmmu,mips-softmmu,mipsel-softmmu,mips64-softmmu,mips64el-softmmu,sh4-softmmu,s390x-softmmu,alpha-softmmu,m68k-softmmu,riscv64-softmmu'
> >   - make
> >   - make check-acceptance
> > 
> > Would you be able to test if that branch works smoothly for you?
> 
> So, with this tree I'm no longer getting problems if openssl-devel is
> not installed, so that much looks good.
> 
> I am getting some different errors - I was seeing this before (with
> openssl-devel installed) sometimes, but only sometimes.  I haven't yet
> worked out a pattern for when they appeared.  They also don't appear
> to be fatal, the rest of the tests seem to be running ok.  Any ideas?
> 
>  VENV    /home/dwg/qemu/build/rhel8/tests/venv
>   PIP     /home/dwg/qemu/tests/requirements.txt
>   AVOCADO tests/acceptance
> Error running method "run" of plugin "html": 'Namespace' object has no attribute 'get'
> Error running method "run" of plugin "varianter_cit": 'Namespace' object has no attribute 'get'

Hi David,

Yes, those should not be fatal.  Anyway, they are caused by two things:

 1) A second set of Avocado plugins installation with different versions
    than the Avocado installation on QEMU's tests venv

 2) The tests' venv, which is created with "--system-site-packages",
    option that is described as "Give the virtual environment access
    to the system site-packages dir."

The motivation to have "--system-site-packages" is that, if the user
had any of the "requirements.txt" depedencies installed system wide,
the venv would simply use it and skip downloads.  Maybe we should make
that an optional feature, disabled by default, so that the test's venv
is better isolated and more predictable by default.

Eduardo,

I remember we discussed #2, so your opinion is appreciated here.

Thanks,
- Cleber.

Re: [Qemu-devel] Cryptic errors from PIP install if missing openssl-devel

[Eduardo Habkost]

On Thu, Sep 05, 2019 at 09:38:24AM -0400, Cleber Rosa wrote:
> On Thu, Sep 05, 2019 at 12:09:24PM +1000, David Gibson wrote:
> > On Wed, Sep 04, 2019 at 03:57:17PM -0400, Cleber Rosa wrote:
> > > 
> > > Hi David,
> > > 
> > > I've pushed a branch here (most of the commits have already been sent
> > > separately):
> > > 
> > >    https://github.com/clebergnu/qemu/tree/ppc64
> > > 
> > > I've tested on a RHEL 8 ppc64le Power 9, and it seems to work for me.
> > > My steps for building QEMU:
> > > 
> > >   - Configured with: '/home/cleber/src/qemu/configure' '--enable-slirp=git' '--python=/usr/bin/python3' '--target-list=x86_64-softmmu,ppc64-softmmu,arm-softmmu,aarch64-softmmu,mips-softmmu,mipsel-softmmu,mips64-softmmu,mips64el-softmmu,sh4-softmmu,s390x-softmmu,alpha-softmmu,m68k-softmmu,riscv64-softmmu'
> > >   - make
> > >   - make check-acceptance
> > > 
> > > Would you be able to test if that branch works smoothly for you?
> > 
> > So, with this tree I'm no longer getting problems if openssl-devel is
> > not installed, so that much looks good.
> > 
> > I am getting some different errors - I was seeing this before (with
> > openssl-devel installed) sometimes, but only sometimes.  I haven't yet
> > worked out a pattern for when they appeared.  They also don't appear
> > to be fatal, the rest of the tests seem to be running ok.  Any ideas?
> > 
> >  VENV    /home/dwg/qemu/build/rhel8/tests/venv
> >   PIP     /home/dwg/qemu/tests/requirements.txt
> >   AVOCADO tests/acceptance
> > Error running method "run" of plugin "html": 'Namespace' object has no attribute 'get'
> > Error running method "run" of plugin "varianter_cit": 'Namespace' object has no attribute 'get'
> 
> Hi David,
> 
> Yes, those should not be fatal.  Anyway, they are caused by two things:
> 
>  1) A second set of Avocado plugins installation with different versions
>     than the Avocado installation on QEMU's tests venv
> 
>  2) The tests' venv, which is created with "--system-site-packages",
>     option that is described as "Give the virtual environment access
>     to the system site-packages dir."
> 
> The motivation to have "--system-site-packages" is that, if the user
> had any of the "requirements.txt" depedencies installed system wide,
> the venv would simply use it and skip downloads.  Maybe we should make
> that an optional feature, disabled by default, so that the test's venv
> is better isolated and more predictable by default.
> 
> Eduardo,
> 
> I remember we discussed #2, so your opinion is appreciated here.

I think I was the person who insisted for --system-site-packages.
Considering that we're often using very recent versions of
Avocado, maybe trying to use system-provided Avocado modules by
default wasn't a good idea after all.

Making --system-site-packages optional and not enabling it by
default sounds like a reasonable workaround.

-- 
Eduardo

Re: [Qemu-devel] Cryptic errors from PIP install if missing openssl-devel

[Cleber Rosa]

On Thu, Sep 05, 2019 at 11:06:18AM -0300, Eduardo Habkost wrote:
> On Thu, Sep 05, 2019 at 09:38:24AM -0400, Cleber Rosa wrote:
> > On Thu, Sep 05, 2019 at 12:09:24PM +1000, David Gibson wrote:
> > > On Wed, Sep 04, 2019 at 03:57:17PM -0400, Cleber Rosa wrote:
> > > > 
> > > > Hi David,
> > > > 
> > > > I've pushed a branch here (most of the commits have already been sent
> > > > separately):
> > > > 
> > > >    https://github.com/clebergnu/qemu/tree/ppc64
> > > > 
> > > > I've tested on a RHEL 8 ppc64le Power 9, and it seems to work for me.
> > > > My steps for building QEMU:
> > > > 
> > > >   - Configured with: '/home/cleber/src/qemu/configure' '--enable-slirp=git' '--python=/usr/bin/python3' '--target-list=x86_64-softmmu,ppc64-softmmu,arm-softmmu,aarch64-softmmu,mips-softmmu,mipsel-softmmu,mips64-softmmu,mips64el-softmmu,sh4-softmmu,s390x-softmmu,alpha-softmmu,m68k-softmmu,riscv64-softmmu'
> > > >   - make
> > > >   - make check-acceptance
> > > > 
> > > > Would you be able to test if that branch works smoothly for you?
> > > 
> > > So, with this tree I'm no longer getting problems if openssl-devel is
> > > not installed, so that much looks good.
> > > 
> > > I am getting some different errors - I was seeing this before (with
> > > openssl-devel installed) sometimes, but only sometimes.  I haven't yet
> > > worked out a pattern for when they appeared.  They also don't appear
> > > to be fatal, the rest of the tests seem to be running ok.  Any ideas?
> > > 
> > >  VENV    /home/dwg/qemu/build/rhel8/tests/venv
> > >   PIP     /home/dwg/qemu/tests/requirements.txt
> > >   AVOCADO tests/acceptance
> > > Error running method "run" of plugin "html": 'Namespace' object has no attribute 'get'
> > > Error running method "run" of plugin "varianter_cit": 'Namespace' object has no attribute 'get'
> > 
> > Hi David,
> > 
> > Yes, those should not be fatal.  Anyway, they are caused by two things:
> > 
> >  1) A second set of Avocado plugins installation with different versions
> >     than the Avocado installation on QEMU's tests venv
> > 
> >  2) The tests' venv, which is created with "--system-site-packages",
> >     option that is described as "Give the virtual environment access
> >     to the system site-packages dir."
> > 
> > The motivation to have "--system-site-packages" is that, if the user
> > had any of the "requirements.txt" depedencies installed system wide,
> > the venv would simply use it and skip downloads.  Maybe we should make
> > that an optional feature, disabled by default, so that the test's venv
> > is better isolated and more predictable by default.
> > 
> > Eduardo,
> > 
> > I remember we discussed #2, so your opinion is appreciated here.
> 
> I think I was the person who insisted for --system-site-packages.
> Considering that we're often using very recent versions of
> Avocado, maybe trying to use system-provided Avocado modules by
> default wasn't a good idea after all.
> 
> Making --system-site-packages optional and not enabling it by
> default sounds like a reasonable workaround.
>

OK, agreed.

FIY, given the fact that pip itself keeps a local cache, not using
"--system-site-packages" this shouldn't be much of a hit when it
comes to downloads.

Cheers,
- Cleber.

> -- 
> Eduardo
> 

Re: [Qemu-devel] Cryptic errors from PIP install if missing openssl-devel

[David Gibson]

[-- Attachment #1: Type: text/plain, Size: 3134 bytes --]

On Thu, Sep 05, 2019 at 09:38:24AM -0400, Cleber Rosa wrote:
> On Thu, Sep 05, 2019 at 12:09:24PM +1000, David Gibson wrote:
> > On Wed, Sep 04, 2019 at 03:57:17PM -0400, Cleber Rosa wrote:
> > > 
> > > Hi David,
> > > 
> > > I've pushed a branch here (most of the commits have already been sent
> > > separately):
> > > 
> > >    https://github.com/clebergnu/qemu/tree/ppc64
> > > 
> > > I've tested on a RHEL 8 ppc64le Power 9, and it seems to work for me.
> > > My steps for building QEMU:
> > > 
> > >   - Configured with: '/home/cleber/src/qemu/configure' '--enable-slirp=git' '--python=/usr/bin/python3' '--target-list=x86_64-softmmu,ppc64-softmmu,arm-softmmu,aarch64-softmmu,mips-softmmu,mipsel-softmmu,mips64-softmmu,mips64el-softmmu,sh4-softmmu,s390x-softmmu,alpha-softmmu,m68k-softmmu,riscv64-softmmu'
> > >   - make
> > >   - make check-acceptance
> > > 
> > > Would you be able to test if that branch works smoothly for you?
> > 
> > So, with this tree I'm no longer getting problems if openssl-devel is
> > not installed, so that much looks good.
> > 
> > I am getting some different errors - I was seeing this before (with
> > openssl-devel installed) sometimes, but only sometimes.  I haven't yet
> > worked out a pattern for when they appeared.  They also don't appear
> > to be fatal, the rest of the tests seem to be running ok.  Any ideas?
> > 
> >  VENV    /home/dwg/qemu/build/rhel8/tests/venv
> >   PIP     /home/dwg/qemu/tests/requirements.txt
> >   AVOCADO tests/acceptance
> > Error running method "run" of plugin "html": 'Namespace' object has no attribute 'get'
> > Error running method "run" of plugin "varianter_cit": 'Namespace' object has no attribute 'get'
> 
> Hi David,
> 
> Yes, those should not be fatal.  Anyway, they are caused by two things:
> 
>  1) A second set of Avocado plugins installation with different versions
>     than the Avocado installation on QEMU's tests venv

Ah!  That'll be from the Avocado setup that Lukaš set up on there for
some different tests.

>  2) The tests' venv, which is created with "--system-site-packages",
>     option that is described as "Give the virtual environment access
>     to the system site-packages dir."
> 
> The motivation to have "--system-site-packages" is that, if the user
> had any of the "requirements.txt" depedencies installed system wide,
> the venv would simply use it and skip downloads.  Maybe we should make
> that an optional feature, disabled by default, so that the test's venv
> is better isolated and more predictable by default.

If pip doesn't have good enough version tracking that it's verifying
the correct versions of the installed packages as well as the
presence, then yes I tend to agree that keeping it better isolated
would be preferable.

> 
> Eduardo,
> 
> I remember we discussed #2, so your opinion is appreciated here.
> 
> Thanks,
> - Cleber.
> 

-- 
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [Qemu-devel] Cryptic errors from PIP install if missing openssl-devel

[Eduardo Habkost]

On Fri, Sep 06, 2019 at 10:34:16AM +1000, David Gibson wrote:
> On Thu, Sep 05, 2019 at 09:38:24AM -0400, Cleber Rosa wrote:
> > On Thu, Sep 05, 2019 at 12:09:24PM +1000, David Gibson wrote:
> > > On Wed, Sep 04, 2019 at 03:57:17PM -0400, Cleber Rosa wrote:
> > > > 
> > > > Hi David,
> > > > 
> > > > I've pushed a branch here (most of the commits have already been sent
> > > > separately):
> > > > 
> > > >    https://github.com/clebergnu/qemu/tree/ppc64
> > > > 
> > > > I've tested on a RHEL 8 ppc64le Power 9, and it seems to work for me.
> > > > My steps for building QEMU:
> > > > 
> > > >   - Configured with: '/home/cleber/src/qemu/configure' '--enable-slirp=git' '--python=/usr/bin/python3' '--target-list=x86_64-softmmu,ppc64-softmmu,arm-softmmu,aarch64-softmmu,mips-softmmu,mipsel-softmmu,mips64-softmmu,mips64el-softmmu,sh4-softmmu,s390x-softmmu,alpha-softmmu,m68k-softmmu,riscv64-softmmu'
> > > >   - make
> > > >   - make check-acceptance
> > > > 
> > > > Would you be able to test if that branch works smoothly for you?
> > > 
> > > So, with this tree I'm no longer getting problems if openssl-devel is
> > > not installed, so that much looks good.
> > > 
> > > I am getting some different errors - I was seeing this before (with
> > > openssl-devel installed) sometimes, but only sometimes.  I haven't yet
> > > worked out a pattern for when they appeared.  They also don't appear
> > > to be fatal, the rest of the tests seem to be running ok.  Any ideas?
> > > 
> > >  VENV    /home/dwg/qemu/build/rhel8/tests/venv
> > >   PIP     /home/dwg/qemu/tests/requirements.txt
> > >   AVOCADO tests/acceptance
> > > Error running method "run" of plugin "html": 'Namespace' object has no attribute 'get'
> > > Error running method "run" of plugin "varianter_cit": 'Namespace' object has no attribute 'get'
> > 
> > Hi David,
> > 
> > Yes, those should not be fatal.  Anyway, they are caused by two things:
> > 
> >  1) A second set of Avocado plugins installation with different versions
> >     than the Avocado installation on QEMU's tests venv
> 
> Ah!  That'll be from the Avocado setup that Lukaš set up on there for
> some different tests.
> 
> >  2) The tests' venv, which is created with "--system-site-packages",
> >     option that is described as "Give the virtual environment access
> >     to the system site-packages dir."
> > 
> > The motivation to have "--system-site-packages" is that, if the user
> > had any of the "requirements.txt" depedencies installed system wide,
> > the venv would simply use it and skip downloads.  Maybe we should make
> > that an optional feature, disabled by default, so that the test's venv
> > is better isolated and more predictable by default.
> 
> If pip doesn't have good enough version tracking that it's verifying
> the correct versions of the installed packages as well as the
> presence, then yes I tend to agree that keeping it better isolated
> would be preferable.

pip ensures the right version of packages we request are
installed, but Avocado plugins are separate packages and we don't
tell pip anything about the plugins we are not using (but could be
already installed in the system).

Cleber, does Avocado have any kind of plugin API versioning, so
it could detect incompatible plugins before using them?

-- 
Eduardo

Re: [Qemu-devel] Cryptic errors from PIP install if missing openssl-devel

[Cleber Rosa]

On Fri, Sep 06, 2019 at 11:43:58AM -0300, Eduardo Habkost wrote:
> On Fri, Sep 06, 2019 at 10:34:16AM +1000, David Gibson wrote:
> > On Thu, Sep 05, 2019 at 09:38:24AM -0400, Cleber Rosa wrote:
> > > On Thu, Sep 05, 2019 at 12:09:24PM +1000, David Gibson wrote:
> > > > On Wed, Sep 04, 2019 at 03:57:17PM -0400, Cleber Rosa wrote:
> > > > > 
> > > > > Hi David,
> > > > > 
> > > > > I've pushed a branch here (most of the commits have already been sent
> > > > > separately):
> > > > > 
> > > > >    https://github.com/clebergnu/qemu/tree/ppc64
> > > > > 
> > > > > I've tested on a RHEL 8 ppc64le Power 9, and it seems to work for me.
> > > > > My steps for building QEMU:
> > > > > 
> > > > >   - Configured with: '/home/cleber/src/qemu/configure' '--enable-slirp=git' '--python=/usr/bin/python3' '--target-list=x86_64-softmmu,ppc64-softmmu,arm-softmmu,aarch64-softmmu,mips-softmmu,mipsel-softmmu,mips64-softmmu,mips64el-softmmu,sh4-softmmu,s390x-softmmu,alpha-softmmu,m68k-softmmu,riscv64-softmmu'
> > > > >   - make
> > > > >   - make check-acceptance
> > > > > 
> > > > > Would you be able to test if that branch works smoothly for you?
> > > > 
> > > > So, with this tree I'm no longer getting problems if openssl-devel is
> > > > not installed, so that much looks good.
> > > > 
> > > > I am getting some different errors - I was seeing this before (with
> > > > openssl-devel installed) sometimes, but only sometimes.  I haven't yet
> > > > worked out a pattern for when they appeared.  They also don't appear
> > > > to be fatal, the rest of the tests seem to be running ok.  Any ideas?
> > > > 
> > > >  VENV    /home/dwg/qemu/build/rhel8/tests/venv
> > > >   PIP     /home/dwg/qemu/tests/requirements.txt
> > > >   AVOCADO tests/acceptance
> > > > Error running method "run" of plugin "html": 'Namespace' object has no attribute 'get'
> > > > Error running method "run" of plugin "varianter_cit": 'Namespace' object has no attribute 'get'
> > > 
> > > Hi David,
> > > 
> > > Yes, those should not be fatal.  Anyway, they are caused by two things:
> > > 
> > >  1) A second set of Avocado plugins installation with different versions
> > >     than the Avocado installation on QEMU's tests venv
> > 
> > Ah!  That'll be from the Avocado setup that Lukaš set up on there for
> > some different tests.
> > 
> > >  2) The tests' venv, which is created with "--system-site-packages",
> > >     option that is described as "Give the virtual environment access
> > >     to the system site-packages dir."
> > > 
> > > The motivation to have "--system-site-packages" is that, if the user
> > > had any of the "requirements.txt" depedencies installed system wide,
> > > the venv would simply use it and skip downloads.  Maybe we should make
> > > that an optional feature, disabled by default, so that the test's venv
> > > is better isolated and more predictable by default.
> > 
> > If pip doesn't have good enough version tracking that it's verifying
> > the correct versions of the installed packages as well as the
> > presence, then yes I tend to agree that keeping it better isolated
> > would be preferable.
> 
> pip ensures the right version of packages we request are
> installed, but Avocado plugins are separate packages and we don't
> tell pip anything about the plugins we are not using (but could be
> already installed in the system).

Right.

> 
> Cleber, does Avocado have any kind of plugin API versioning, so
> it could detect incompatible plugins before using them?
>

It does not.  And while making an exact version match requirement
between plugins and core Avocado would be easy, it would put unfair
(IMO) requirements on plugin writers.  It's quite possible that
an external plugin would work for many different avocado versions,
and the author wouldn't feel like updating it just to satisfy the
version check.

I'm not quite sure we should be trying to solve this problem here, but
I'm open to suggestions.

- Cleber.

> -- 
> Eduardo

Re: [Qemu-devel] Cryptic errors from PIP install if missing openssl-devel

[Eduardo Habkost]

On Fri, Sep 06, 2019 at 10:52:02AM -0400, Cleber Rosa wrote:
> On Fri, Sep 06, 2019 at 11:43:58AM -0300, Eduardo Habkost wrote:
> > On Fri, Sep 06, 2019 at 10:34:16AM +1000, David Gibson wrote:
> > > On Thu, Sep 05, 2019 at 09:38:24AM -0400, Cleber Rosa wrote:
> > > > On Thu, Sep 05, 2019 at 12:09:24PM +1000, David Gibson wrote:
> > > > > On Wed, Sep 04, 2019 at 03:57:17PM -0400, Cleber Rosa wrote:
> > > > > > 
> > > > > > Hi David,
> > > > > > 
> > > > > > I've pushed a branch here (most of the commits have already been sent
> > > > > > separately):
> > > > > > 
> > > > > >    https://github.com/clebergnu/qemu/tree/ppc64
> > > > > > 
> > > > > > I've tested on a RHEL 8 ppc64le Power 9, and it seems to work for me.
> > > > > > My steps for building QEMU:
> > > > > > 
> > > > > >   - Configured with: '/home/cleber/src/qemu/configure' '--enable-slirp=git' '--python=/usr/bin/python3' '--target-list=x86_64-softmmu,ppc64-softmmu,arm-softmmu,aarch64-softmmu,mips-softmmu,mipsel-softmmu,mips64-softmmu,mips64el-softmmu,sh4-softmmu,s390x-softmmu,alpha-softmmu,m68k-softmmu,riscv64-softmmu'
> > > > > >   - make
> > > > > >   - make check-acceptance
> > > > > > 
> > > > > > Would you be able to test if that branch works smoothly for you?
> > > > > 
> > > > > So, with this tree I'm no longer getting problems if openssl-devel is
> > > > > not installed, so that much looks good.
> > > > > 
> > > > > I am getting some different errors - I was seeing this before (with
> > > > > openssl-devel installed) sometimes, but only sometimes.  I haven't yet
> > > > > worked out a pattern for when they appeared.  They also don't appear
> > > > > to be fatal, the rest of the tests seem to be running ok.  Any ideas?
> > > > > 
> > > > >  VENV    /home/dwg/qemu/build/rhel8/tests/venv
> > > > >   PIP     /home/dwg/qemu/tests/requirements.txt
> > > > >   AVOCADO tests/acceptance
> > > > > Error running method "run" of plugin "html": 'Namespace' object has no attribute 'get'
> > > > > Error running method "run" of plugin "varianter_cit": 'Namespace' object has no attribute 'get'
> > > > 
> > > > Hi David,
> > > > 
> > > > Yes, those should not be fatal.  Anyway, they are caused by two things:
> > > > 
> > > >  1) A second set of Avocado plugins installation with different versions
> > > >     than the Avocado installation on QEMU's tests venv
> > > 
> > > Ah!  That'll be from the Avocado setup that Lukaš set up on there for
> > > some different tests.
> > > 
> > > >  2) The tests' venv, which is created with "--system-site-packages",
> > > >     option that is described as "Give the virtual environment access
> > > >     to the system site-packages dir."
> > > > 
> > > > The motivation to have "--system-site-packages" is that, if the user
> > > > had any of the "requirements.txt" depedencies installed system wide,
> > > > the venv would simply use it and skip downloads.  Maybe we should make
> > > > that an optional feature, disabled by default, so that the test's venv
> > > > is better isolated and more predictable by default.
> > > 
> > > If pip doesn't have good enough version tracking that it's verifying
> > > the correct versions of the installed packages as well as the
> > > presence, then yes I tend to agree that keeping it better isolated
> > > would be preferable.
> > 
> > pip ensures the right version of packages we request are
> > installed, but Avocado plugins are separate packages and we don't
> > tell pip anything about the plugins we are not using (but could be
> > already installed in the system).
> 
> Right.
> 
> > 
> > Cleber, does Avocado have any kind of plugin API versioning, so
> > it could detect incompatible plugins before using them?
> >
> 
> It does not.  And while making an exact version match requirement
> between plugins and core Avocado would be easy, it would put unfair
> (IMO) requirements on plugin writers.  It's quite possible that
> an external plugin would work for many different avocado versions,
> and the author wouldn't feel like updating it just to satisfy the
> version check.

I agree that tying a plugin to a specific Avocado version seems
too strict.  Are there any rules about API stability that plugin
writers can expect Avocado to follow?

> 
> I'm not quite sure we should be trying to solve this problem here, but
> I'm open to suggestions.

This will probably become a minor problem once we stop using
--system-site-packages by default.

-- 
Eduardo

Re: [Qemu-devel] Cryptic errors from PIP install if missing openssl-devel

[Cleber Rosa]

On Fri, Sep 06, 2019 at 12:14:15PM -0300, Eduardo Habkost wrote:
> On Fri, Sep 06, 2019 at 10:52:02AM -0400, Cleber Rosa wrote:
> > On Fri, Sep 06, 2019 at 11:43:58AM -0300, Eduardo Habkost wrote:
> > > On Fri, Sep 06, 2019 at 10:34:16AM +1000, David Gibson wrote:
> > > > On Thu, Sep 05, 2019 at 09:38:24AM -0400, Cleber Rosa wrote:
> > > > > On Thu, Sep 05, 2019 at 12:09:24PM +1000, David Gibson wrote:
> > > > > > On Wed, Sep 04, 2019 at 03:57:17PM -0400, Cleber Rosa wrote:
> > > > > > > 
> > > > > > > Hi David,
> > > > > > > 
> > > > > > > I've pushed a branch here (most of the commits have already been sent
> > > > > > > separately):
> > > > > > > 
> > > > > > >    https://github.com/clebergnu/qemu/tree/ppc64
> > > > > > > 
> > > > > > > I've tested on a RHEL 8 ppc64le Power 9, and it seems to work for me.
> > > > > > > My steps for building QEMU:
> > > > > > > 
> > > > > > >   - Configured with: '/home/cleber/src/qemu/configure' '--enable-slirp=git' '--python=/usr/bin/python3' '--target-list=x86_64-softmmu,ppc64-softmmu,arm-softmmu,aarch64-softmmu,mips-softmmu,mipsel-softmmu,mips64-softmmu,mips64el-softmmu,sh4-softmmu,s390x-softmmu,alpha-softmmu,m68k-softmmu,riscv64-softmmu'
> > > > > > >   - make
> > > > > > >   - make check-acceptance
> > > > > > > 
> > > > > > > Would you be able to test if that branch works smoothly for you?
> > > > > > 
> > > > > > So, with this tree I'm no longer getting problems if openssl-devel is
> > > > > > not installed, so that much looks good.
> > > > > > 
> > > > > > I am getting some different errors - I was seeing this before (with
> > > > > > openssl-devel installed) sometimes, but only sometimes.  I haven't yet
> > > > > > worked out a pattern for when they appeared.  They also don't appear
> > > > > > to be fatal, the rest of the tests seem to be running ok.  Any ideas?
> > > > > > 
> > > > > >  VENV    /home/dwg/qemu/build/rhel8/tests/venv
> > > > > >   PIP     /home/dwg/qemu/tests/requirements.txt
> > > > > >   AVOCADO tests/acceptance
> > > > > > Error running method "run" of plugin "html": 'Namespace' object has no attribute 'get'
> > > > > > Error running method "run" of plugin "varianter_cit": 'Namespace' object has no attribute 'get'
> > > > > 
> > > > > Hi David,
> > > > > 
> > > > > Yes, those should not be fatal.  Anyway, they are caused by two things:
> > > > > 
> > > > >  1) A second set of Avocado plugins installation with different versions
> > > > >     than the Avocado installation on QEMU's tests venv
> > > > 
> > > > Ah!  That'll be from the Avocado setup that Lukaš set up on there for
> > > > some different tests.
> > > > 
> > > > >  2) The tests' venv, which is created with "--system-site-packages",
> > > > >     option that is described as "Give the virtual environment access
> > > > >     to the system site-packages dir."
> > > > > 
> > > > > The motivation to have "--system-site-packages" is that, if the user
> > > > > had any of the "requirements.txt" depedencies installed system wide,
> > > > > the venv would simply use it and skip downloads.  Maybe we should make
> > > > > that an optional feature, disabled by default, so that the test's venv
> > > > > is better isolated and more predictable by default.
> > > > 
> > > > If pip doesn't have good enough version tracking that it's verifying
> > > > the correct versions of the installed packages as well as the
> > > > presence, then yes I tend to agree that keeping it better isolated
> > > > would be preferable.
> > > 
> > > pip ensures the right version of packages we request are
> > > installed, but Avocado plugins are separate packages and we don't
> > > tell pip anything about the plugins we are not using (but could be
> > > already installed in the system).
> > 
> > Right.
> > 
> > > 
> > > Cleber, does Avocado have any kind of plugin API versioning, so
> > > it could detect incompatible plugins before using them?
> > >
> > 
> > It does not.  And while making an exact version match requirement
> > between plugins and core Avocado would be easy, it would put unfair
> > (IMO) requirements on plugin writers.  It's quite possible that
> > an external plugin would work for many different avocado versions,
> > and the author wouldn't feel like updating it just to satisfy the
> > version check.
> 
> I agree that tying a plugin to a specific Avocado version seems
> too strict.  Are there any rules about API stability that plugin
> writers can expect Avocado to follow?
>

TBH the plugin interfaces[1] are quite stable (have changed very
little), but:

 1) They're still part of the "avocado.core" namespace, which means
    the stability is a best effort, and not a promise/guarantee

 2) Most plugins make use of other "avocado.core" modules, which
    follow the same logic as #1

It's a known fact that it'd be good to have a supportable plugin
API[2], but we haven't had the resources to implement that.

> > 
> > I'm not quite sure we should be trying to solve this problem here, but
> > I'm open to suggestions.
> 
> This will probably become a minor problem once we stop using
> --system-site-packages by default.
>

Exactly, I think at this moment, this is the simple and best course of
action.

> -- 
> Eduardo


Thanks,
- Cleber.

[1] - https://github.com/avocado-framework/avocado/blob/master/avocado/core/plugin_interfaces.py
[2] - https://trello.com/c/rKIfx7rY/1285-create-a-sufficient-and-supportable-plugin-api