iotests and python dependencies

iotests and python dependencies

[John Snow]

Howdy!

So, I want to finally delete python/qemu/qmp from qemu.git, and this
creates a small problem -- namely, iotests needs access to it in order
to run the python-based tests.

What I think needs to happen is that we create a virtual environment
that installs python/qemu/. The reason this cannot be done with
PYTHONPATH alone anymore is because the qmp package itself won't be
there anymore, we need an installer like `pip` to actually fetch it
for us and put it somewhere. (i.e., we need to process the
dependencies of python/qemu now and can't treat it as a pre-installed
location.)

Avocado tests are already creating a venv for the purposes of
installing and running Avocado. We can amend e.g. "../../python" to
tests/requirements.txt and the Avocado environment is A-OK good-to-go.
The Makefile magic for avocado tests creates a venv-per-build. It
seems to work well enough. One thing to note here is that the
supported invocation for avocado tests is only through the Makefile,
which handles creating and entering the venv to make the command
seamless.

iotests, however, manages its own execution environment with
testenv.py, and we support running iotests from outside of the
Makefile, for example by going to $build/tests/qemu-iotests and
running ./check.

Now ... I could update testenv.py to be smart enough to create and
enter a python venv, and have even prototyped this. It seems to work
pretty well! This approach seemed like the least invasive to how
iotests are expected to be run and used. But a downside with this
approach is that now avocado tests and iotests are each managing their
own python venv. Worse, vm-tests and device-crash-test are still
unhandled entirely.

I'd like to find a solution where I create a unified python testing
venv tied to the build shared by avocado, iotests, vm-tests and
device-crash-test. I'm not completely sure how exactly I'll manage
that right now, but I wanted to throw this out there in case there are
some requirements I might be overlooking.

I think vm-tests and avocado-tests can both have a venv created for
them and activated before the test runs. device-crash-test I believe
will need a script change in the gitlab ci yaml. iotests is somewhat
unique in that it needs to run both by manual invocation and from
makefile invocations. If I want a shared VM between all of these, I'll
need to isolate the create-and-enter-venv logic somewhere where it can
be shared both inside and outside of a Makefile.

I'll see what I can cook up, but if you have any concerns or Cool
Ideas, lemme know. I want to make sure this is as painless as I can
think to make it.

Thanks,
--js

Re: iotests and python dependencies

[Daniel P. Berrangé]

On Wed, May 04, 2022 at 03:38:45PM -0400, John Snow wrote:
> Howdy!
> 
> So, I want to finally delete python/qemu/qmp from qemu.git, and this
> creates a small problem -- namely, iotests needs access to it in order
> to run the python-based tests.
> 
> What I think needs to happen is that we create a virtual environment
> that installs python/qemu/. The reason this cannot be done with
> PYTHONPATH alone anymore is because the qmp package itself won't be
> there anymore, we need an installer like `pip` to actually fetch it
> for us and put it somewhere. (i.e., we need to process the
> dependencies of python/qemu now and can't treat it as a pre-installed
> location.)

Having pip fetch it on the fly creates a problem for RPM builds,
because the koji build env has no network access. We will, however,
have an RPM of python-qemu-qmp installed on the host system though.
IOW we need to be able to run iotests using system python and its
installed libs, not a virtual env.  So if we do anything with a
virtual env, it will need to be optional I believe.


With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: iotests and python dependencies

[Kevin Wolf]

Am 04.05.2022 um 21:38 hat John Snow geschrieben:
> Howdy!
> 
> So, I want to finally delete python/qemu/qmp from qemu.git, and this
> creates a small problem -- namely, iotests needs access to it in order
> to run the python-based tests.
> 
> What I think needs to happen is that we create a virtual environment
> that installs python/qemu/. The reason this cannot be done with
> PYTHONPATH alone anymore is because the qmp package itself won't be
> there anymore, we need an installer like `pip` to actually fetch it
> for us and put it somewhere. (i.e., we need to process the
> dependencies of python/qemu now and can't treat it as a pre-installed
> location.)
> 
> Avocado tests are already creating a venv for the purposes of
> installing and running Avocado. We can amend e.g. "../../python" to
> tests/requirements.txt and the Avocado environment is A-OK good-to-go.
> The Makefile magic for avocado tests creates a venv-per-build.

"per build" sounded pretty bad, because I thought it meant building a
new venv every time I run either 'make' or the tests. This would
obviously be very noticable for short-running tests like some iotests.
But fortunately this is not what it does, it keeps the venv around in
the build directory. So it's only per build directory really, which I
guess is fine.

> It seems to work well enough. One thing to note here is that the
> supported invocation for avocado tests is only through the Makefile,
> which handles creating and entering the venv to make the command
> seamless.
> 
> iotests, however, manages its own execution environment with
> testenv.py, and we support running iotests from outside of the
> Makefile, for example by going to $build/tests/qemu-iotests and
> running ./check.

Yes, and I agree that this is important.

> Now ... I could update testenv.py to be smart enough to create and
> enter a python venv, and have even prototyped this. It seems to work
> pretty well! This approach seemed like the least invasive to how
> iotests are expected to be run and used. But a downside with this
> approach is that now avocado tests and iotests are each managing their
> own python venv. Worse, vm-tests and device-crash-test are still
> unhandled entirely.

Is there a reason why testenv.py couldn't enter just the shared venv in
build/tests/venv?

If not, I guess it would be enough if iotests just checks that the venv
exists and all of the dependencies are there in the right version and
error out if not, telling the user to run 'make check-venv'.

Or actually, it could just unconditionally run 'make check-venv' by
itself, which is probably easier to implement than checking the
dependencies and more convenient for the user, too.

(One more detail: 'make check-venv GIT_SUBMODULES_ACTION=ignore' seems
to make it pretty much instantaneous if the venv is current, but leaving
the submodule mechanism enabled adds a noticable delay.)

> I'd like to find a solution where I create a unified python testing
> venv tied to the build shared by avocado, iotests, vm-tests and
> device-crash-test. I'm not completely sure how exactly I'll manage
> that right now, but I wanted to throw this out there in case there are
> some requirements I might be overlooking.
> 
> I think vm-tests and avocado-tests can both have a venv created for
> them and activated before the test runs. device-crash-test I believe
> will need a script change in the gitlab ci yaml. iotests is somewhat
> unique in that it needs to run both by manual invocation and from
> makefile invocations. If I want a shared VM between all of these, I'll
> need to isolate the create-and-enter-venv logic somewhere where it can
> be shared both inside and outside of a Makefile.

If just calling 'make' isn't an option, then moving that part out into a
separate script probably wouldn't be too hard either. But 'make' has the
advantage that it already contains the check if the venv is already
there and requirements.txt hasn't changed, which you would have to
replicate otherwise.

Kevin

Re: iotests and python dependencies

[Paolo Bonzini]

On 5/5/22 10:51, Kevin Wolf wrote:
> If not, I guess it would be enough if iotests just checks that the venv
> exists and all of the dependencies are there in the right version and
> error out if not, telling the user to run 'make check-venv'.
> 
> Or actually, it could just unconditionally run 'make check-venv' by
> itself, which is probably easier to implement than checking the
> dependencies and more convenient for the user, too.

Note that you would still have to add a 'check-block: check-venv' 
dependency in the Makefile, otherwise two "instances" of check-venv 
could run in parallel.

One small complication is that on BSD systems the binary is actually 
called "gmake", so you'd have to pass the variable somehow

Paolo

Re: iotests and python dependencies

[Kevin Wolf]

Am 05.05.2022 um 11:28 hat Paolo Bonzini geschrieben:
> On 5/5/22 10:51, Kevin Wolf wrote:
> > If not, I guess it would be enough if iotests just checks that the venv
> > exists and all of the dependencies are there in the right version and
> > error out if not, telling the user to run 'make check-venv'.
> > 
> > Or actually, it could just unconditionally run 'make check-venv' by
> > itself, which is probably easier to implement than checking the
> > dependencies and more convenient for the user, too.
> 
> Note that you would still have to add a 'check-block: check-venv'
> dependency in the Makefile, otherwise two "instances" of check-venv
> could run in parallel.

Good point. I only considered manual invocations, but for 'make
check-block' that sounds right.

> One small complication is that on BSD systems the binary is actually
> called "gmake", so you'd have to pass the variable somehow

I guess we could just export $MAKE as an environment variable?

Kevin

Re: iotests and python dependencies

[John Snow]

[-- Attachment #1: Type: text/plain, Size: 5289 bytes --]

On Thu, May 5, 2022, 4:51 AM Kevin Wolf <kwolf@redhat.com> wrote:

> Am 04.05.2022 um 21:38 hat John Snow geschrieben:
> > Howdy!
> >
> > So, I want to finally delete python/qemu/qmp from qemu.git, and this
> > creates a small problem -- namely, iotests needs access to it in order
> > to run the python-based tests.
> >
> > What I think needs to happen is that we create a virtual environment
> > that installs python/qemu/. The reason this cannot be done with
> > PYTHONPATH alone anymore is because the qmp package itself won't be
> > there anymore, we need an installer like `pip` to actually fetch it
> > for us and put it somewhere. (i.e., we need to process the
> > dependencies of python/qemu now and can't treat it as a pre-installed
> > location.)
> >
> > Avocado tests are already creating a venv for the purposes of
> > installing and running Avocado. We can amend e.g. "../../python" to
> > tests/requirements.txt and the Avocado environment is A-OK good-to-go.
> > The Makefile magic for avocado tests creates a venv-per-build.
>
> "per build" sounded pretty bad, because I thought it meant building a
> new venv every time I run either 'make' or the tests. This would
> obviously be very noticable for short-running tests like some iotests.
> But fortunately this is not what it does, it keeps the venv around in
> the build directory. So it's only per build directory really, which I
> guess is fine.
>

Whoops, yeah. I meant per build directory. In contrast to the Python unit
tests themselves, which create some test venvs tied directly to the source
directory and are build-agnostic.


> > It seems to work well enough. One thing to note here is that the
> > supported invocation for avocado tests is only through the Makefile,
> > which handles creating and entering the venv to make the command
> > seamless.
> >
> > iotests, however, manages its own execution environment with
> > testenv.py, and we support running iotests from outside of the
> > Makefile, for example by going to $build/tests/qemu-iotests and
> > running ./check.
>
> Yes, and I agree that this is important.
>

Figured as much. Not plotting to take this away, I promise. Just getting my
requirements straight before I spend time concocting something.


> > Now ... I could update testenv.py to be smart enough to create and
> > enter a python venv, and have even prototyped this. It seems to work
> > pretty well! This approach seemed like the least invasive to how
> > iotests are expected to be run and used. But a downside with this
> > approach is that now avocado tests and iotests are each managing their
> > own python venv. Worse, vm-tests and device-crash-test are still
> > unhandled entirely.
>
> Is there a reason why testenv.py couldn't enter just the shared venv in
> build/tests/venv?
>

It can, but it'd have to be made first so it can enter it. I figured this
would only happen "on-demand", like how check-avocado works, so I'd need
some way for iotests and the Makefile to share the venv creation code,
which is certainly an option.


> If not, I guess it would be enough if iotests just checks that the venv
> exists and all of the dependencies are there in the right version and
> error out if not, telling the user to run 'make check-venv'.
>

I will spend unreasonable engineering hours avoiding this, if only for
pride. I want everything to be as seamless and easy as it's always been.


> Or actually, it could just unconditionally run 'make check-venv' by
> itself, which is probably easier to implement than checking the
> dependencies and more convenient for the user, too.
>

Oh, that's one way to get them to share the venv-creation pathway. Hadn't
occurred to me, but this might be easy to do.


> (One more detail: 'make check-venv GIT_SUBMODULES_ACTION=ignore' seems
> to make it pretty much instantaneous if the venv is current, but leaving
> the submodule mechanism enabled adds a noticable delay.)
>

Noted.


> > I'd like to find a solution where I create a unified python testing
> > venv tied to the build shared by avocado, iotests, vm-tests and
> > device-crash-test. I'm not completely sure how exactly I'll manage
> > that right now, but I wanted to throw this out there in case there are
> > some requirements I might be overlooking.
> >
> > I think vm-tests and avocado-tests can both have a venv created for
> > them and activated before the test runs. device-crash-test I believe
> > will need a script change in the gitlab ci yaml. iotests is somewhat
> > unique in that it needs to run both by manual invocation and from
> > makefile invocations. If I want a shared VM between all of these, I'll
> > need to isolate the create-and-enter-venv logic somewhere where it can
> > be shared both inside and outside of a Makefile.
>
> If just calling 'make' isn't an option, then moving that part out into a
> separate script probably wouldn't be too hard either. But 'make' has the
> advantage that it already contains the check if the venv is already
> there and requirements.txt hasn't changed, which you would have to
> replicate otherwise.
>

Yeah, exactly.


> Kevin
>

Thanks for the ideas!

As of now, I've got a branch where I've moved vm-tests over onto the same
venv with avocado-tests. If I can bridge the gap to iotests, I'll be pretty
happy.

[-- Attachment #2: Type: text/html, Size: 8102 bytes --]

Re: iotests and python dependencies

[John Snow]

[-- Attachment #1: Type: text/plain, Size: 1828 bytes --]

On Thu, May 5, 2022, 4:09 AM Daniel P. Berrangé <berrange@redhat.com> wrote:

> On Wed, May 04, 2022 at 03:38:45PM -0400, John Snow wrote:
> > Howdy!
> >
> > So, I want to finally delete python/qemu/qmp from qemu.git, and this
> > creates a small problem -- namely, iotests needs access to it in order
> > to run the python-based tests.
> >
> > What I think needs to happen is that we create a virtual environment
> > that installs python/qemu/. The reason this cannot be done with
> > PYTHONPATH alone anymore is because the qmp package itself won't be
> > there anymore, we need an installer like `pip` to actually fetch it
> > for us and put it somewhere. (i.e., we need to process the
> > dependencies of python/qemu now and can't treat it as a pre-installed
> > location.)
>
> Having pip fetch it on the fly creates a problem for RPM builds,
> because the koji build env has no network access. We will, however,
> have an RPM of python-qemu-qmp installed on the host system though.
> IOW we need to be able to run iotests using system python and its
> installed libs, not a virtual env.  So if we do anything with a
> virtual env, it will need to be optional I believe.
>

Hm, do we need iotests during an rpm build? Is it because of "make check"?

It's possible to create a venv and run pip in no-network mode, too. If the
package we want is installed on the system or otherwise in pip's cache,
it'll succeed without network. If the dependencies require a qemu.qmp
that's too new, the pip install will just fail instead.

I have to test a way to craft a pip statement that's network *optional*
though. i.e. try to fetch and fall back to local otherwise. I think it's
worth trying to keep the environment setup code unified, and always use a
venv.

I think this is tractable, though.

[-- Attachment #2: Type: text/html, Size: 2422 bytes --]

Re: iotests and python dependencies

[Paolo Bonzini]

On 5/5/22 12:59, Kevin Wolf wrote:
> Am 05.05.2022 um 11:28 hat Paolo Bonzini geschrieben:
>>> Or actually, it could just unconditionally run 'make check-venv' by
>>> itself, which is probably easier to implement than checking the
>>> dependencies and more convenient for the user, too.
>>
>> One small complication is that on BSD systems the binary is actually
>> called "gmake", so you'd have to pass the variable somehow
> 
> I guess we could just export $MAKE as an environment variable?

That would work when invoked by "make", but then that's the case in 
which the venv would be there anyway.

For the other case, it would have to parse config-host.mak and/or 
reintroduce something like tests/qemu-iotests/common.env.  All in all it 
seems like an unnecessary complication over just printing a clear and 
polite error message.

Paolo

Re: iotests and python dependencies

[Daniel P. Berrangé]

On Thu, May 05, 2022 at 08:08:42AM -0400, John Snow wrote:
> On Thu, May 5, 2022, 4:09 AM Daniel P. Berrangé <berrange@redhat.com> wrote:
> 
> > On Wed, May 04, 2022 at 03:38:45PM -0400, John Snow wrote:
> > > Howdy!
> > >
> > > So, I want to finally delete python/qemu/qmp from qemu.git, and this
> > > creates a small problem -- namely, iotests needs access to it in order
> > > to run the python-based tests.
> > >
> > > What I think needs to happen is that we create a virtual environment
> > > that installs python/qemu/. The reason this cannot be done with
> > > PYTHONPATH alone anymore is because the qmp package itself won't be
> > > there anymore, we need an installer like `pip` to actually fetch it
> > > for us and put it somewhere. (i.e., we need to process the
> > > dependencies of python/qemu now and can't treat it as a pre-installed
> > > location.)
> >
> > Having pip fetch it on the fly creates a problem for RPM builds,
> > because the koji build env has no network access. We will, however,
> > have an RPM of python-qemu-qmp installed on the host system though.
> > IOW we need to be able to run iotests using system python and its
> > installed libs, not a virtual env.  So if we do anything with a
> > virtual env, it will need to be optional I believe.
> >
> 
> Hm, do we need iotests during an rpm build? Is it because of "make check"?

Yes, and this is good, because it prevents us from outputting an
RPM build that has a broken QEMU in it. 

> It's possible to create a venv and run pip in no-network mode, too. If the
> package we want is installed on the system or otherwise in pip's cache,
> it'll succeed without network. If the dependencies require a qemu.qmp
> that's too new, the pip install will just fail instead.
> 
> I have to test a way to craft a pip statement that's network *optional*
> though. i.e. try to fetch and fall back to local otherwise. I think it's
> worth trying to keep the environment setup code unified, and always use a
> venv.

As long as it is no-network, that's good enough.

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: iotests and python dependencies

[Kevin Wolf]

Am 05.05.2022 um 14:24 hat Paolo Bonzini geschrieben:
> On 5/5/22 12:59, Kevin Wolf wrote:
> > Am 05.05.2022 um 11:28 hat Paolo Bonzini geschrieben:
> > > > Or actually, it could just unconditionally run 'make check-venv' by
> > > > itself, which is probably easier to implement than checking the
> > > > dependencies and more convenient for the user, too.
> > > 
> > > One small complication is that on BSD systems the binary is actually
> > > called "gmake", so you'd have to pass the variable somehow
> > 
> > I guess we could just export $MAKE as an environment variable?
> 
> That would work when invoked by "make", but then that's the case in which
> the venv would be there anyway.
> 
> For the other case, it would have to parse config-host.mak and/or
> reintroduce something like tests/qemu-iotests/common.env.  All in all it
> seems like an unnecessary complication over just printing a clear and polite
> error message.

Or try 'make' and print the error message only if that fails.

Kevin

Re: iotests and python dependencies

[John Snow]

[-- Attachment #1: Type: text/plain, Size: 1755 bytes --]

On Thu, May 5, 2022, 8:33 AM Daniel P. Berrangé <berrange@redhat.com> wrote:

> On Thu, May 05, 2022 at 08:08:42AM -0400, John Snow wrote:
> > On Thu, May 5, 2022, 4:09 AM Daniel P. Berrangé <berrange@redhat.com>
> wrote:
> >
> > > On Wed, May 04, 2022 at 03:38:45PM -0400, John Snow wrote:
> > > > Howdy!
> > > >
> > > > So, I want to finally delete python/qemu/qmp from qemu.git, and this
> > > > creates a small problem -- namely, iotests needs access to it in
> order
> > > > to run the python-based tests.
> > > >
> > > > What I think needs to happen is that we create a virtual environment
> > > > that installs python/qemu/. The reason this cannot be done with
> > > > PYTHONPATH alone anymore is because the qmp package itself won't be
> > > > there anymore, we need an installer like `pip` to actually fetch it
> > > > for us and put it somewhere. (i.e., we need to process the
> > > > dependencies of python/qemu now and can't treat it as a pre-installed
> > > > location.)
> > >
> > > Having pip fetch it on the fly creates a problem for RPM builds,
> > > because the koji build env has no network access. We will, however,
> > > have an RPM of python-qemu-qmp installed on the host system though.
> > > IOW we need to be able to run iotests using system python and its
> > > installed libs, not a virtual env.  So if we do anything with a
> > > virtual env, it will need to be optional I believe.
> > >
> >
> > Hm, do we need iotests during an rpm build? Is it because of "make
> check"?
>
> Yes, and this is good, because it prevents us from outputting an
> RPM build that has a broken QEMU in it.


Guess this means I need to make a Fedora package too, though. My hubris.

OK, plenty of work to do.

[-- Attachment #2: Type: text/html, Size: 2491 bytes --]

Re: iotests and python dependencies

[Paolo Bonzini]

On 5/5/22 15:10, John Snow wrote:
> 
>      > Hm, do we need iotests during an rpm build? Is it because of
>     "make check"?
> 
>     Yes, and this is good, because it prevents us from outputting an
>     RPM build that has a broken QEMU in it.
> 
> Guess this means I need to make a Fedora package too, though. My hubris.

I would rather keep python/qemu/qmp as a submodule for a longer time, 
and still go through a virtual environment that installs it together 
with its pip dependencies.

Paolo

Re: iotests and python dependencies

[John Snow]

[-- Attachment #1: Type: text/plain, Size: 972 bytes --]

On Thu, May 5, 2022, 9:16 AM Paolo Bonzini <pbonzini@redhat.com> wrote:

> On 5/5/22 15:10, John Snow wrote:
> >
> >      > Hm, do we need iotests during an rpm build? Is it because of
> >     "make check"?
> >
> >     Yes, and this is good, because it prevents us from outputting an
> >     RPM build that has a broken QEMU in it.
> >
> > Guess this means I need to make a Fedora package too, though. My hubris.
>
> I would rather keep python/qemu/qmp as a submodule for a longer time,
> and still go through a virtual environment that installs it together
> with its pip dependencies.
>

A small headache relating fixes to both locations, but if you'd like to see
it to prove that the installation mechanism works in general, then OK. I'm
willing to deal with the pain until the next release to let us go through a
testing cycle. Reluctantly. Maybe.

I'm assuming you mean as a subpackage and not a [git] submodule. If you do
mean git, then ... uh. That might be messy.

[-- Attachment #2: Type: text/html, Size: 1640 bytes --]

Re: iotests and python dependencies

[Paolo Bonzini]

On 5/5/22 16:13, John Snow wrote:
> 
>     I would rather keep python/qemu/qmp as a submodule for a longer time,
>     and still go through a virtual environment that installs it together
>     with its pip dependencies.
> 
> 
> A small headache relating fixes to both locations, but if you'd like to 
> see it to prove that the installation mechanism works in general, then 
> OK. I'm willing to deal with the pain until the next release to let us 
> go through a testing cycle. Reluctantly. Maybe.
> 
> I'm assuming you mean as a subpackage and not a [git] submodule. If you 
> do mean git, then ... uh. That might be messy.

Yeah, I meant a git submodule in qemu.git...  It would also be the 
easiest way to build a subpackage in Fedora, since it would be part of 
the QEMU tarballs.

Paolo

Re: iotests and python dependencies

[Daniel P. Berrangé]

On Thu, May 05, 2022 at 05:50:00PM +0200, Paolo Bonzini wrote:
> On 5/5/22 16:13, John Snow wrote:
> > 
> >     I would rather keep python/qemu/qmp as a submodule for a longer time,
> >     and still go through a virtual environment that installs it together
> >     with its pip dependencies.
> > 
> > 
> > A small headache relating fixes to both locations, but if you'd like to
> > see it to prove that the installation mechanism works in general, then
> > OK. I'm willing to deal with the pain until the next release to let us
> > go through a testing cycle. Reluctantly. Maybe.
> > 
> > I'm assuming you mean as a subpackage and not a [git] submodule. If you
> > do mean git, then ... uh. That might be messy.
> 
> Yeah, I meant a git submodule in qemu.git...  It would also be the easiest
> way to build a subpackage in Fedora, since it would be part of the QEMU
> tarballs.

When qemu.qmp is uploaded to PyPi, then Fedora packaging guidelines on
unbundling will expect us to create a dedicated python-qemu.qmp src.rpm,
and use that, not anything QEMU might bundle.

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: iotests and python dependencies

[Paolo Bonzini]

If we want to provide an installable rpm in Fedora then yes; but we can leave distribution to pypi and use a bundled copy in the virtual environment that is used to run tests.

Paolo

Il 5 maggio 2022 17:57:24 CEST, "Daniel P. Berrangé" <berrange@redhat.com> ha scritto:
>On Thu, May 05, 2022 at 05:50:00PM +0200, Paolo Bonzini wrote:
>> On 5/5/22 16:13, John Snow wrote:
>> > 
>> >     I would rather keep python/qemu/qmp as a submodule for a longer time,
>> >     and still go through a virtual environment that installs it together
>> >     with its pip dependencies.
>> > 
>> > 
>> > A small headache relating fixes to both locations, but if you'd like to
>> > see it to prove that the installation mechanism works in general, then
>> > OK. I'm willing to deal with the pain until the next release to let us
>> > go through a testing cycle. Reluctantly. Maybe.
>> > 
>> > I'm assuming you mean as a subpackage and not a [git] submodule. If you
>> > do mean git, then ... uh. That might be messy.
>> 
>> Yeah, I meant a git submodule in qemu.git...  It would also be the easiest
>> way to build a subpackage in Fedora, since it would be part of the QEMU
>> tarballs.
>
>When qemu.qmp is uploaded to PyPi, then Fedora packaging guidelines on
>unbundling will expect us to create a dedicated python-qemu.qmp src.rpm,
>and use that, not anything QEMU might bundle.
>
>With regards,
>Daniel

Re: iotests and python dependencies

[John Snow]

On Thu, May 5, 2022 at 5:28 AM Paolo Bonzini <pbonzini@redhat.com> wrote:
>
> On 5/5/22 10:51, Kevin Wolf wrote:
> > If not, I guess it would be enough if iotests just checks that the venv
> > exists and all of the dependencies are there in the right version and
> > error out if not, telling the user to run 'make check-venv'.
> >
> > Or actually, it could just unconditionally run 'make check-venv' by
> > itself, which is probably easier to implement than checking the
> > dependencies and more convenient for the user, too.
>
> Note that you would still have to add a 'check-block: check-venv'
> dependency in the Makefile, otherwise two "instances" of check-venv
> could run in parallel.
>
> One small complication is that on BSD systems the binary is actually
> called "gmake", so you'd have to pass the variable somehow
>
> Paolo
>

Dumb question: where would I express this dependency? I don't know
where the top-level "check-block" recipe gets defined.

--js