A new name for kvm-unit-tests ?

A new name for kvm-unit-tests ?

[Thomas Huth]

 Hi all,

since Paolo recently suggested to decrease the bus factor for
kvm-unit-tests [1], I suggested (in a private mail) to maybe also move
the repo to one of the git forges where we could benefit from a CI
system (so that we do not merge bugs so often anymore as it happened in
the previous months). If we do that step, that might be a good point in
time to rename the kvm-unit-tests to something more adequate. "Why?" you
might ask ... well, the unit tests are not only useful for kvm anymore:

- They can also be used for testing TCG in QEMU

- There have been attempts in the past to use them for OpenBSD [2]

- They can be used for hvf on macOS now [3]

- The s390x tests can also be run with the z/VM and LPAR hypervisors

- Some people also try to run certain tests on x86 bare metal
  (for comparing the behavior of virtual CPUs with real CPUs)

Liran already suggested the name "cpu-unit-tests" a while ago [4], which
is certainly a better description of what the tests are doing nowadays.
But I wonder whether that name is maybe too generic? For example, if
someone does not know that there should be dashes in between and just
searches for "cpu unit tests" with their favorite search engine, would
they find the right project site?

Maybe we should come up with a more fancy name for the test suite? For
example something like "HECATE" - "*H*ypervisor *E*xecution and *C*pu
instruction *A*dvanced *T*est *E*nvironment" (and Hecate is also the
goddess of boundaries - which you could translate to the hypervisor
being the boundary between the virtual and real machine ... with a
little bit of imagination, of course) ... well, yeah, that's just what
came to my mind so far, of course. Let's brainstorm ... do you have any
good ideas for a new name of the kvm-unit-tests? Or do you love the old
name and think it should stay? Or do you think cpu-unit-tests would be
the best fit? Please let us know!

 Thanks,
  Thomas



[1]
https://lore.kernel.org/kvm/e79b76ae-c554-6d28-7556-88b280b8f02f@redhat.com

[2] https://patchwork.kernel.org/patch/9625205/#20244627

[3]
https://lore.kernel.org/kvm/20200320145541.38578-3-r.bolshakov@yadro.com/

[4] https://patchwork.kernel.org/patch/11260891/#23021859

Re: A new name for kvm-unit-tests ?

[Christian Borntraeger]

On 30.07.20 07:41, Thomas Huth wrote:
> 
>  Hi all,
> 
> since Paolo recently suggested to decrease the bus factor for
> kvm-unit-tests [1], I suggested (in a private mail) to maybe also move
> the repo to one of the git forges where we could benefit from a CI

ack.

> system (so that we do not merge bugs so often anymore as it happened in
> the previous months). If we do that step, that might be a good point in
> time to rename the kvm-unit-tests to something more adequate. "Why?" you
> might ask ... well, the unit tests are not only useful for kvm anymore:

I personally dislike renames as you will have old references lurking in
the internet for decades. A rename will result in people continue to using
the old code because the old name is the only thing that they know.

[...] 
> Maybe we should come up with a more fancy name for the test suite? For
> example something like "HECATE" - "*H*ypervisor *E*xecution and *C*pu
> instruction *A*dvanced *T*est *E*nvironment" (and Hecate is also the
> goddess of boundaries - which you could translate to the hypervisor
> being the boundary between the virtual and real machine ... with a
> little bit of imagination, of course) ... well, yeah, that's just what
> came to my mind so far, of course. Let's brainstorm ... do you have any
> good ideas for a new name of the kvm-unit-tests? Or do you love the old
> name and think it should stay? Or do you think cpu-unit-tests would be
> the best fit? Please let us know!

If we rename than hecate or cpu-unit-tests is fine for me, but I prefer
to keep the old name.

Re: A new name for kvm-unit-tests ?

[David Hildenbrand]

On 30.07.20 08:38, Christian Borntraeger wrote:
> 
> 
> On 30.07.20 07:41, Thomas Huth wrote:
>>
>>  Hi all,
>>
>> since Paolo recently suggested to decrease the bus factor for
>> kvm-unit-tests [1], I suggested (in a private mail) to maybe also move
>> the repo to one of the git forges where we could benefit from a CI
> 
> ack.
> 
>> system (so that we do not merge bugs so often anymore as it happened in
>> the previous months). If we do that step, that might be a good point in
>> time to rename the kvm-unit-tests to something more adequate. "Why?" you
>> might ask ... well, the unit tests are not only useful for kvm anymore:
> 
> I personally dislike renames as you will have old references lurking in
> the internet for decades. A rename will result in people continue to using
> the old code because the old name is the only thing that they know.
> 
> [...] 
>> Maybe we should come up with a more fancy name for the test suite? For
>> example something like "HECATE" - "*H*ypervisor *E*xecution and *C*pu
>> instruction *A*dvanced *T*est *E*nvironment" (and Hecate is also the
>> goddess of boundaries - which you could translate to the hypervisor
>> being the boundary between the virtual and real machine ... with a
>> little bit of imagination, of course) ... well, yeah, that's just what
>> came to my mind so far, of course. Let's brainstorm ... do you have any
>> good ideas for a new name of the kvm-unit-tests? Or do you love the old
>> name and think it should stay? Or do you think cpu-unit-tests would be
>> the best fit? Please let us know!
> 
> If we rename than hecate or cpu-unit-tests is fine for me, but I prefer
> to keep the old name.

+1 for keeping the old name.

cpu-unit-tests might also not be completely fitting (I remember we
already do test, or will test in the future I/O stuff like PCI, CCW, ...).

IMHO, It's much more a collection of tests to verify
architecture/standard/whatever compliance (including paravirtualized
interfaces if available).

-- 
Thanks,

David / dhildenb

Re: A new name for kvm-unit-tests ?

[Wanpeng Li]

On Thu, 30 Jul 2020 at 15:10, David Hildenbrand <david@redhat.com> wrote:
>
> On 30.07.20 08:38, Christian Borntraeger wrote:
> >
> >
> > On 30.07.20 07:41, Thomas Huth wrote:
> >>
> >>  Hi all,
> >>
> >> since Paolo recently suggested to decrease the bus factor for
> >> kvm-unit-tests [1], I suggested (in a private mail) to maybe also move
> >> the repo to one of the git forges where we could benefit from a CI
> >
> > ack.
> >
> >> system (so that we do not merge bugs so often anymore as it happened in
> >> the previous months). If we do that step, that might be a good point in
> >> time to rename the kvm-unit-tests to something more adequate. "Why?" you
> >> might ask ... well, the unit tests are not only useful for kvm anymore:
> >
> > I personally dislike renames as you will have old references lurking in
> > the internet for decades. A rename will result in people continue to using
> > the old code because the old name is the only thing that they know.
> >
> > [...]
> >> Maybe we should come up with a more fancy name for the test suite? For
> >> example something like "HECATE" - "*H*ypervisor *E*xecution and *C*pu
> >> instruction *A*dvanced *T*est *E*nvironment" (and Hecate is also the
> >> goddess of boundaries - which you could translate to the hypervisor
> >> being the boundary between the virtual and real machine ... with a
> >> little bit of imagination, of course) ... well, yeah, that's just what
> >> came to my mind so far, of course. Let's brainstorm ... do you have any
> >> good ideas for a new name of the kvm-unit-tests? Or do you love the old
> >> name and think it should stay? Or do you think cpu-unit-tests would be
> >> the best fit? Please let us know!
> >
> > If we rename than hecate or cpu-unit-tests is fine for me, but I prefer
> > to keep the old name.
>
> +1 for keeping the old name.
>
> cpu-unit-tests might also not be completely fitting (I remember we
> already do test, or will test in the future I/O stuff like PCI, CCW, ...).
>
> IMHO, It's much more a collection of tests to verify
> architecture/standard/whatever compliance (including paravirtualized
> interfaces if available).

Vote for keeping the old name.

    Wanpeng

Re: A new name for kvm-unit-tests ?

[Paolo Bonzini]

On 30/07/20 09:13, Wanpeng Li wrote:
>>> I personally dislike renames as you will have old references lurking in
>>> the internet for decades. A rename will result in people continue to using
>>> the old code because the old name is the only thing that they know.
>>
>> +1 for keeping the old name.
>>
>> cpu-unit-tests might also not be completely fitting (I remember we
>> already do test, or will test in the future I/O stuff like PCI, CCW, ...).
>>
>> IMHO, It's much more a collection of tests to verify
>> architecture/standard/whatever compliance (including paravirtualized
>> interfaces if available).

Good point.

> Vote for keeping the old name.

Ok, so either old name or alternatively arch-unit-tests?  But the
majority seems to be for kvm-unit-tests, and if Nadav has no trouble
contributing to them I suppose everyone else can too.

Paolo

Re: A new name for kvm-unit-tests ?

[Nadav Amit]

> On Jul 30, 2020, at 12:35 AM, Paolo Bonzini <pbonzini@redhat.com> wrote:
> 
> On 30/07/20 09:13, Wanpeng Li wrote:
>>>> I personally dislike renames as you will have old references lurking in
>>>> the internet for decades. A rename will result in people continue to using
>>>> the old code because the old name is the only thing that they know.
>>> 
>>> +1 for keeping the old name.
>>> 
>>> cpu-unit-tests might also not be completely fitting (I remember we
>>> already do test, or will test in the future I/O stuff like PCI, CCW, ...).
>>> 
>>> IMHO, It's much more a collection of tests to verify
>>> architecture/standard/whatever compliance (including paravirtualized
>>> interfaces if available).
> 
> Good point.
> 
>> Vote for keeping the old name.
> 
> Ok, so either old name or alternatively arch-unit-tests?  But the
> majority seems to be for kvm-unit-tests, and if Nadav has no trouble
> contributing to them I suppose everyone else can too.

Indeed. My employer (VMware) did not give me hard time (so far) in
contributing to the project just because it has KVM in its name. We (VMware)
also benefit from kvm-unit-tests, and Paolo and others were receptive to
changes that I made to make it more kvm/qemu -independent. This is what
matters.

So I am ok with the name being kvm-unit-tests. But I would ask/recommend
that the project description [1] be updated to reflect the fact that the
project is hypervisor-agnostic.

This should have practical implications. I remember, for example, that I had
a discussion with Paolo in the past regarding “xpass” being reported as a
failure. The rationale was that if a test that is expected to fail on KVM
(since KVM is known to be broken) surprisingly passes, there is some problem
that should be reported as a failure. I would argue that if the project is
hypervisor-agnostic, “xpass” is not a failure.

So it is much more important how the project is defined than how it is
named.

[1] http://www.linux-kvm.org/page/KVM-unit-tests

Re: A new name for kvm-unit-tests ?

[Andrew Jones]

On Thu, Jul 30, 2020 at 07:50:39AM +0000, Nadav Amit wrote:
> > On Jul 30, 2020, at 12:35 AM, Paolo Bonzini <pbonzini@redhat.com> wrote:
> > 
> > On 30/07/20 09:13, Wanpeng Li wrote:
> >>>> I personally dislike renames as you will have old references lurking in
> >>>> the internet for decades. A rename will result in people continue to using
> >>>> the old code because the old name is the only thing that they know.
> >>> 
> >>> +1 for keeping the old name.
> >>> 
> >>> cpu-unit-tests might also not be completely fitting (I remember we
> >>> already do test, or will test in the future I/O stuff like PCI, CCW, ...).
> >>> 
> >>> IMHO, It's much more a collection of tests to verify
> >>> architecture/standard/whatever compliance (including paravirtualized
> >>> interfaces if available).
> > 
> > Good point.
> > 
> >> Vote for keeping the old name.
> > 
> > Ok, so either old name or alternatively arch-unit-tests?  But the
> > majority seems to be for kvm-unit-tests, and if Nadav has no trouble
> > contributing to them I suppose everyone else can too.
> 
> Indeed. My employer (VMware) did not give me hard time (so far) in
> contributing to the project just because it has KVM in its name. We (VMware)
> also benefit from kvm-unit-tests, and Paolo and others were receptive to
> changes that I made to make it more kvm/qemu -independent. This is what
> matters.
> 
> So I am ok with the name being kvm-unit-tests. But I would ask/recommend
> that the project description [1] be updated to reflect the fact that the
> project is hypervisor-agnostic.

Good idea. Although while I authored what you see there, I don't really
want to sign up to do all the writing. How about when we create the gitlab
project we also create a .md file that we redirect [1] to? Then anybody
can submit patches for it going forward.

> 
> This should have practical implications. I remember, for example, that I had
> a discussion with Paolo in the past regarding “xpass” being reported as a
> failure. The rationale was that if a test that is expected to fail on KVM
> (since KVM is known to be broken) surprisingly passes, there is some problem
> that should be reported as a failure. I would argue that if the project is
> hypervisor-agnostic, “xpass” is not a failure.

We can use compile-time or run-time logic that depends on the target to
decide whether a test should be a normal test (pass/fail) or an
xpass/xfail test.

Thanks,
drew

> 
> So it is much more important how the project is defined than how it is
> named.
> 
> [1] http://www.linux-kvm.org/page/KVM-unit-tests

Re: A new name for kvm-unit-tests ?

[Paolo Bonzini]

On 30/07/20 13:32, Andrew Jones wrote:
>> This should have practical implications. I remember, for example, that I had
>> a discussion with Paolo in the past regarding “xpass” being reported as a
>> failure. The rationale was that if a test that is expected to fail on KVM
>> (since KVM is known to be broken) surprisingly passes, there is some problem
>> that should be reported as a failure. I would argue that if the project is
>> hypervisor-agnostic, “xpass” is not a failure.
> We can use compile-time or run-time logic that depends on the target to
> decide whether a test should be a normal test (pass/fail) or an
> xpass/xfail test.

Yeah, that would be basically the old "errata" mechanism.

Probably we should have some kind of configure script that builds
"errata" files based on hypervisor, uname or whatnot.  (Upstream should
do the bare minimum, just the hypervisor).

Paolo

Re: A new name for kvm-unit-tests ?

[Nadav Amit]

> On Jul 30, 2020, at 4:32 AM, Andrew Jones <drjones@redhat.com> wrote:
> 
> On Thu, Jul 30, 2020 at 07:50:39AM +0000, Nadav Amit wrote:
>>> On Jul 30, 2020, at 12:35 AM, Paolo Bonzini <pbonzini@redhat.com> wrote:
>>> 
>>> On 30/07/20 09:13, Wanpeng Li wrote:
>>>>>> I personally dislike renames as you will have old references lurking in
>>>>>> the internet for decades. A rename will result in people continue to using
>>>>>> the old code because the old name is the only thing that they know.
>>>>> 
>>>>> +1 for keeping the old name.
>>>>> 
>>>>> cpu-unit-tests might also not be completely fitting (I remember we
>>>>> already do test, or will test in the future I/O stuff like PCI, CCW, ...).
>>>>> 
>>>>> IMHO, It's much more a collection of tests to verify
>>>>> architecture/standard/whatever compliance (including paravirtualized
>>>>> interfaces if available).
>>> 
>>> Good point.
>>> 
>>>> Vote for keeping the old name.
>>> 
>>> Ok, so either old name or alternatively arch-unit-tests?  But the
>>> majority seems to be for kvm-unit-tests, and if Nadav has no trouble
>>> contributing to them I suppose everyone else can too.
>> 
>> Indeed. My employer (VMware) did not give me hard time (so far) in
>> contributing to the project just because it has KVM in its name. We (VMware)
>> also benefit from kvm-unit-tests, and Paolo and others were receptive to
>> changes that I made to make it more kvm/qemu -independent. This is what
>> matters.
>> 
>> So I am ok with the name being kvm-unit-tests. But I would ask/recommend
>> that the project description [1] be updated to reflect the fact that the
>> project is hypervisor-agnostic.
> 
> Good idea. Although while I authored what you see there, I don't really
> want to sign up to do all the writing. How about when we create the gitlab
> project we also create a .md file that we redirect [1] to? Then anybody
> can submit patches for it going forward.

Fine with me. Right now the README.md pretty much redirects to [1] in regard
to the project definition.

>> This should have practical implications. I remember, for example, that I had
>> a discussion with Paolo in the past regarding “xpass” being reported as a
>> failure. The rationale was that if a test that is expected to fail on KVM
>> (since KVM is known to be broken) surprisingly passes, there is some problem
>> that should be reported as a failure. I would argue that if the project is
>> hypervisor-agnostic, “xpass” is not a failure.
> 
> We can use compile-time or run-time logic that depends on the target to
> decide whether a test should be a normal test (pass/fail) or an
> xpass/xfail test.

This is simple. When I find some time, I will send some patches for that.

Regards,
Nadav

Re: A new name for kvm-unit-tests ?

[Paolo Bonzini]

On 30/07/20 19:32, Nadav Amit wrote:
>> We can use compile-time or run-time logic that depends on the target to
>> decide whether a test should be a normal test (pass/fail) or an
>> xpass/xfail test.
> This is simple. When I find some time, I will send some patches for that.

Not too quick, let's first look at a design.

Paolo

Re: A new name for kvm-unit-tests ?

[Nadav Amit]

> On Jul 30, 2020, at 10:35 AM, Paolo Bonzini <pbonzini@redhat.com> wrote:
> 
> On 30/07/20 19:32, Nadav Amit wrote:
>>> We can use compile-time or run-time logic that depends on the target to
>>> decide whether a test should be a normal test (pass/fail) or an
>>> xpass/xfail test.
>> This is simple. When I find some time, I will send some patches for that.
> 
> Not too quick, let's first look at a design.

Don’t worry, I am busy ;-)

The subtext of my response was that a fully fledged errata mechanism is more
complicated.

Re: A new name for kvm-unit-tests ?

[Thomas Huth]

On 30/07/2020 13.32, Andrew Jones wrote:
> On Thu, Jul 30, 2020 at 07:50:39AM +0000, Nadav Amit wrote:
>>> On Jul 30, 2020, at 12:35 AM, Paolo Bonzini <pbonzini@redhat.com> wrote:
>>>
>>> On 30/07/20 09:13, Wanpeng Li wrote:
>>>>>> I personally dislike renames as you will have old references lurking in
>>>>>> the internet for decades. A rename will result in people continue to using
>>>>>> the old code because the old name is the only thing that they know.
>>>>>
>>>>> +1 for keeping the old name.
>>>>>
>>>>> cpu-unit-tests might also not be completely fitting (I remember we
>>>>> already do test, or will test in the future I/O stuff like PCI, CCW, ...).
>>>>>
>>>>> IMHO, It's much more a collection of tests to verify
>>>>> architecture/standard/whatever compliance (including paravirtualized
>>>>> interfaces if available).
>>>
>>> Good point.
>>>
>>>> Vote for keeping the old name.
>>>
>>> Ok, so either old name or alternatively arch-unit-tests?  But the
>>> majority seems to be for kvm-unit-tests, and if Nadav has no trouble
>>> contributing to them I suppose everyone else can too.
>>
>> Indeed. My employer (VMware) did not give me hard time (so far) in
>> contributing to the project just because it has KVM in its name. We (VMware)
>> also benefit from kvm-unit-tests, and Paolo and others were receptive to
>> changes that I made to make it more kvm/qemu -independent. This is what
>> matters.
>>
>> So I am ok with the name being kvm-unit-tests. But I would ask/recommend
>> that the project description [1] be updated to reflect the fact that the
>> project is hypervisor-agnostic.
> 
> Good idea. Although while I authored what you see there, I don't really
> want to sign up to do all the writing. How about when we create the gitlab
> project we also create a .md file that we redirect [1] to? Then anybody
> can submit patches for it going forward.

The README.md can now be viewed here:

https://gitlab.com/kvm-unit-tests/kvm-unit-tests/-/blob/master/README.md

 Thomas

Re: A new name for kvm-unit-tests ?

[Andrew Jones]

On Fri, Jul 31, 2020 at 07:53:31AM +0200, Thomas Huth wrote:
> On 30/07/2020 13.32, Andrew Jones wrote:
> > On Thu, Jul 30, 2020 at 07:50:39AM +0000, Nadav Amit wrote:
> >>> On Jul 30, 2020, at 12:35 AM, Paolo Bonzini <pbonzini@redhat.com> wrote:
> >>>
> >>> On 30/07/20 09:13, Wanpeng Li wrote:
> >>>>>> I personally dislike renames as you will have old references lurking in
> >>>>>> the internet for decades. A rename will result in people continue to using
> >>>>>> the old code because the old name is the only thing that they know.
> >>>>>
> >>>>> +1 for keeping the old name.
> >>>>>
> >>>>> cpu-unit-tests might also not be completely fitting (I remember we
> >>>>> already do test, or will test in the future I/O stuff like PCI, CCW, ...).
> >>>>>
> >>>>> IMHO, It's much more a collection of tests to verify
> >>>>> architecture/standard/whatever compliance (including paravirtualized
> >>>>> interfaces if available).
> >>>
> >>> Good point.
> >>>
> >>>> Vote for keeping the old name.
> >>>
> >>> Ok, so either old name or alternatively arch-unit-tests?  But the
> >>> majority seems to be for kvm-unit-tests, and if Nadav has no trouble
> >>> contributing to them I suppose everyone else can too.
> >>
> >> Indeed. My employer (VMware) did not give me hard time (so far) in
> >> contributing to the project just because it has KVM in its name. We (VMware)
> >> also benefit from kvm-unit-tests, and Paolo and others were receptive to
> >> changes that I made to make it more kvm/qemu -independent. This is what
> >> matters.
> >>
> >> So I am ok with the name being kvm-unit-tests. But I would ask/recommend
> >> that the project description [1] be updated to reflect the fact that the
> >> project is hypervisor-agnostic.
> > 
> > Good idea. Although while I authored what you see there, I don't really
> > want to sign up to do all the writing. How about when we create the gitlab
> > project we also create a .md file that we redirect [1] to? Then anybody
> > can submit patches for it going forward.
> 
> The README.md can now be viewed here:
> 
> https://gitlab.com/kvm-unit-tests/kvm-unit-tests/-/blob/master/README.md
>

Yup, and now I'll try to find time to reformat and import the content at
[1] to another .md file, which we'll keep in the git repo too. I'll then
change the link at the top of the README to point to that and change [1]
to point to it as well.

Thanks,
drew

[1] http://www.linux-kvm.org/page/KVM-unit-tests