[Qemu-devel] qemu 2.7.0 incompatible with host kernel < 3.19

[Qemu-devel] qemu 2.7.0 incompatible with host kernel < 3.19

[Brian Rak]

I have some KVM hosts that I just upgraded to qemu 2.7.0.  When the host 
is running a kernel newer then 3.19, everything works fine.

If the host is running on something lower then 3.19, some guests fail to 
start.  I noticed this on a Ubuntu 16.04 guest, it fails to find the 
virtio NIC, and this is printed to dmesg:

virtio_net virtio0: virtio: device uses modern interface but does not 
have VIRTIO_F_VERSION_1

Looking around, I found 
https://lists.gnu.org/archive/html/qemu-devel/2015-12/msg00373.html , 
but this does not appear to fully resolve the issue.  Any suggestions 
here?  I'm currently just rolling back the upgrade until we can get the 
kernel upgraded on the rest of the hosts.

Re: [Qemu-devel] qemu 2.7.0 incompatible with host kernel < 3.19

[Laszlo Ersek]

Hi,

(CC Michael and Cornelia,)

On 09/13/16 21:10, Brian Rak wrote:
> I have some KVM hosts that I just upgraded to qemu 2.7.0.  When the host
> is running a kernel newer then 3.19, everything works fine.
> 
> If the host is running on something lower then 3.19, some guests fail to
> start.  I noticed this on a Ubuntu 16.04 guest, it fails to find the
> virtio NIC, and this is printed to dmesg:
> 
> virtio_net virtio0: virtio: device uses modern interface but does not
> have VIRTIO_F_VERSION_1
> 
> Looking around, I found
> https://lists.gnu.org/archive/html/qemu-devel/2015-12/msg00373.html ,
> but this does not appear to fully resolve the issue.  Any suggestions
> here?  I'm currently just rolling back the upgrade until we can get the
> kernel upgraded on the rest of the hosts.

let me begin with saying that I probably don't know what I'm talking
about :)

So, based on your report and the message you linked from the archive, I
think the problem is that vhost-net, in host kernels strictly older than
v3.19, doesn't set VIRTIO_F_VERSION_1.

See this kernel commit:

commit 41e3e42108bc5ebc77d40d6fe1216c483a6b1f9d
Author: Michael S. Tsirkin <mst@redhat.com>
Date:   Fri Oct 24 14:25:03 2014 +0300

    vhost/net: enable virtio 1.0

    Signed-off-by: Michael S. Tsirkin <mst@redhat.com>

This was first released in v3.19. (See "git tag --contains 41e3e42108bc".)

(

Later this would be moved to a more central location, with:

commit 4e9fa50c6ccbebef0c4a4aae84090badf81359e6
Author: Michael S. Tsirkin <mst@redhat.com>
Date:   Wed Sep 9 22:24:56 2015 +0300

    vhost: move features to core

    virtio 1 and any layout are core features, move them
    there. This fixes vhost test.

    Signed-off-by: Michael S. Tsirkin <mst@redhat.com>

but this commit was released in v4.3, and I don't think it should make a
visible difference to your use case.

)

I *guess* that with qemu-2.7.0, any lack of VIRTIO_F_VERSION_1 in the
host kernel's device model has become visible to the guest, and then the
guest driver bails out. (I don't know what exactly brought about this
change in qemu-2.7.0.)

Now, the libvirt documentation says,

http://libvirt.org/formatdomain.html#elementsDriverBackendOptions

    [...] Currently the following attributes are available for the
    "virtio" NIC driver:

    name

        The optional name attribute forces which type of backend driver
        to use. The value can be either 'qemu' (a user-space backend)
        or 'vhost' (a kernel backend, which requires the vhost module
        to be provided by the kernel); an attempt to require the vhost
        driver without kernel support will be rejected. If this
        attribute is not present, then the domain defaults to 'vhost'
        if present, but silently falls back to 'qemu' without error.

So, rather than rolling back the QEMU upgrade, can you test turning off
vhost in your domain XMLs (or, on the qemu command line, with

  -netdev tap,...,vhost=off

) until you can upgrade the host kernels?

Alternatively, you could rmmod / blacklist the "vhost_net" module on
your affected hosts. (This would only work if the domain XML doesn't
explicitly ask for vhost, see above.)

Again, this is just a guess.

Thanks
Laszlo

Re: [Qemu-devel] qemu 2.7.0 incompatible with host kernel < 3.19

[Michael S. Tsirkin]

On Tue, Sep 13, 2016 at 11:09:27PM +0200, Laszlo Ersek wrote:
> Hi,
> 
> (CC Michael and Cornelia,)
> 
> On 09/13/16 21:10, Brian Rak wrote:
> > I have some KVM hosts that I just upgraded to qemu 2.7.0.  When the host
> > is running a kernel newer then 3.19, everything works fine.
> > 
> > If the host is running on something lower then 3.19, some guests fail to
> > start.  I noticed this on a Ubuntu 16.04 guest, it fails to find the
> > virtio NIC, and this is printed to dmesg:
> > 
> > virtio_net virtio0: virtio: device uses modern interface but does not
> > have VIRTIO_F_VERSION_1
> > 
> > Looking around, I found
> > https://lists.gnu.org/archive/html/qemu-devel/2015-12/msg00373.html ,
> > but this does not appear to fully resolve the issue.  Any suggestions
> > here?  I'm currently just rolling back the upgrade until we can get the
> > kernel upgraded on the rest of the hosts.
> 
> let me begin with saying that I probably don't know what I'm talking
> about :)
> 
> So, based on your report and the message you linked from the archive, I
> think the problem is that vhost-net, in host kernels strictly older than
> v3.19, doesn't set VIRTIO_F_VERSION_1.
> 
> See this kernel commit:
> 
> commit 41e3e42108bc5ebc77d40d6fe1216c483a6b1f9d
> Author: Michael S. Tsirkin <mst@redhat.com>
> Date:   Fri Oct 24 14:25:03 2014 +0300
> 
>     vhost/net: enable virtio 1.0
> 
>     Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
> 
> This was first released in v3.19. (See "git tag --contains 41e3e42108bc".)
> 
> (
> 
> Later this would be moved to a more central location, with:
> 
> commit 4e9fa50c6ccbebef0c4a4aae84090badf81359e6
> Author: Michael S. Tsirkin <mst@redhat.com>
> Date:   Wed Sep 9 22:24:56 2015 +0300
> 
>     vhost: move features to core
> 
>     virtio 1 and any layout are core features, move them
>     there. This fixes vhost test.
> 
>     Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
> 
> but this commit was released in v4.3, and I don't think it should make a
> visible difference to your use case.
> 
> )
> 
> I *guess* that with qemu-2.7.0, any lack of VIRTIO_F_VERSION_1 in the
> host kernel's device model has become visible to the guest, and then the
> guest driver bails out. (I don't know what exactly brought about this
> change in qemu-2.7.0.)
> 
> Now, the libvirt documentation says,
> 
> http://libvirt.org/formatdomain.html#elementsDriverBackendOptions
> 
>     [...] Currently the following attributes are available for the
>     "virtio" NIC driver:
> 
>     name
> 
>         The optional name attribute forces which type of backend driver
>         to use. The value can be either 'qemu' (a user-space backend)
>         or 'vhost' (a kernel backend, which requires the vhost module
>         to be provided by the kernel); an attempt to require the vhost
>         driver without kernel support will be rejected. If this
>         attribute is not present, then the domain defaults to 'vhost'
>         if present, but silently falls back to 'qemu' without error.
> 
> So, rather than rolling back the QEMU upgrade, can you test turning off
> vhost in your domain XMLs (or, on the qemu command line, with
> 
>   -netdev tap,...,vhost=off
> 
> ) until you can upgrade the host kernels?
> 
> Alternatively, you could rmmod / blacklist the "vhost_net" module on
> your affected hosts. (This would only work if the domain XML doesn't
> explicitly ask for vhost, see above.)
> 
> Again, this is just a guess.
> 
> Thanks
> Laszlo


This patch:
    virtio-bus: Plug devices after features are negotiated
should help I think.

Could you try it please?

-- 
MST

Re: [Qemu-devel] qemu 2.7.0 incompatible with host kernel < 3.19

[Brian Rak]

On 9/14/2016 12:35 AM, Michael S. Tsirkin wrote:
> On Tue, Sep 13, 2016 at 11:09:27PM +0200, Laszlo Ersek wrote:
>> Hi,
>>
>> (CC Michael and Cornelia,)
>>
>> On 09/13/16 21:10, Brian Rak wrote:
>>> I have some KVM hosts that I just upgraded to qemu 2.7.0.  When the host
>>> is running a kernel newer then 3.19, everything works fine.
>>>
>>> If the host is running on something lower then 3.19, some guests fail to
>>> start.  I noticed this on a Ubuntu 16.04 guest, it fails to find the
>>> virtio NIC, and this is printed to dmesg:
>>>
>>> virtio_net virtio0: virtio: device uses modern interface but does not
>>> have VIRTIO_F_VERSION_1
>>>
>>> Looking around, I found
>>> https://lists.gnu.org/archive/html/qemu-devel/2015-12/msg00373.html ,
>>> but this does not appear to fully resolve the issue.  Any suggestions
>>> here?  I'm currently just rolling back the upgrade until we can get the
>>> kernel upgraded on the rest of the hosts.
>> let me begin with saying that I probably don't know what I'm talking
>> about :)
>>
>> So, based on your report and the message you linked from the archive, I
>> think the problem is that vhost-net, in host kernels strictly older than
>> v3.19, doesn't set VIRTIO_F_VERSION_1.
>>
>> See this kernel commit:
>>
>> commit 41e3e42108bc5ebc77d40d6fe1216c483a6b1f9d
>> Author: Michael S. Tsirkin <mst@redhat.com>
>> Date:   Fri Oct 24 14:25:03 2014 +0300
>>
>>      vhost/net: enable virtio 1.0
>>
>>      Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
>>
>> This was first released in v3.19. (See "git tag --contains 41e3e42108bc".)
>>
>> (
>>
>> Later this would be moved to a more central location, with:
>>
>> commit 4e9fa50c6ccbebef0c4a4aae84090badf81359e6
>> Author: Michael S. Tsirkin <mst@redhat.com>
>> Date:   Wed Sep 9 22:24:56 2015 +0300
>>
>>      vhost: move features to core
>>
>>      virtio 1 and any layout are core features, move them
>>      there. This fixes vhost test.
>>
>>      Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
>>
>> but this commit was released in v4.3, and I don't think it should make a
>> visible difference to your use case.
>>
>> )
>>
>> I *guess* that with qemu-2.7.0, any lack of VIRTIO_F_VERSION_1 in the
>> host kernel's device model has become visible to the guest, and then the
>> guest driver bails out. (I don't know what exactly brought about this
>> change in qemu-2.7.0.)
>>
>> Now, the libvirt documentation says,
>>
>> http://libvirt.org/formatdomain.html#elementsDriverBackendOptions
>>
>>      [...] Currently the following attributes are available for the
>>      "virtio" NIC driver:
>>
>>      name
>>
>>          The optional name attribute forces which type of backend driver
>>          to use. The value can be either 'qemu' (a user-space backend)
>>          or 'vhost' (a kernel backend, which requires the vhost module
>>          to be provided by the kernel); an attempt to require the vhost
>>          driver without kernel support will be rejected. If this
>>          attribute is not present, then the domain defaults to 'vhost'
>>          if present, but silently falls back to 'qemu' without error.
>>
>> So, rather than rolling back the QEMU upgrade, can you test turning off
>> vhost in your domain XMLs (or, on the qemu command line, with
>>
>>    -netdev tap,...,vhost=off
>>
>> ) until you can upgrade the host kernels?
>>
>> Alternatively, you could rmmod / blacklist the "vhost_net" module on
>> your affected hosts. (This would only work if the domain XML doesn't
>> explicitly ask for vhost, see above.)
>>
>> Again, this is just a guess.
>>
>> Thanks
>> Laszlo
>
> This patch:
>      virtio-bus: Plug devices after features are negotiated
> should help I think.
>
> Could you try it please?
>

I can confirm after adding that patch I'm able to see the NIC on a 
Ubuntu 16.04 guest on a host running a kernel older then 3.19.

This gets printed in dmesg on the guest, but it might be there normally 
as well:

[    0.649758] virtio-pci 0000:00:03.0: virtio_pci: leaving for legacy 
driver