* Re: virtio-net: Unpermitted usage of virtqueue before virtio driver initialization [not found] <c0c17b91-747e-ab58-83e5-b6f7dfa55e75@opensynergy.com> @ 2022-04-19 15:03 ` Michael S. Tsirkin 2022-04-20 3:07 ` Jason Wang 0 siblings, 1 reply; 9+ messages in thread From: Michael S. Tsirkin @ 2022-04-19 15:03 UTC (permalink / raw) To: Maciej Szymański; +Cc: Willem de Bruijn, virtualization On Tue, Apr 19, 2022 at 04:12:31PM +0200, Maciej Szymański wrote: > Hello, > > I've found a problem in virtio-net driver. > If virtio-net backend device advertises guest offload features, there is > an unpermitted usage of control virtqueue before driver is initialized. > According to VIRTIO specification 2.1.2 : > "The device MUST NOT consume buffers or send any used buffer > notifications to the driver before DRIVER_OK." Right. > During an initialization, driver calls register_netdevice which invokes > callback function virtnet_set_features from __netdev_update_features. > If guest offload features are advertised by the device, > virtnet_set_guest_offloads is using virtnet_send_command to write and > read from VQ. > That leads to initialization stuck as device is not permitted yet to use VQ. Hmm so we have this: if ((dev->features ^ features) & NETIF_F_GRO_HW) { if (vi->xdp_enabled) return -EBUSY; if (features & NETIF_F_GRO_HW) offloads = vi->guest_offloads_capable; else offloads = vi->guest_offloads_capable & ~GUEST_OFFLOAD_GRO_HW_MASK; err = virtnet_set_guest_offloads(vi, offloads); if (err) return err; vi->guest_offloads = offloads; } which I guess should have prevented virtnet_set_guest_offloads from ever running. From your description it sounds like you have observed this in practice, right? > I have attached a patch for kernel 5.18-rc3 which fixes the problem by > deferring feature set after virtio driver initialization. > > Best Regards, > > -- > Maciej Szymański > Senior Staff Engineer > > OpenSynergy GmbH > Rotherstr. 20, 10245 Berlin > > Phone: +49 30 60 98 54 0 -86 > Fax: +49 30 60 98 54 0 -99 > E-Mail: maciej.szymanski@opensynergy.com > > www.opensynergy.com > > Handelsregister/Commercial Registry: Amtsgericht Charlottenburg, HRB 108616B > Geschäftsführer/Managing Director: Regis Adjamah > > Please mind our privacy notice<https://www.opensynergy.com/datenschutzerklaerung/privacy-notice-for-business-partners-pursuant-to-article-13-of-the-general-data-protection-regulation-gdpr/> pursuant to Art. 13 GDPR. // Unsere Hinweise zum Datenschutz gem. Art. 13 DSGVO finden Sie hier.<https://www.opensynergy.com/de/datenschutzerklaerung/datenschutzhinweise-fuer-geschaeftspartner-gem-art-13-dsgvo/> > diff --git a/drivers/net/virtio_net.c b/drivers/net/virtio_net.c > index 87838cb..a44462d 100644 > --- a/drivers/net/virtio_net.c > +++ b/drivers/net/virtio_net.c > @@ -264,6 +264,8 @@ struct virtnet_info { > unsigned long guest_offloads; > unsigned long guest_offloads_capable; > > + netdev_features_t features; > + I don't much like how we are forced to keep a copy of features here :( At least pls add a comment explaining what's going on, who owns this etc. > /* failover when STANDBY feature enabled */ > struct failover *failover; > }; > @@ -2976,6 +2978,15 @@ static int virtnet_get_phys_port_name(struct net_device *dev, char *buf, > > static int virtnet_set_features(struct net_device *dev, > netdev_features_t features) > +{ > + struct virtnet_info *vi = netdev_priv(dev); > + vi->features = features; > + > + return 0; > +} Looks like this breaks changing features after initialization - these will never be propagated to hardware now. > + > +static int virtnet_set_features_deferred(struct net_device *dev, > + netdev_features_t features) > { > struct virtnet_info *vi = netdev_priv(dev); > u64 offloads; > @@ -3644,6 +3655,13 @@ static int virtnet_probe(struct virtio_device *vdev) > > virtio_device_ready(vdev); > > + /* Deferred feature set after device ready */ > + err = virtnet_set_features_deferred(dev, vi->features); It seems that if this is called e.g. for a device without a CVQ and there are things that actually need to change then it will BUG_ON. > + if (err) { > + pr_debug("virtio_net: set features failed\n"); > + goto free_unregister_netdev; > + } > + > err = virtnet_cpu_notif_add(vi); > if (err) { > pr_debug("virtio_net: registering cpu notifier failed\n"); > -- MST _______________________________________________ Virtualization mailing list Virtualization@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/virtualization ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: virtio-net: Unpermitted usage of virtqueue before virtio driver initialization 2022-04-19 15:03 ` virtio-net: Unpermitted usage of virtqueue before virtio driver initialization Michael S. Tsirkin @ 2022-04-20 3:07 ` Jason Wang 2022-04-20 6:35 ` Michael S. Tsirkin 0 siblings, 1 reply; 9+ messages in thread From: Jason Wang @ 2022-04-20 3:07 UTC (permalink / raw) To: Michael S. Tsirkin Cc: Willem de Bruijn, Maciej Szymański, virtualization On Tue, Apr 19, 2022 at 11:03 PM Michael S. Tsirkin <mst@redhat.com> wrote: > > On Tue, Apr 19, 2022 at 04:12:31PM +0200, Maciej Szymański wrote: > > Hello, > > > > I've found a problem in virtio-net driver. > > If virtio-net backend device advertises guest offload features, there is > > an unpermitted usage of control virtqueue before driver is initialized. > > According to VIRTIO specification 2.1.2 : > > "The device MUST NOT consume buffers or send any used buffer > > notifications to the driver before DRIVER_OK." > > Right. > > > During an initialization, driver calls register_netdevice which invokes > > callback function virtnet_set_features from __netdev_update_features. > > If guest offload features are advertised by the device, > > virtnet_set_guest_offloads is using virtnet_send_command to write and > > read from VQ. > > That leads to initialization stuck as device is not permitted yet to use VQ. > > > > Hmm so we have this: > > > if ((dev->features ^ features) & NETIF_F_GRO_HW) { > if (vi->xdp_enabled) > return -EBUSY; > > if (features & NETIF_F_GRO_HW) > offloads = vi->guest_offloads_capable; > else > offloads = vi->guest_offloads_capable & > ~GUEST_OFFLOAD_GRO_HW_MASK; > > err = virtnet_set_guest_offloads(vi, offloads); > if (err) > return err; > vi->guest_offloads = offloads; > } > > which I guess should have prevented virtnet_set_guest_offloads from ever running. > > From your description it sounds like you have observed this > in practice, right? > > > > > I have attached a patch for kernel 5.18-rc3 which fixes the problem by > > deferring feature set after virtio driver initialization. > > > > Best Regards, > > > > -- > > Maciej Szymański > > Senior Staff Engineer > > > > OpenSynergy GmbH > > Rotherstr. 20, 10245 Berlin > > > > Phone: +49 30 60 98 54 0 -86 > > Fax: +49 30 60 98 54 0 -99 > > E-Mail: maciej.szymanski@opensynergy.com > > > > www.opensynergy.com > > > > Handelsregister/Commercial Registry: Amtsgericht Charlottenburg, HRB 108616B > > Geschäftsführer/Managing Director: Regis Adjamah > > > > Please mind our privacy notice<https://www.opensynergy.com/datenschutzerklaerung/privacy-notice-for-business-partners-pursuant-to-article-13-of-the-general-data-protection-regulation-gdpr/> pursuant to Art. 13 GDPR. // Unsere Hinweise zum Datenschutz gem. Art. 13 DSGVO finden Sie hier.<https://www.opensynergy.com/de/datenschutzerklaerung/datenschutzhinweise-fuer-geschaeftspartner-gem-art-13-dsgvo/> > > > diff --git a/drivers/net/virtio_net.c b/drivers/net/virtio_net.c > > index 87838cb..a44462d 100644 > > --- a/drivers/net/virtio_net.c > > +++ b/drivers/net/virtio_net.c > > @@ -264,6 +264,8 @@ struct virtnet_info { > > unsigned long guest_offloads; > > unsigned long guest_offloads_capable; > > > > + netdev_features_t features; > > + > > I don't much like how we are forced to keep a copy of features > here :( At least pls add a comment explaining what's going on, > who owns this etc. > > > /* failover when STANDBY feature enabled */ > > struct failover *failover; > > }; > > @@ -2976,6 +2978,15 @@ static int virtnet_get_phys_port_name(struct net_device *dev, char *buf, > > > > static int virtnet_set_features(struct net_device *dev, > > netdev_features_t features) > > +{ > > + struct virtnet_info *vi = netdev_priv(dev); > > + vi->features = features; > > + > > + return 0; > > +} > > > Looks like this breaks changing features after initialization - > these will never be propagated to hardware now. Yes, I think we need to have a check and only defer the setting when virtio device is not ready. Thanks > > > + > > +static int virtnet_set_features_deferred(struct net_device *dev, > > + netdev_features_t features) > > { > > struct virtnet_info *vi = netdev_priv(dev); > > u64 offloads; > > @@ -3644,6 +3655,13 @@ static int virtnet_probe(struct virtio_device *vdev) > > > > virtio_device_ready(vdev); > > > > + /* Deferred feature set after device ready */ > > + err = virtnet_set_features_deferred(dev, vi->features); > > > It seems that if this is called e.g. for a device without a CVQ and > there are things that actually need to change then it will BUG_ON. > > > > + if (err) { > > + pr_debug("virtio_net: set features failed\n"); > > + goto free_unregister_netdev; > > + } > > + > > err = virtnet_cpu_notif_add(vi); > > if (err) { > > pr_debug("virtio_net: registering cpu notifier failed\n"); > > > > -- > MST > _______________________________________________ Virtualization mailing list Virtualization@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/virtualization ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: virtio-net: Unpermitted usage of virtqueue before virtio driver initialization 2022-04-20 3:07 ` Jason Wang @ 2022-04-20 6:35 ` Michael S. Tsirkin 2022-04-20 8:05 ` Jason Wang [not found] ` <4080d799-b42e-018a-8b14-621295e55a8d@opensynergy.com> 0 siblings, 2 replies; 9+ messages in thread From: Michael S. Tsirkin @ 2022-04-20 6:35 UTC (permalink / raw) To: Jason Wang; +Cc: Willem de Bruijn, Maciej Szymański, virtualization On Wed, Apr 20, 2022 at 11:07:00AM +0800, Jason Wang wrote: > On Tue, Apr 19, 2022 at 11:03 PM Michael S. Tsirkin <mst@redhat.com> wrote: > > > > On Tue, Apr 19, 2022 at 04:12:31PM +0200, Maciej Szymański wrote: > > > Hello, > > > > > > I've found a problem in virtio-net driver. > > > If virtio-net backend device advertises guest offload features, there is > > > an unpermitted usage of control virtqueue before driver is initialized. > > > According to VIRTIO specification 2.1.2 : > > > "The device MUST NOT consume buffers or send any used buffer > > > notifications to the driver before DRIVER_OK." > > > > Right. > > > > > During an initialization, driver calls register_netdevice which invokes > > > callback function virtnet_set_features from __netdev_update_features. > > > If guest offload features are advertised by the device, > > > virtnet_set_guest_offloads is using virtnet_send_command to write and > > > read from VQ. > > > That leads to initialization stuck as device is not permitted yet to use VQ. > > > > > > > > Hmm so we have this: > > > > > > if ((dev->features ^ features) & NETIF_F_GRO_HW) { > > if (vi->xdp_enabled) > > return -EBUSY; > > > > if (features & NETIF_F_GRO_HW) > > offloads = vi->guest_offloads_capable; > > else > > offloads = vi->guest_offloads_capable & > > ~GUEST_OFFLOAD_GRO_HW_MASK; > > > > err = virtnet_set_guest_offloads(vi, offloads); > > if (err) > > return err; > > vi->guest_offloads = offloads; > > } > > > > which I guess should have prevented virtnet_set_guest_offloads from ever running. > > > > From your description it sounds like you have observed this > > in practice, right? > > > > > > > > > I have attached a patch for kernel 5.18-rc3 which fixes the problem by > > > deferring feature set after virtio driver initialization. > > > > > > Best Regards, > > > > > > -- > > > Maciej Szymański > > > Senior Staff Engineer > > > > > > OpenSynergy GmbH > > > Rotherstr. 20, 10245 Berlin > > > > > > Phone: +49 30 60 98 54 0 -86 > > > Fax: +49 30 60 98 54 0 -99 > > > E-Mail: maciej.szymanski@opensynergy.com > > > > > > www.opensynergy.com > > > > > > Handelsregister/Commercial Registry: Amtsgericht Charlottenburg, HRB 108616B > > > Geschäftsführer/Managing Director: Regis Adjamah > > > > > > Please mind our privacy notice<https://www.opensynergy.com/datenschutzerklaerung/privacy-notice-for-business-partners-pursuant-to-article-13-of-the-general-data-protection-regulation-gdpr/> pursuant to Art. 13 GDPR. // Unsere Hinweise zum Datenschutz gem. Art. 13 DSGVO finden Sie hier.<https://www.opensynergy.com/de/datenschutzerklaerung/datenschutzhinweise-fuer-geschaeftspartner-gem-art-13-dsgvo/> > > > > > diff --git a/drivers/net/virtio_net.c b/drivers/net/virtio_net.c > > > index 87838cb..a44462d 100644 > > > --- a/drivers/net/virtio_net.c > > > +++ b/drivers/net/virtio_net.c > > > @@ -264,6 +264,8 @@ struct virtnet_info { > > > unsigned long guest_offloads; > > > unsigned long guest_offloads_capable; > > > > > > + netdev_features_t features; > > > + > > > > I don't much like how we are forced to keep a copy of features > > here :( At least pls add a comment explaining what's going on, > > who owns this etc. > > > > > /* failover when STANDBY feature enabled */ > > > struct failover *failover; > > > }; > > > @@ -2976,6 +2978,15 @@ static int virtnet_get_phys_port_name(struct net_device *dev, char *buf, > > > > > > static int virtnet_set_features(struct net_device *dev, > > > netdev_features_t features) > > > +{ > > > + struct virtnet_info *vi = netdev_priv(dev); > > > + vi->features = features; > > > + > > > + return 0; > > > +} > > > > > > Looks like this breaks changing features after initialization - > > these will never be propagated to hardware now. > > Yes, I think we need to have a check and only defer the setting when > virtio device is not ready. > > Thanks I think we should first understand how does the issue trigger, is this a theoretical or a practical issue. > > > > > + > > > +static int virtnet_set_features_deferred(struct net_device *dev, > > > + netdev_features_t features) > > > { > > > struct virtnet_info *vi = netdev_priv(dev); > > > u64 offloads; > > > @@ -3644,6 +3655,13 @@ static int virtnet_probe(struct virtio_device *vdev) > > > > > > virtio_device_ready(vdev); > > > > > > + /* Deferred feature set after device ready */ > > > + err = virtnet_set_features_deferred(dev, vi->features); > > > > > > It seems that if this is called e.g. for a device without a CVQ and > > there are things that actually need to change then it will BUG_ON. > > > > > > > + if (err) { > > > + pr_debug("virtio_net: set features failed\n"); > > > + goto free_unregister_netdev; > > > + } > > > + > > > err = virtnet_cpu_notif_add(vi); > > > if (err) { > > > pr_debug("virtio_net: registering cpu notifier failed\n"); > > > > > > > -- > > MST > > _______________________________________________ Virtualization mailing list Virtualization@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/virtualization ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: virtio-net: Unpermitted usage of virtqueue before virtio driver initialization 2022-04-20 6:35 ` Michael S. Tsirkin @ 2022-04-20 8:05 ` Jason Wang [not found] ` <4080d799-b42e-018a-8b14-621295e55a8d@opensynergy.com> 1 sibling, 0 replies; 9+ messages in thread From: Jason Wang @ 2022-04-20 8:05 UTC (permalink / raw) To: Michael S. Tsirkin Cc: Willem de Bruijn, Maciej Szymański, virtualization 在 2022/4/20 14:35, Michael S. Tsirkin 写道: > On Wed, Apr 20, 2022 at 11:07:00AM +0800, Jason Wang wrote: >> On Tue, Apr 19, 2022 at 11:03 PM Michael S. Tsirkin <mst@redhat.com> wrote: >>> On Tue, Apr 19, 2022 at 04:12:31PM +0200, Maciej Szymański wrote: >>>> Hello, >>>> >>>> I've found a problem in virtio-net driver. >>>> If virtio-net backend device advertises guest offload features, there is >>>> an unpermitted usage of control virtqueue before driver is initialized. >>>> According to VIRTIO specification 2.1.2 : >>>> "The device MUST NOT consume buffers or send any used buffer >>>> notifications to the driver before DRIVER_OK." >>> Right. >>> >>>> During an initialization, driver calls register_netdevice which invokes >>>> callback function virtnet_set_features from __netdev_update_features. >>>> If guest offload features are advertised by the device, >>>> virtnet_set_guest_offloads is using virtnet_send_command to write and >>>> read from VQ. >>>> That leads to initialization stuck as device is not permitted yet to use VQ. >>> >>> >>> Hmm so we have this: >>> >>> >>> if ((dev->features ^ features) & NETIF_F_GRO_HW) { >>> if (vi->xdp_enabled) >>> return -EBUSY; >>> >>> if (features & NETIF_F_GRO_HW) >>> offloads = vi->guest_offloads_capable; >>> else >>> offloads = vi->guest_offloads_capable & >>> ~GUEST_OFFLOAD_GRO_HW_MASK; >>> >>> err = virtnet_set_guest_offloads(vi, offloads); >>> if (err) >>> return err; >>> vi->guest_offloads = offloads; >>> } >>> >>> which I guess should have prevented virtnet_set_guest_offloads from ever running. >>> >>> From your description it sounds like you have observed this >>> in practice, right? >>> >>> >>> >>>> I have attached a patch for kernel 5.18-rc3 which fixes the problem by >>>> deferring feature set after virtio driver initialization. >>>> >>>> Best Regards, >>>> >>>> -- >>>> Maciej Szymański >>>> Senior Staff Engineer >>>> >>>> OpenSynergy GmbH >>>> Rotherstr. 20, 10245 Berlin >>>> >>>> Phone: +49 30 60 98 54 0 -86 >>>> Fax: +49 30 60 98 54 0 -99 >>>> E-Mail: maciej.szymanski@opensynergy.com >>>> >>>> www.opensynergy.com >>>> >>>> Handelsregister/Commercial Registry: Amtsgericht Charlottenburg, HRB 108616B >>>> Geschäftsführer/Managing Director: Regis Adjamah >>>> >>>> Please mind our privacy notice<https://www.opensynergy.com/datenschutzerklaerung/privacy-notice-for-business-partners-pursuant-to-article-13-of-the-general-data-protection-regulation-gdpr/> pursuant to Art. 13 GDPR. // Unsere Hinweise zum Datenschutz gem. Art. 13 DSGVO finden Sie hier.<https://www.opensynergy.com/de/datenschutzerklaerung/datenschutzhinweise-fuer-geschaeftspartner-gem-art-13-dsgvo/> >>>> diff --git a/drivers/net/virtio_net.c b/drivers/net/virtio_net.c >>>> index 87838cb..a44462d 100644 >>>> --- a/drivers/net/virtio_net.c >>>> +++ b/drivers/net/virtio_net.c >>>> @@ -264,6 +264,8 @@ struct virtnet_info { >>>> unsigned long guest_offloads; >>>> unsigned long guest_offloads_capable; >>>> >>>> + netdev_features_t features; >>>> + >>> I don't much like how we are forced to keep a copy of features >>> here :( At least pls add a comment explaining what's going on, >>> who owns this etc. >>> >>>> /* failover when STANDBY feature enabled */ >>>> struct failover *failover; >>>> }; >>>> @@ -2976,6 +2978,15 @@ static int virtnet_get_phys_port_name(struct net_device *dev, char *buf, >>>> >>>> static int virtnet_set_features(struct net_device *dev, >>>> netdev_features_t features) >>>> +{ >>>> + struct virtnet_info *vi = netdev_priv(dev); >>>> + vi->features = features; >>>> + >>>> + return 0; >>>> +} >>> >>> Looks like this breaks changing features after initialization - >>> these will never be propagated to hardware now. >> Yes, I think we need to have a check and only defer the setting when >> virtio device is not ready. >> >> Thanks > I think we should first understand how does the issue trigger, > is this a theoretical or a practical issue. It looks like a practical issue. We don't meet it in Qemu probably because Qemu doesn't check DRIVER_OK when processing ctrl vq. Thanks > >>>> + >>>> +static int virtnet_set_features_deferred(struct net_device *dev, >>>> + netdev_features_t features) >>>> { >>>> struct virtnet_info *vi = netdev_priv(dev); >>>> u64 offloads; >>>> @@ -3644,6 +3655,13 @@ static int virtnet_probe(struct virtio_device *vdev) >>>> >>>> virtio_device_ready(vdev); >>>> >>>> + /* Deferred feature set after device ready */ >>>> + err = virtnet_set_features_deferred(dev, vi->features); >>> >>> It seems that if this is called e.g. for a device without a CVQ and >>> there are things that actually need to change then it will BUG_ON. >>> >>> >>>> + if (err) { >>>> + pr_debug("virtio_net: set features failed\n"); >>>> + goto free_unregister_netdev; >>>> + } >>>> + >>>> err = virtnet_cpu_notif_add(vi); >>>> if (err) { >>>> pr_debug("virtio_net: registering cpu notifier failed\n"); >>>> >>> -- >>> MST >>> _______________________________________________ Virtualization mailing list Virtualization@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/virtualization ^ permalink raw reply [flat|nested] 9+ messages in thread
[parent not found: <4080d799-b42e-018a-8b14-621295e55a8d@opensynergy.com>]
* Re: virtio-net: Unpermitted usage of virtqueue before virtio driver initialization [not found] ` <4080d799-b42e-018a-8b14-621295e55a8d@opensynergy.com> @ 2022-04-20 8:07 ` Jason Wang [not found] ` <0a118236-bb98-9183-8be2-84f6b83e2581@opensynergy.com> 0 siblings, 1 reply; 9+ messages in thread From: Jason Wang @ 2022-04-20 8:07 UTC (permalink / raw) To: Maciej Szymański, Michael S. Tsirkin Cc: Willem de Bruijn, virtualization 在 2022/4/20 15:32, Maciej Szymański 写道: > On 20.04.2022 08:35, Michael S. Tsirkin wrote: >> On Wed, Apr 20, 2022 at 11:07:00AM +0800, Jason Wang wrote: >>> On Tue, Apr 19, 2022 at 11:03 PM Michael S. Tsirkin <mst@redhat.com> >>> wrote: >>>> On Tue, Apr 19, 2022 at 04:12:31PM +0200, Maciej Szymański wrote: >>>>> Hello, >>>>> >>>>> I've found a problem in virtio-net driver. >>>>> If virtio-net backend device advertises guest offload features, >>>>> there is >>>>> an unpermitted usage of control virtqueue before driver is >>>>> initialized. >>>>> According to VIRTIO specification 2.1.2 : >>>>> "The device MUST NOT consume buffers or send any used buffer >>>>> notifications to the driver before DRIVER_OK." >>>> Right. >>>> >>>>> During an initialization, driver calls register_netdevice which >>>>> invokes >>>>> callback function virtnet_set_features from __netdev_update_features. >>>>> If guest offload features are advertised by the device, >>>>> virtnet_set_guest_offloads is using virtnet_send_command to write and >>>>> read from VQ. >>>>> That leads to initialization stuck as device is not permitted yet >>>>> to use VQ. >>>> >>>> >>>> Hmm so we have this: >>>> >>>> >>>> if ((dev->features ^ features) & NETIF_F_GRO_HW) { >>>> if (vi->xdp_enabled) >>>> return -EBUSY; >>>> >>>> if (features & NETIF_F_GRO_HW) >>>> offloads = vi->guest_offloads_capable; >>>> else >>>> offloads = vi->guest_offloads_capable & >>>> ~GUEST_OFFLOAD_GRO_HW_MASK; >>>> >>>> err = virtnet_set_guest_offloads(vi, offloads); >>>> if (err) >>>> return err; >>>> vi->guest_offloads = offloads; >>>> } >>>> >>>> which I guess should have prevented virtnet_set_guest_offloads from >>>> ever running. >>>> >>>> From your description it sounds like you have observed this >>>> in practice, right? >>>> > Yes. I have proprietary virtio-net device which advertises following > guest offload features : > - VIRTIO_NET_F_GUEST_CSUM > - VIRTIO_NET_F_GUEST_TSO4 > - VIRTIO_NET_F_GUEST_TSO6 > - VIRTIO_NET_F_GUEST_UFO > > This feature set passes the condition in virtnet_set_features. > > When I disable guest offloads in my device - virtnet_set_guest_offloads > is not called and driver initialization completes successfully. >>>> >>>>> I have attached a patch for kernel 5.18-rc3 which fixes the >>>>> problem by >>>>> deferring feature set after virtio driver initialization. >>>>> >>>>> Best Regards, >>>>> >>>>> -- >>>>> Maciej Szymański >>>>> Senior Staff Engineer >>>>> >>>>> OpenSynergy GmbH >>>>> Rotherstr. 20, 10245 Berlin >>>>> >>>>> Phone: +49 30 60 98 54 0 -86 >>>>> Fax: +49 30 60 98 54 0 -99 >>>>> E-Mail: maciej.szymanski@opensynergy.com >>>>> >>>>> www.opensynergy.com >>>>> >>>>> Handelsregister/Commercial Registry: Amtsgericht Charlottenburg, >>>>> HRB 108616B >>>>> Geschäftsführer/Managing Director: Regis Adjamah >>>>> >>>>> Please mind our privacy >>>>> notice<https://www.opensynergy.com/datenschutzerklaerung/privacy-notice-for-business-partners-pursuant-to-article-13-of-the-general-data-protection-regulation-gdpr/> >>>>> pursuant to Art. 13 GDPR. // Unsere Hinweise zum Datenschutz gem. >>>>> Art. 13 DSGVO finden Sie >>>>> hier.<https://www.opensynergy.com/de/datenschutzerklaerung/datenschutzhinweise-fuer-geschaeftspartner-gem-art-13-dsgvo/> >>>>> diff --git a/drivers/net/virtio_net.c b/drivers/net/virtio_net.c >>>>> index 87838cb..a44462d 100644 >>>>> --- a/drivers/net/virtio_net.c >>>>> +++ b/drivers/net/virtio_net.c >>>>> @@ -264,6 +264,8 @@ struct virtnet_info { >>>>> unsigned long guest_offloads; >>>>> unsigned long guest_offloads_capable; >>>>> >>>>> + netdev_features_t features; >>>>> + >>>> I don't much like how we are forced to keep a copy of features >>>> here :( At least pls add a comment explaining what's going on, >>>> who owns this etc. > Personally I don't like this as well but I think it is only way to keep > the features for deferred set. >>>> >>>>> /* failover when STANDBY feature enabled */ >>>>> struct failover *failover; >>>>> }; >>>>> @@ -2976,6 +2978,15 @@ static int >>>>> virtnet_get_phys_port_name(struct net_device *dev, char *buf, >>>>> >>>>> static int virtnet_set_features(struct net_device *dev, >>>>> netdev_features_t features) >>>>> +{ >>>>> + struct virtnet_info *vi = netdev_priv(dev); >>>>> + vi->features = features; >>>>> + >>>>> + return 0; >>>>> +} >>>> >>>> Looks like this breaks changing features after initialization - >>>> these will never be propagated to hardware now. > Indeed. An original function won't be used. > >>> Yes, I think we need to have a check and only defer the setting when >>> virtio device is not ready. > Right. I have updated my patch to check status to deffer feature set. > That will also solve the problem from comment above. > >>> Thanks >> I think we should first understand how does the issue trigger, >> is this a theoretical or a practical issue. > As mentioned above - practical issue. It does not occur for f.e. QEMU > built-in virtio-net device as it does not advertise guest offload > features. Qemu has that support (enabled by default): DEFINE_PROP_BIT64("ctrl_guest_offloads", VirtIONet, host_features, VIRTIO_NET_F_CTRL_GUEST_OFFLOADS, true), But Qemu doesn't check DRIVER_OK when processing ctrl vq, that's why we never see any report before. Thanks > >>>>> + >>>>> +static int virtnet_set_features_deferred(struct net_device *dev, >>>>> + netdev_features_t features) >>>>> { >>>>> struct virtnet_info *vi = netdev_priv(dev); >>>>> u64 offloads; >>>>> @@ -3644,6 +3655,13 @@ static int virtnet_probe(struct >>>>> virtio_device *vdev) >>>>> >>>>> virtio_device_ready(vdev); >>>>> >>>>> + /* Deferred feature set after device ready */ >>>>> + err = virtnet_set_features_deferred(dev, vi->features); >>>> >>>> It seems that if this is called e.g. for a device without a CVQ and >>>> there are things that actually need to change then it will BUG_ON. >>>> >>>> >>>>> + if (err) { >>>>> + pr_debug("virtio_net: set features failed\n"); >>>>> + goto free_unregister_netdev; >>>>> + } >>>>> + >>>>> err = virtnet_cpu_notif_add(vi); >>>>> if (err) { >>>>> pr_debug("virtio_net: registering cpu notifier >>>>> failed\n"); >>>>> >>>> -- >>>> MST >>>> > > Please mind our privacy > notice<https://www.opensynergy.com/datenschutzerklaerung/privacy-notice-for-business-partners-pursuant-to-article-13-of-the-general-data-protection-regulation-gdpr/> > pursuant to Art. 13 GDPR. // Unsere Hinweise zum Datenschutz gem. Art. > 13 DSGVO finden Sie > hier.<https://www.opensynergy.com/de/datenschutzerklaerung/datenschutzhinweise-fuer-geschaeftspartner-gem-art-13-dsgvo/> _______________________________________________ Virtualization mailing list Virtualization@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/virtualization ^ permalink raw reply [flat|nested] 9+ messages in thread
[parent not found: <0a118236-bb98-9183-8be2-84f6b83e2581@opensynergy.com>]
* Re: virtio-net: Unpermitted usage of virtqueue before virtio driver initialization [not found] ` <0a118236-bb98-9183-8be2-84f6b83e2581@opensynergy.com> @ 2022-04-20 11:10 ` Michael S. Tsirkin [not found] ` <4050a523-85a8-0f3c-b7de-c371a42c8f6c@opensynergy.com> 0 siblings, 1 reply; 9+ messages in thread From: Michael S. Tsirkin @ 2022-04-20 11:10 UTC (permalink / raw) To: Maciej Szymański; +Cc: Willem de Bruijn, virtualization On Wed, Apr 20, 2022 at 10:17:27AM +0200, Maciej Szymański wrote: > > > > > > Hmm so we have this: > > > > > > > > > > > > > > > > > > if ((dev->features ^ features) & NETIF_F_GRO_HW) { > > > > > > if (vi->xdp_enabled) > > > > > > return -EBUSY; > > > > > > > > > > > > if (features & NETIF_F_GRO_HW) > > > > > > offloads = vi->guest_offloads_capable; > > > > > > else > > > > > > offloads = vi->guest_offloads_capable & > > > > > > ~GUEST_OFFLOAD_GRO_HW_MASK; > > > > > > > > > > > > err = virtnet_set_guest_offloads(vi, offloads); > > > > > > if (err) > > > > > > return err; > > > > > > vi->guest_offloads = offloads; > > > > > > } > > > > > > > > > > > > which I guess should have prevented virtnet_set_guest_offloads > > > > > > from ever running. > > > > > > > > > > > > From your description it sounds like you have observed this > > > > > > in practice, right? > > > > > > > > > Yes. I have proprietary virtio-net device which advertises following > > > guest offload features : > > > - VIRTIO_NET_F_GUEST_CSUM > > > - VIRTIO_NET_F_GUEST_TSO4 > > > - VIRTIO_NET_F_GUEST_TSO6 > > > - VIRTIO_NET_F_GUEST_UFO > > > > > > This feature set passes the condition in virtnet_set_features. So why isn't dev->features equal to features? -- MST _______________________________________________ Virtualization mailing list Virtualization@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/virtualization ^ permalink raw reply [flat|nested] 9+ messages in thread
[parent not found: <4050a523-85a8-0f3c-b7de-c371a42c8f6c@opensynergy.com>]
* Re: virtio-net: Unpermitted usage of virtqueue before virtio driver initialization [not found] ` <4050a523-85a8-0f3c-b7de-c371a42c8f6c@opensynergy.com> @ 2022-04-20 17:54 ` Michael S. Tsirkin [not found] ` <06dc4f89-1770-67cc-a843-6e956c0504dc@opensynergy.com> 0 siblings, 1 reply; 9+ messages in thread From: Michael S. Tsirkin @ 2022-04-20 17:54 UTC (permalink / raw) To: Maciej Szymański; +Cc: Willem de Bruijn, virtualization On Wed, Apr 20, 2022 at 04:58:51PM +0200, Maciej Szymański wrote: > On 20.04.2022 13:10, Michael S. Tsirkin wrote: > > On Wed, Apr 20, 2022 at 10:17:27AM +0200, Maciej Szymański wrote: > > > > > > > > Hmm so we have this: > > > > > > > > > > > > > > > > > > > > > > > > if ((dev->features ^ features) & NETIF_F_GRO_HW) { > > > > > > > > if (vi->xdp_enabled) > > > > > > > > return -EBUSY; > > > > > > > > > > > > > > > > if (features & NETIF_F_GRO_HW) > > > > > > > > offloads = vi->guest_offloads_capable; > > > > > > > > else > > > > > > > > offloads = vi->guest_offloads_capable & > > > > > > > > ~GUEST_OFFLOAD_GRO_HW_MASK; > > > > > > > > > > > > > > > > err = virtnet_set_guest_offloads(vi, offloads); > > > > > > > > if (err) > > > > > > > > return err; > > > > > > > > vi->guest_offloads = offloads; > > > > > > > > } > > > > > > > > > > > > > > > > which I guess should have prevented virtnet_set_guest_offloads > > > > > > > > from ever running. > > > > > > > > > > > > > > > > From your description it sounds like you have observed this > > > > > > > > in practice, right? > > > > > > > > > > > > > Yes. I have proprietary virtio-net device which advertises following > > > > > guest offload features : > > > > > - VIRTIO_NET_F_GUEST_CSUM > > > > > - VIRTIO_NET_F_GUEST_TSO4 > > > > > - VIRTIO_NET_F_GUEST_TSO6 > > > > > - VIRTIO_NET_F_GUEST_UFO > > > > > > > > > > This feature set passes the condition in virtnet_set_features. > > So why isn't dev->features equal to features? > > > I just double verified and found that my device advertises > VIRTIO_NET_GUEST_TSO4 and VIRTIO_NET_F_GUEST_TSO6 but not > VIRTIO_NET_F_GUEST_CSUM as mentioned before. So, your device is out of spec: VIRTIO_NET_F_GUEST_TSO4 Requires VIRTIO_NET_F_GUEST_CSUM. And The device MUST NOT offer a feature which requires another feature which was not offered. Is this a production device? Can it be fixed? > That leads to the following situation : > > in virtio_probe : > > if (virtio_has_feature(vdev, VIRTIO_NET_F_GUEST_CSUM)) > dev->features |= NETIF_F_RXCSUM; > if (virtio_has_feature(vdev, VIRTIO_NET_F_GUEST_TSO4) || > virtio_has_feature(vdev, VIRTIO_NET_F_GUEST_TSO6)) > dev->features |= NETIF_F_GRO_HW; > if (virtio_has_feature(vdev, VIRTIO_NET_F_CTRL_GUEST_OFFLOADS)) > dev->hw_features |= NETIF_F_GRO_HW; > > > while in netdev_fix_features : > > if (!(features & NETIF_F_RXCSUM)) { > /* NETIF_F_GRO_HW implies doing RXCSUM since every packet > * successfully merged by hardware must also have the > * checksum verified by hardware. If the user does not > * want to enable RXCSUM, logically, we should disable GRO_HW. > */ > if (features & NETIF_F_GRO_HW) { > netdev_dbg(dev, "Dropping NETIF_F_GRO_HW since no RXCSUM > feature.\n"); > features &= ~NETIF_F_GRO_HW; > } > } > > As result dev->features and features passed from > __netdev_update_features differs exactly in NETIF_F_GRO_HW bit. > > > Please mind our privacy notice<https://www.opensynergy.com/datenschutzerklaerung/privacy-notice-for-business-partners-pursuant-to-article-13-of-the-general-data-protection-regulation-gdpr/> pursuant to Art. 13 GDPR. // Unsere Hinweise zum Datenschutz gem. Art. 13 DSGVO finden Sie hier.<https://www.opensynergy.com/de/datenschutzerklaerung/datenschutzhinweise-fuer-geschaeftspartner-gem-art-13-dsgvo/> _______________________________________________ Virtualization mailing list Virtualization@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/virtualization ^ permalink raw reply [flat|nested] 9+ messages in thread
[parent not found: <06dc4f89-1770-67cc-a843-6e956c0504dc@opensynergy.com>]
* Re: virtio-net: Unpermitted usage of virtqueue before virtio driver initialization [not found] ` <06dc4f89-1770-67cc-a843-6e956c0504dc@opensynergy.com> @ 2022-04-20 20:17 ` Michael S. Tsirkin 2022-04-21 2:47 ` Jason Wang 0 siblings, 1 reply; 9+ messages in thread From: Michael S. Tsirkin @ 2022-04-20 20:17 UTC (permalink / raw) To: Maciej Szymański; +Cc: Willem de Bruijn, virtualization On Wed, Apr 20, 2022 at 08:57:18PM +0200, Maciej Szymański wrote: > On 20.04.2022 19:54, Michael S. Tsirkin wrote: > > On Wed, Apr 20, 2022 at 04:58:51PM +0200, Maciej Szymański wrote: > > > On 20.04.2022 13:10, Michael S. Tsirkin wrote: > > > > On Wed, Apr 20, 2022 at 10:17:27AM +0200, Maciej Szymański wrote: > > > > > > > > > > Hmm so we have this: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > if ((dev->features ^ features) & NETIF_F_GRO_HW) { > > > > > > > > > > if (vi->xdp_enabled) > > > > > > > > > > return -EBUSY; > > > > > > > > > > > > > > > > > > > > if (features & NETIF_F_GRO_HW) > > > > > > > > > > offloads = vi->guest_offloads_capable; > > > > > > > > > > else > > > > > > > > > > offloads = vi->guest_offloads_capable & > > > > > > > > > > ~GUEST_OFFLOAD_GRO_HW_MASK; > > > > > > > > > > > > > > > > > > > > err = virtnet_set_guest_offloads(vi, offloads); > > > > > > > > > > if (err) > > > > > > > > > > return err; > > > > > > > > > > vi->guest_offloads = offloads; > > > > > > > > > > } > > > > > > > > > > > > > > > > > > > > which I guess should have prevented virtnet_set_guest_offloads > > > > > > > > > > from ever running. > > > > > > > > > > > > > > > > > > > > From your description it sounds like you have observed this > > > > > > > > > > in practice, right? > > > > > > > > > > > > > > > > > Yes. I have proprietary virtio-net device which advertises following > > > > > > > guest offload features : > > > > > > > - VIRTIO_NET_F_GUEST_CSUM > > > > > > > - VIRTIO_NET_F_GUEST_TSO4 > > > > > > > - VIRTIO_NET_F_GUEST_TSO6 > > > > > > > - VIRTIO_NET_F_GUEST_UFO > > > > > > > > > > > > > > This feature set passes the condition in virtnet_set_features. > > > > So why isn't dev->features equal to features? > > > > > > > I just double verified and found that my device advertises > > > VIRTIO_NET_GUEST_TSO4 and VIRTIO_NET_F_GUEST_TSO6 but not > > > VIRTIO_NET_F_GUEST_CSUM as mentioned before. > > So, your device is out of spec: > > > > VIRTIO_NET_F_GUEST_TSO4 Requires VIRTIO_NET_F_GUEST_CSUM. > > > > And > > > > The device MUST NOT offer a feature which requires another feature which was not offered. > > > > > > Is this a production device? Can it be fixed? > The problem seems to be more complicated. In fact > VIRTIO_NET_F_GUEST_CSUM is offered by our device, but during feature > negotiation it is being dropped. > This most likely does not happen when we use MMIO, but for some reason > happens in QEMU for VHOST_USER + PCI. > I need to investigate this more deeply... I don't see where linux would drop it. I suspect it's dropped between QEMU and vhost user. I'd say let's fix it in the device first. We can next consider marking vqs broken before device is ready - Jason what do you think? Finally, we can add code to avoid acking dependent features if the feature they depend on has not been negotiated - doing so is also a spec violation after all. > > > > > > That leads to the following situation : > > > > > > in virtio_probe : > > > > > > if (virtio_has_feature(vdev, VIRTIO_NET_F_GUEST_CSUM)) > > > dev->features |= NETIF_F_RXCSUM; > > > if (virtio_has_feature(vdev, VIRTIO_NET_F_GUEST_TSO4) || > > > virtio_has_feature(vdev, VIRTIO_NET_F_GUEST_TSO6)) > > > dev->features |= NETIF_F_GRO_HW; > > > if (virtio_has_feature(vdev, VIRTIO_NET_F_CTRL_GUEST_OFFLOADS)) > > > dev->hw_features |= NETIF_F_GRO_HW; > > > > > > > > > while in netdev_fix_features : > > > > > > if (!(features & NETIF_F_RXCSUM)) { > > > /* NETIF_F_GRO_HW implies doing RXCSUM since every packet > > > * successfully merged by hardware must also have the > > > * checksum verified by hardware. If the user does not > > > * want to enable RXCSUM, logically, we should disable GRO_HW. > > > */ > > > if (features & NETIF_F_GRO_HW) { > > > netdev_dbg(dev, "Dropping NETIF_F_GRO_HW since no RXCSUM > > > feature.\n"); > > > features &= ~NETIF_F_GRO_HW; > > > } > > > } > > > > > > As result dev->features and features passed from > > > __netdev_update_features differs exactly in NETIF_F_GRO_HW bit. > > > > > > > > > Please mind our privacy notice<https://www.opensynergy.com/datenschutzerklaerung/privacy-notice-for-business-partners-pursuant-to-article-13-of-the-general-data-protection-regulation-gdpr/> pursuant to Art. 13 GDPR. // Unsere Hinweise zum Datenschutz gem. Art. 13 DSGVO finden Sie hier.<https://www.opensynergy.com/de/datenschutzerklaerung/datenschutzhinweise-fuer-geschaeftspartner-gem-art-13-dsgvo/> > > > Please mind our privacy notice<https://www.opensynergy.com/datenschutzerklaerung/privacy-notice-for-business-partners-pursuant-to-article-13-of-the-general-data-protection-regulation-gdpr/> pursuant to Art. 13 GDPR. // Unsere Hinweise zum Datenschutz gem. Art. 13 DSGVO finden Sie hier.<https://www.opensynergy.com/de/datenschutzerklaerung/datenschutzhinweise-fuer-geschaeftspartner-gem-art-13-dsgvo/> _______________________________________________ Virtualization mailing list Virtualization@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/virtualization ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: virtio-net: Unpermitted usage of virtqueue before virtio driver initialization 2022-04-20 20:17 ` Michael S. Tsirkin @ 2022-04-21 2:47 ` Jason Wang 0 siblings, 0 replies; 9+ messages in thread From: Jason Wang @ 2022-04-21 2:47 UTC (permalink / raw) To: Michael S. Tsirkin Cc: Willem de Bruijn, Maciej Szymański, virtualization On Thu, Apr 21, 2022 at 4:18 AM Michael S. Tsirkin <mst@redhat.com> wrote: > > On Wed, Apr 20, 2022 at 08:57:18PM +0200, Maciej Szymański wrote: > > On 20.04.2022 19:54, Michael S. Tsirkin wrote: > > > On Wed, Apr 20, 2022 at 04:58:51PM +0200, Maciej Szymański wrote: > > > > On 20.04.2022 13:10, Michael S. Tsirkin wrote: > > > > > On Wed, Apr 20, 2022 at 10:17:27AM +0200, Maciej Szymański wrote: > > > > > > > > > > > Hmm so we have this: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > if ((dev->features ^ features) & NETIF_F_GRO_HW) { > > > > > > > > > > > if (vi->xdp_enabled) > > > > > > > > > > > return -EBUSY; > > > > > > > > > > > > > > > > > > > > > > if (features & NETIF_F_GRO_HW) > > > > > > > > > > > offloads = vi->guest_offloads_capable; > > > > > > > > > > > else > > > > > > > > > > > offloads = vi->guest_offloads_capable & > > > > > > > > > > > ~GUEST_OFFLOAD_GRO_HW_MASK; > > > > > > > > > > > > > > > > > > > > > > err = virtnet_set_guest_offloads(vi, offloads); > > > > > > > > > > > if (err) > > > > > > > > > > > return err; > > > > > > > > > > > vi->guest_offloads = offloads; > > > > > > > > > > > } > > > > > > > > > > > > > > > > > > > > > > which I guess should have prevented virtnet_set_guest_offloads > > > > > > > > > > > from ever running. > > > > > > > > > > > > > > > > > > > > > > From your description it sounds like you have observed this > > > > > > > > > > > in practice, right? > > > > > > > > > > > > > > > > > > > Yes. I have proprietary virtio-net device which advertises following > > > > > > > > guest offload features : > > > > > > > > - VIRTIO_NET_F_GUEST_CSUM > > > > > > > > - VIRTIO_NET_F_GUEST_TSO4 > > > > > > > > - VIRTIO_NET_F_GUEST_TSO6 > > > > > > > > - VIRTIO_NET_F_GUEST_UFO > > > > > > > > > > > > > > > > This feature set passes the condition in virtnet_set_features. > > > > > So why isn't dev->features equal to features? > > > > > > > > > I just double verified and found that my device advertises > > > > VIRTIO_NET_GUEST_TSO4 and VIRTIO_NET_F_GUEST_TSO6 but not > > > > VIRTIO_NET_F_GUEST_CSUM as mentioned before. > > > So, your device is out of spec: > > > > > > VIRTIO_NET_F_GUEST_TSO4 Requires VIRTIO_NET_F_GUEST_CSUM. > > > > > > And > > > > > > The device MUST NOT offer a feature which requires another feature which was not offered. > > > > > > > > > Is this a production device? Can it be fixed? > > The problem seems to be more complicated. In fact > > VIRTIO_NET_F_GUEST_CSUM is offered by our device, but during feature > > negotiation it is being dropped. > > This most likely does not happen when we use MMIO, but for some reason > > happens in QEMU for VHOST_USER + PCI. > > I need to investigate this more deeply... > > > I don't see where linux would drop it. I suspect it's dropped between > QEMU and vhost user. I'd say let's fix it in the device first. > We can next consider marking vqs broken before device is ready - > Jason what do you think? Yes, we can do that. Thanks > Finally, we can add code to avoid acking dependent features > if the feature they depend on has not been negotiated - doing > so is also a spec violation after all. > > > > > > > > > > > That leads to the following situation : > > > > > > > > in virtio_probe : > > > > > > > > if (virtio_has_feature(vdev, VIRTIO_NET_F_GUEST_CSUM)) > > > > dev->features |= NETIF_F_RXCSUM; > > > > if (virtio_has_feature(vdev, VIRTIO_NET_F_GUEST_TSO4) || > > > > virtio_has_feature(vdev, VIRTIO_NET_F_GUEST_TSO6)) > > > > dev->features |= NETIF_F_GRO_HW; > > > > if (virtio_has_feature(vdev, VIRTIO_NET_F_CTRL_GUEST_OFFLOADS)) > > > > dev->hw_features |= NETIF_F_GRO_HW; > > > > > > > > > > > > while in netdev_fix_features : > > > > > > > > if (!(features & NETIF_F_RXCSUM)) { > > > > /* NETIF_F_GRO_HW implies doing RXCSUM since every packet > > > > * successfully merged by hardware must also have the > > > > * checksum verified by hardware. If the user does not > > > > * want to enable RXCSUM, logically, we should disable GRO_HW. > > > > */ > > > > if (features & NETIF_F_GRO_HW) { > > > > netdev_dbg(dev, "Dropping NETIF_F_GRO_HW since no RXCSUM > > > > feature.\n"); > > > > features &= ~NETIF_F_GRO_HW; > > > > } > > > > } > > > > > > > > As result dev->features and features passed from > > > > __netdev_update_features differs exactly in NETIF_F_GRO_HW bit. > > > > > > > > > > > > Please mind our privacy notice<https://www.opensynergy.com/datenschutzerklaerung/privacy-notice-for-business-partners-pursuant-to-article-13-of-the-general-data-protection-regulation-gdpr/> pursuant to Art. 13 GDPR. // Unsere Hinweise zum Datenschutz gem. Art. 13 DSGVO finden Sie hier.<https://www.opensynergy.com/de/datenschutzerklaerung/datenschutzhinweise-fuer-geschaeftspartner-gem-art-13-dsgvo/> > > > > > > Please mind our privacy notice<https://www.opensynergy.com/datenschutzerklaerung/privacy-notice-for-business-partners-pursuant-to-article-13-of-the-general-data-protection-regulation-gdpr/> pursuant to Art. 13 GDPR. // Unsere Hinweise zum Datenschutz gem. Art. 13 DSGVO finden Sie hier.<https://www.opensynergy.com/de/datenschutzerklaerung/datenschutzhinweise-fuer-geschaeftspartner-gem-art-13-dsgvo/> > _______________________________________________ Virtualization mailing list Virtualization@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/virtualization ^ permalink raw reply [flat|nested] 9+ messages in thread
end of thread, other threads:[~2022-04-21 2:48 UTC | newest] Thread overview: 9+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- [not found] <c0c17b91-747e-ab58-83e5-b6f7dfa55e75@opensynergy.com> 2022-04-19 15:03 ` virtio-net: Unpermitted usage of virtqueue before virtio driver initialization Michael S. Tsirkin 2022-04-20 3:07 ` Jason Wang 2022-04-20 6:35 ` Michael S. Tsirkin 2022-04-20 8:05 ` Jason Wang [not found] ` <4080d799-b42e-018a-8b14-621295e55a8d@opensynergy.com> 2022-04-20 8:07 ` Jason Wang [not found] ` <0a118236-bb98-9183-8be2-84f6b83e2581@opensynergy.com> 2022-04-20 11:10 ` Michael S. Tsirkin [not found] ` <4050a523-85a8-0f3c-b7de-c371a42c8f6c@opensynergy.com> 2022-04-20 17:54 ` Michael S. Tsirkin [not found] ` <06dc4f89-1770-67cc-a843-6e956c0504dc@opensynergy.com> 2022-04-20 20:17 ` Michael S. Tsirkin 2022-04-21 2:47 ` Jason Wang
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.