Re: virtio_balloon regression in 5.19-rc3

Re: virtio_balloon regression in 5.19-rc3

[Thorsten Leemhuis]

[TLDR: I'm adding this regression report to the list of tracked
regressions; all text from me you find below is based on a few templates
paragraphs you might have encountered already already in similar form.]

On 20.06.22 20:49, Ben Hutchings wrote:
> I've tested a 5.19-rc3 kernel on top of QEMU/KVM with machine type
> pc-q35-5.2.  It has a virtio balloon device defined in libvirt as:
> 
>     <memballoon model="virtio">
>       <address type="pci" domain="0x0000" bus="0x05" slot="0x00" function="0x0"/>
>     </memballoon>
> 
> but the virtio_balloon driver fails to bind to it:
> 
>     virtio_balloon virtio4: init_vqs: add stat_vq failed
>     virtio_balloon: probe of virtio4 failed with error -5
> 
> On a 5.18 kernel with similar configuration, it binds successfully.
> 
> I've attached the kernel config for 5.19-rc3.

CCing the regression mailing list, as it should be in the loop for all
regressions, as explained here:
https://www.kernel.org/doc/html/latest/admin-guide/reporting-issues.html

Thanks for the report. To be sure below issue doesn't fall through the
cracks unnoticed, I'm adding it to regzbot, my Linux kernel regression
tracking bot:

#regzbot ^introduced v5.18..v5.19-rc3
#regzbot ignore-activity

This isn't a regression? This issue or a fix for it are already
discussed somewhere else? It was fixed already? You want to clarify when
the regression started to happen? Or point out I got the title or
something else totally wrong? Then just reply -- ideally with also
telling regzbot about it, as explained here:
https://linux-regtracking.leemhuis.info/tracked-regression/

Reminder for developers: When fixing the issue, add 'Link:' tags
pointing to the report (the mail this one replies to), as explained for
in the Linux kernel's documentation; above webpage explains why this is
important for tracked regressions.

Ciao, Thorsten (wearing his 'the Linux kernel's regression tracker' hat)

P.S.: As the Linux kernel's regression tracker I deal with a lot of
reports and sometimes miss something important when writing mails like
this. If that's the case here, don't hesitate to tell me in a public
reply, it's in everyone's interest to set the public record straight.

Re: virtio_balloon regression in 5.19-rc3

[Ben Hutchings]

[-- Attachment #1: Type: text/plain, Size: 1503 bytes --]

On Tue, 2022-06-21 at 17:34 +0800, Jason Wang wrote:
> On Tue, Jun 21, 2022 at 5:24 PM David Hildenbrand <david@redhat.com> wrote:
> > 
> > On 20.06.22 20:49, Ben Hutchings wrote:
> > > I've tested a 5.19-rc3 kernel on top of QEMU/KVM with machine type
> > > pc-q35-5.2.  It has a virtio balloon device defined in libvirt as:
> > > 
> > >     <memballoon model="virtio">
> > >       <address type="pci" domain="0x0000" bus="0x05" slot="0x00" function="0x0"/>
> > >     </memballoon>
> > > 
> > > but the virtio_balloon driver fails to bind to it:
> > > 
> > >     virtio_balloon virtio4: init_vqs: add stat_vq failed
> > >     virtio_balloon: probe of virtio4 failed with error -5
> > > 
> > 
> > Hmm, I don't see any recent changes to drivers/virtio/virtio_balloon.c
> > 
> > virtqueue_add_outbuf() fails with -EIO if I'm not wrong. That's the
> > first call of virtqueue_add_outbuf() when virtio_balloon initializes.
> > 
> > 
> > Maybe something in generic virtio code changed?
> 
> Yes, we introduced the IRQ hardening. That could be the root cause and
> we've received lots of reports so we decide to disable it by default.
> 
> Ben, could you please try this patch: (and make sure
> CONFIG_VIRTIO_HARDEN_NOTIFICATION is not set)
> 
> https://lore.kernel.org/lkml/20220620024158.2505-1-jasowang@redhat.com/T/

Yes, that patch fixes the regression for me.

Ben.

-- 
Ben Hutchings
Any smoothly functioning technology is indistinguishable
from a rigged demo.

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: virtio_balloon regression in 5.19-rc3

[Michael S. Tsirkin]

On Tue, Jun 21, 2022 at 06:10:00PM +0200, Ben Hutchings wrote:
> On Tue, 2022-06-21 at 17:34 +0800, Jason Wang wrote:
> > On Tue, Jun 21, 2022 at 5:24 PM David Hildenbrand <david@redhat.com> wrote:
> > > 
> > > On 20.06.22 20:49, Ben Hutchings wrote:
> > > > I've tested a 5.19-rc3 kernel on top of QEMU/KVM with machine type
> > > > pc-q35-5.2.  It has a virtio balloon device defined in libvirt as:
> > > > 
> > > >     <memballoon model="virtio">
> > > >       <address type="pci" domain="0x0000" bus="0x05" slot="0x00" function="0x0"/>
> > > >     </memballoon>
> > > > 
> > > > but the virtio_balloon driver fails to bind to it:
> > > > 
> > > >     virtio_balloon virtio4: init_vqs: add stat_vq failed
> > > >     virtio_balloon: probe of virtio4 failed with error -5
> > > > 
> > > 
> > > Hmm, I don't see any recent changes to drivers/virtio/virtio_balloon.c
> > > 
> > > virtqueue_add_outbuf() fails with -EIO if I'm not wrong. That's the
> > > first call of virtqueue_add_outbuf() when virtio_balloon initializes.
> > > 
> > > 
> > > Maybe something in generic virtio code changed?
> > 
> > Yes, we introduced the IRQ hardening. That could be the root cause and
> > we've received lots of reports so we decide to disable it by default.
> > 
> > Ben, could you please try this patch: (and make sure
> > CONFIG_VIRTIO_HARDEN_NOTIFICATION is not set)
> > 
> > https://lore.kernel.org/lkml/20220620024158.2505-1-jasowang@redhat.com/T/
> 
> Yes, that patch fixes the regression for me.
> 
> Ben.


Jason are you going to fix balloon to call device_ready before
registering device with linux?
> -- 
> Ben Hutchings
> Any smoothly functioning technology is indistinguishable
> from a rigged demo.

Re: virtio_balloon regression in 5.19-rc3

[Jason Wang]

On Fri, Jun 24, 2022 at 5:14 PM Michael S. Tsirkin <mst@redhat.com> wrote:
>
> On Tue, Jun 21, 2022 at 06:10:00PM +0200, Ben Hutchings wrote:
> > On Tue, 2022-06-21 at 17:34 +0800, Jason Wang wrote:
> > > On Tue, Jun 21, 2022 at 5:24 PM David Hildenbrand <david@redhat.com> wrote:
> > > >
> > > > On 20.06.22 20:49, Ben Hutchings wrote:
> > > > > I've tested a 5.19-rc3 kernel on top of QEMU/KVM with machine type
> > > > > pc-q35-5.2.  It has a virtio balloon device defined in libvirt as:
> > > > >
> > > > >     <memballoon model="virtio">
> > > > >       <address type="pci" domain="0x0000" bus="0x05" slot="0x00" function="0x0"/>
> > > > >     </memballoon>
> > > > >
> > > > > but the virtio_balloon driver fails to bind to it:
> > > > >
> > > > >     virtio_balloon virtio4: init_vqs: add stat_vq failed
> > > > >     virtio_balloon: probe of virtio4 failed with error -5
> > > > >
> > > >
> > > > Hmm, I don't see any recent changes to drivers/virtio/virtio_balloon.c
> > > >
> > > > virtqueue_add_outbuf() fails with -EIO if I'm not wrong. That's the
> > > > first call of virtqueue_add_outbuf() when virtio_balloon initializes.
> > > >
> > > >
> > > > Maybe something in generic virtio code changed?
> > >
> > > Yes, we introduced the IRQ hardening. That could be the root cause and
> > > we've received lots of reports so we decide to disable it by default.
> > >
> > > Ben, could you please try this patch: (and make sure
> > > CONFIG_VIRTIO_HARDEN_NOTIFICATION is not set)
> > >
> > > https://lore.kernel.org/lkml/20220620024158.2505-1-jasowang@redhat.com/T/
> >
> > Yes, that patch fixes the regression for me.
> >
> > Ben.
>
>
> Jason are you going to fix balloon to call device_ready before
> registering device with linux?

I'm working on the fix, (spot various bugs during review).

Thanks

> > --
> > Ben Hutchings
> > Any smoothly functioning technology is indistinguishable
> > from a rigged demo.
>
>

Re: virtio_balloon regression in 5.19-rc3 #forregzbot

[Thorsten Leemhuis]

TWIMC: this mail is primarily send for documentation purposes and for
regzbot, my Linux kernel regression tracking bot. These mails usually
contain '#forregzbot' in the subject, to make them easy to spot and filter.

On 21.06.22 11:35, Thorsten Leemhuis wrote:
> [TLDR: I'm adding this regression report to the list of tracked
> regressions; all text from me you find below is based on a few templates
> paragraphs you might have encountered already already in similar form.]
> 
> On 20.06.22 20:49, Ben Hutchings wrote:
>> I've tested a 5.19-rc3 kernel on top of QEMU/KVM with machine type
>> pc-q35-5.2.  It has a virtio balloon device defined in libvirt as:
>>
>>     <memballoon model="virtio">
>>       <address type="pci" domain="0x0000" bus="0x05" slot="0x00" function="0x0"/>
>>     </memballoon>
>>
>> but the virtio_balloon driver fails to bind to it:
>>
>>     virtio_balloon virtio4: init_vqs: add stat_vq failed
>>     virtio_balloon: probe of virtio4 failed with error -5
>>
> [...]
> #regzbot ^introduced v5.18..v5.19-rc3
> #regzbot ignore-activity

#regzbot introduced 8b4ec69d7e09
#regzbot monitor
https://lore.kernel.org/all/20220622012940.21441-1-jasowang@redhat.com/

Re: virtio_balloon regression in 5.19-rc3 #forregzbot

[Thorsten Leemhuis]

On 04.07.22 11:40, Thorsten Leemhuis wrote:
> TWIMC: this mail is primarily send for documentation purposes and for
> regzbot, my Linux kernel regression tracking bot. These mails usually
> contain '#forregzbot' in the subject, to make them easy to spot and filter.
> 
> On 21.06.22 11:35, Thorsten Leemhuis wrote:
>> [TLDR: I'm adding this regression report to the list of tracked
>> regressions; all text from me you find below is based on a few templates
>> paragraphs you might have encountered already already in similar form.]
>>
>> On 20.06.22 20:49, Ben Hutchings wrote:
>>> I've tested a 5.19-rc3 kernel on top of QEMU/KVM with machine type
>>> pc-q35-5.2.  It has a virtio balloon device defined in libvirt as:
>>>
>>>     <memballoon model="virtio">
>>>       <address type="pci" domain="0x0000" bus="0x05" slot="0x00" function="0x0"/>
>>>     </memballoon>
>>>
>>> but the virtio_balloon driver fails to bind to it:
>>>
>>>     virtio_balloon virtio4: init_vqs: add stat_vq failed
>>>     virtio_balloon: probe of virtio4 failed with error -5
>>>
>> [...]
>> #regzbot ^introduced v5.18..v5.19-rc3
>> #regzbot ignore-activity
> 
> #regzbot introduced 8b4ec69d7e09
> #regzbot monitor
> https://lore.kernel.org/all/20220622012940.21441-1-jasowang@redhat.com/

#regzbot fixed-by: 6a9720576c
#regzbot ignore-activity

For details see:
https://lore.kernel.org/all/CACGkMEu8EECPamY__OqQNf7iuKU7nho_-mij2zWULfV2rV+d7A@mail.gmail.com/

Ciao, Thorsten (wearing his 'the Linux kernel's regression tracker' hat)

P.S.: As the Linux kernel's regression tracker I deal with a lot of
reports and sometimes miss something important when writing mails like
this. If that's the case here, don't hesitate to tell me in a public
reply, it's in everyone's interest to set the public record straight.

Re: virtio_balloon regression in 5.19-rc3 #forregzbot

[Thorsten Leemhuis]

On 10.07.22 10:06, Thorsten Leemhuis wrote:
> On 04.07.22 11:40, Thorsten Leemhuis wrote:
>> TWIMC: this mail is primarily send for documentation purposes and for
>> regzbot, my Linux kernel regression tracking bot. These mails usually
>> contain '#forregzbot' in the subject, to make them easy to spot and filter.
>>
>> On 21.06.22 11:35, Thorsten Leemhuis wrote:
>>> On 20.06.22 20:49, Ben Hutchings wrote:
>>>> I've tested a 5.19-rc3 kernel on top of QEMU/KVM with machine type
>>>> pc-q35-5.2.  It has a virtio balloon device defined in libvirt as:
>>>>
>>>>     <memballoon model="virtio">
>>>>       <address type="pci" domain="0x0000" bus="0x05" slot="0x00" function="0x0"/>
>>>>     </memballoon>
>>>>
>>>> but the virtio_balloon driver fails to bind to it:
>>>>
>>>>     virtio_balloon virtio4: init_vqs: add stat_vq failed
>>>>     virtio_balloon: probe of virtio4 failed with error -5
>>>>
>>> [...]
>>> #regzbot ^introduced v5.18..v5.19-rc3
>>> #regzbot ignore-activity
>>
>> #regzbot introduced 8b4ec69d7e09
>> #regzbot monitor
>> https://lore.kernel.org/all/20220622012940.21441-1-jasowang@redhat.com/
> 
> #regzbot fixed-by: 6a9720576c
> #regzbot ignore-activity

For the record: the fix was merged through a different branch and thus
got a different commit id:

#regzbot fixed-by: ebe797f25f68f28581f46a9cb9c1997ac15c39a0