vhost changes (batched) in linux-next after 12/13 trigger random crashes in KVM guests after reboot

vhost changes (batched) in linux-next after 12/13 trigger random crashes in KVM guests after reboot

[Christian Borntraeger]

Michael,

with 
commit db7286b100b503ef80612884453bed53d74c9a16 (refs/bisect/skip-db7286b100b503ef80612884453bed53d74c9a16)
    vhost: use batched version by default
plus
commit 6bd262d5eafcdf8cdfae491e2e748e4e434dcda6 (HEAD, refs/bisect/bad)
    Revert "vhost/net: add an option to test new code"
to make things compile (your next tree is not easily bisectable, can you fix that as well?).

I get random crashes in my s390 KVM guests after reboot.
Reverting both patches together with commit decd9b8 "vhost: use vhost_desc instead of vhost_log" to
make it compile again) on top of linux-next-1218 makes the problem go away.

Looks like the batched version is not yet ready for prime time. Can you drop these patches until
we have fixed the issues?

Christian

Re: vhost changes (batched) in linux-next after 12/13 trigger random crashes in KVM guests after reboot

[Michael S. Tsirkin]

On Wed, Dec 18, 2019 at 03:43:43PM +0100, Christian Borntraeger wrote:
> Michael,
> 
> with 
> commit db7286b100b503ef80612884453bed53d74c9a16 (refs/bisect/skip-db7286b100b503ef80612884453bed53d74c9a16)
>     vhost: use batched version by default
> plus
> commit 6bd262d5eafcdf8cdfae491e2e748e4e434dcda6 (HEAD, refs/bisect/bad)
>     Revert "vhost/net: add an option to test new code"
> to make things compile (your next tree is not easily bisectable, can you fix that as well?).

I'll try.

> 
> I get random crashes in my s390 KVM guests after reboot.
> Reverting both patches together with commit decd9b8 "vhost: use vhost_desc instead of vhost_log" to
> make it compile again) on top of linux-next-1218 makes the problem go away.
> 
> Looks like the batched version is not yet ready for prime time. Can you drop these patches until
> we have fixed the issues?
> 
> Christian
> 

Will do, thanks for letting me know.

-- 
MST

Re: vhost changes (batched) in linux-next after 12/13 trigger random crashes in KVM guests after reboot

[Christian Borntraeger]

On 18.12.19 16:10, Michael S. Tsirkin wrote:
> On Wed, Dec 18, 2019 at 03:43:43PM +0100, Christian Borntraeger wrote:
>> Michael,
>>
>> with 
>> commit db7286b100b503ef80612884453bed53d74c9a16 (refs/bisect/skip-db7286b100b503ef80612884453bed53d74c9a16)
>>     vhost: use batched version by default
>> plus
>> commit 6bd262d5eafcdf8cdfae491e2e748e4e434dcda6 (HEAD, refs/bisect/bad)
>>     Revert "vhost/net: add an option to test new code"
>> to make things compile (your next tree is not easily bisectable, can you fix that as well?).
> 
> I'll try.
> 
>>
>> I get random crashes in my s390 KVM guests after reboot.
>> Reverting both patches together with commit decd9b8 "vhost: use vhost_desc instead of vhost_log" to
>> make it compile again) on top of linux-next-1218 makes the problem go away.
>>
>> Looks like the batched version is not yet ready for prime time. Can you drop these patches until
>> we have fixed the issues?
>>
>> Christian
>>
> 
> Will do, thanks for letting me know.

I have confirmed with the initial reporter (internal test team) that <driver name='qemu'/> 
with a known to be broken linux next kernel also fixes the problem, so it is really the
vhost changes.

Re: vhost changes (batched) in linux-next after 12/13 trigger random crashes in KVM guests after reboot

[Michael S. Tsirkin]

On Wed, Dec 18, 2019 at 04:59:02PM +0100, Christian Borntraeger wrote:
> On 18.12.19 16:10, Michael S. Tsirkin wrote:
> > On Wed, Dec 18, 2019 at 03:43:43PM +0100, Christian Borntraeger wrote:
> >> Michael,
> >>
> >> with 
> >> commit db7286b100b503ef80612884453bed53d74c9a16 (refs/bisect/skip-db7286b100b503ef80612884453bed53d74c9a16)
> >>     vhost: use batched version by default
> >> plus
> >> commit 6bd262d5eafcdf8cdfae491e2e748e4e434dcda6 (HEAD, refs/bisect/bad)
> >>     Revert "vhost/net: add an option to test new code"
> >> to make things compile (your next tree is not easily bisectable, can you fix that as well?).
> > 
> > I'll try.
> > 
> >>
> >> I get random crashes in my s390 KVM guests after reboot.
> >> Reverting both patches together with commit decd9b8 "vhost: use vhost_desc instead of vhost_log" to
> >> make it compile again) on top of linux-next-1218 makes the problem go away.
> >>
> >> Looks like the batched version is not yet ready for prime time. Can you drop these patches until
> >> we have fixed the issues?
> >>
> >> Christian
> >>
> > 
> > Will do, thanks for letting me know.
> 
> I have confirmed with the initial reporter (internal test team) that <driver name='qemu'/> 
> with a known to be broken linux next kernel also fixes the problem, so it is really the
> vhost changes.

OK I'm back and trying to make it more bisectable.

I pushed a new tag "batch-v2".
It's same code but with this bisect should get more information.


I suspect one of the following:

commit 1414d7ee3d10d2ec2bc4ee652d1d90ec91da1c79
Author: Michael S. Tsirkin <mst@redhat.com>
Date:   Mon Oct 7 06:11:18 2019 -0400

    vhost: batching fetches
    
    With this patch applied, new and old code perform identically.
    
    Lots of extra optimizations are now possible, e.g.
    we can fetch multiple heads with copy_from/to_user now.
    We can get rid of maintaining the log array.  Etc etc.
    
    Signed-off-by: Michael S. Tsirkin <mst@redhat.com>

commit 50297a8480b439efc5f3f23088cb2d90b799acef
Author: Michael S. Tsirkin <mst@redhat.com>
Date:   Wed Dec 11 12:19:26 2019 -0500

    vhost: use batched version by default
    
    As testing shows no performance change, switch to that now.
    
    Signed-off-by: Michael S. Tsirkin <mst@redhat.com>


and would like to know which.

Thanks!

Re: vhost changes (batched) in linux-next after 12/13 trigger random crashes in KVM guests after reboot

[Christian Borntraeger]

On 06.01.20 11:50, Michael S. Tsirkin wrote:
> On Wed, Dec 18, 2019 at 04:59:02PM +0100, Christian Borntraeger wrote:
>> On 18.12.19 16:10, Michael S. Tsirkin wrote:
>>> On Wed, Dec 18, 2019 at 03:43:43PM +0100, Christian Borntraeger wrote:
>>>> Michael,
>>>>
>>>> with 
>>>> commit db7286b100b503ef80612884453bed53d74c9a16 (refs/bisect/skip-db7286b100b503ef80612884453bed53d74c9a16)
>>>>     vhost: use batched version by default
>>>> plus
>>>> commit 6bd262d5eafcdf8cdfae491e2e748e4e434dcda6 (HEAD, refs/bisect/bad)
>>>>     Revert "vhost/net: add an option to test new code"
>>>> to make things compile (your next tree is not easily bisectable, can you fix that as well?).
>>>
>>> I'll try.
>>>
>>>>
>>>> I get random crashes in my s390 KVM guests after reboot.
>>>> Reverting both patches together with commit decd9b8 "vhost: use vhost_desc instead of vhost_log" to
>>>> make it compile again) on top of linux-next-1218 makes the problem go away.
>>>>
>>>> Looks like the batched version is not yet ready for prime time. Can you drop these patches until
>>>> we have fixed the issues?
>>>>
>>>> Christian
>>>>
>>>
>>> Will do, thanks for letting me know.
>>
>> I have confirmed with the initial reporter (internal test team) that <driver name='qemu'/> 
>> with a known to be broken linux next kernel also fixes the problem, so it is really the
>> vhost changes.
> 
> OK I'm back and trying to make it more bisectable.
> 
> I pushed a new tag "batch-v2".
> It's same code but with this bisect should get more information.

I get the following with this tag

drivers/vhost/net.c: In function ‘vhost_net_tx_get_vq_desc’:
drivers/vhost/net.c:574:7: error: implicit declaration of function ‘vhost_get_vq_desc_batch’; did you mean ‘vhost_get_vq_desc’? [-Werror=implicit-function-declaration]
  574 |   r = vhost_get_vq_desc_batch(tvq, tvq->iov, ARRAY_SIZE(tvq->iov),
      |       ^~~~~~~~~~~~~~~~~~~~~~~
      |       vhost_get_vq_desc

Re: vhost changes (batched) in linux-next after 12/13 trigger random crashes in KVM guests after reboot

[Michael S. Tsirkin]

On Tue, Jan 07, 2020 at 09:59:16AM +0100, Christian Borntraeger wrote:
> 
> 
> On 06.01.20 11:50, Michael S. Tsirkin wrote:
> > On Wed, Dec 18, 2019 at 04:59:02PM +0100, Christian Borntraeger wrote:
> >> On 18.12.19 16:10, Michael S. Tsirkin wrote:
> >>> On Wed, Dec 18, 2019 at 03:43:43PM +0100, Christian Borntraeger wrote:
> >>>> Michael,
> >>>>
> >>>> with 
> >>>> commit db7286b100b503ef80612884453bed53d74c9a16 (refs/bisect/skip-db7286b100b503ef80612884453bed53d74c9a16)
> >>>>     vhost: use batched version by default
> >>>> plus
> >>>> commit 6bd262d5eafcdf8cdfae491e2e748e4e434dcda6 (HEAD, refs/bisect/bad)
> >>>>     Revert "vhost/net: add an option to test new code"
> >>>> to make things compile (your next tree is not easily bisectable, can you fix that as well?).
> >>>
> >>> I'll try.
> >>>
> >>>>
> >>>> I get random crashes in my s390 KVM guests after reboot.
> >>>> Reverting both patches together with commit decd9b8 "vhost: use vhost_desc instead of vhost_log" to
> >>>> make it compile again) on top of linux-next-1218 makes the problem go away.
> >>>>
> >>>> Looks like the batched version is not yet ready for prime time. Can you drop these patches until
> >>>> we have fixed the issues?
> >>>>
> >>>> Christian
> >>>>
> >>>
> >>> Will do, thanks for letting me know.
> >>
> >> I have confirmed with the initial reporter (internal test team) that <driver name='qemu'/> 
> >> with a known to be broken linux next kernel also fixes the problem, so it is really the
> >> vhost changes.
> > 
> > OK I'm back and trying to make it more bisectable.
> > 
> > I pushed a new tag "batch-v2".
> > It's same code but with this bisect should get more information.
> 
> I get the following with this tag
> 
> drivers/vhost/net.c: In function ‘vhost_net_tx_get_vq_desc’:
> drivers/vhost/net.c:574:7: error: implicit declaration of function ‘vhost_get_vq_desc_batch’; did you mean ‘vhost_get_vq_desc’? [-Werror=implicit-function-declaration]
>   574 |   r = vhost_get_vq_desc_batch(tvq, tvq->iov, ARRAY_SIZE(tvq->iov),
>       |       ^~~~~~~~~~~~~~~~~~~~~~~
>       |       vhost_get_vq_desc
> 

Not sure why but I pushed a wrong commit. Sorry. Should be good now.

-- 
MST

Re: vhost changes (batched) in linux-next after 12/13 trigger random crashes in KVM guests after reboot

[Christian Borntraeger]

On 07.01.20 10:39, Michael S. Tsirkin wrote:
> On Tue, Jan 07, 2020 at 09:59:16AM +0100, Christian Borntraeger wrote:
>>
>>
>> On 06.01.20 11:50, Michael S. Tsirkin wrote:
>>> On Wed, Dec 18, 2019 at 04:59:02PM +0100, Christian Borntraeger wrote:
>>>> On 18.12.19 16:10, Michael S. Tsirkin wrote:
>>>>> On Wed, Dec 18, 2019 at 03:43:43PM +0100, Christian Borntraeger wrote:
>>>>>> Michael,
>>>>>>
>>>>>> with 
>>>>>> commit db7286b100b503ef80612884453bed53d74c9a16 (refs/bisect/skip-db7286b100b503ef80612884453bed53d74c9a16)
>>>>>>     vhost: use batched version by default
>>>>>> plus
>>>>>> commit 6bd262d5eafcdf8cdfae491e2e748e4e434dcda6 (HEAD, refs/bisect/bad)
>>>>>>     Revert "vhost/net: add an option to test new code"
>>>>>> to make things compile (your next tree is not easily bisectable, can you fix that as well?).
>>>>>
>>>>> I'll try.
>>>>>
>>>>>>
>>>>>> I get random crashes in my s390 KVM guests after reboot.
>>>>>> Reverting both patches together with commit decd9b8 "vhost: use vhost_desc instead of vhost_log" to
>>>>>> make it compile again) on top of linux-next-1218 makes the problem go away.
>>>>>>
>>>>>> Looks like the batched version is not yet ready for prime time. Can you drop these patches until
>>>>>> we have fixed the issues?
>>>>>>
>>>>>> Christian
>>>>>>
>>>>>
>>>>> Will do, thanks for letting me know.
>>>>
>>>> I have confirmed with the initial reporter (internal test team) that <driver name='qemu'/> 
>>>> with a known to be broken linux next kernel also fixes the problem, so it is really the
>>>> vhost changes.
>>>
>>> OK I'm back and trying to make it more bisectable.
>>>
>>> I pushed a new tag "batch-v2".
>>> It's same code but with this bisect should get more information.
>>
>> I get the following with this tag
>>
>> drivers/vhost/net.c: In function ‘vhost_net_tx_get_vq_desc’:
>> drivers/vhost/net.c:574:7: error: implicit declaration of function ‘vhost_get_vq_desc_batch’; did you mean ‘vhost_get_vq_desc’? [-Werror=implicit-function-declaration]
>>   574 |   r = vhost_get_vq_desc_batch(tvq, tvq->iov, ARRAY_SIZE(tvq->iov),
>>       |       ^~~~~~~~~~~~~~~~~~~~~~~
>>       |       vhost_get_vq_desc
>>
> 
> Not sure why but I pushed a wrong commit. Sorry. Should be good now.
> 

during bisect:

drivers/vhost/vhost.c: In function ‘vhost_get_vq_desc_batch’:
drivers/vhost/vhost.c:2634:8: error: ‘id’ undeclared (first use in this function); did you mean ‘i’?
 2634 |  ret = id;
      |        ^~
      |        i

I changed that to i


The last step then gave me  (on commit 50297a8480b439efc5f3f23088cb2d90b799acef vhost: use batched version by default)
net enc1: Unexpected TXQ (0) queue failure: -5
in the guest.

bisect log so far:
[cborntra@m83lp52 linux]$ git bisect log
git bisect start
# bad: [3131e79bb9e9892a5a6bd33513de9bc90b20e867] vhost: use vhost_desc instead of vhost_log
git bisect bad 3131e79bb9e9892a5a6bd33513de9bc90b20e867
# good: [d1281e3a562ec6a08f944a876481dd043ba739b9] virtio-blk: remove VIRTIO_BLK_F_SCSI support
git bisect good d1281e3a562ec6a08f944a876481dd043ba739b9
# good: [5b00aab5b6332a67e32dace1dcd3a198ab94ed56] vhost: option to fetch descriptors through an independent struct
git bisect good 5b00aab5b6332a67e32dace1dcd3a198ab94ed56
# good: [5b00aab5b6332a67e32dace1dcd3a198ab94ed56] vhost: option to fetch descriptors through an independent struct
git bisect good 5b00aab5b6332a67e32dace1dcd3a198ab94ed56
# bad: [1414d7ee3d10d2ec2bc4ee652d1d90ec91da1c79] vhost: batching fetches
git bisect bad 1414d7ee3d10d2ec2bc4ee652d1d90ec91da1c79

Re: vhost changes (batched) in linux-next after 12/13 trigger random crashes in KVM guests after reboot

[Michael S. Tsirkin]

On Tue, Jan 07, 2020 at 12:34:50PM +0100, Christian Borntraeger wrote:
> 
> 
> On 07.01.20 10:39, Michael S. Tsirkin wrote:
> > On Tue, Jan 07, 2020 at 09:59:16AM +0100, Christian Borntraeger wrote:
> >>
> >>
> >> On 06.01.20 11:50, Michael S. Tsirkin wrote:
> >>> On Wed, Dec 18, 2019 at 04:59:02PM +0100, Christian Borntraeger wrote:
> >>>> On 18.12.19 16:10, Michael S. Tsirkin wrote:
> >>>>> On Wed, Dec 18, 2019 at 03:43:43PM +0100, Christian Borntraeger wrote:
> >>>>>> Michael,
> >>>>>>
> >>>>>> with 
> >>>>>> commit db7286b100b503ef80612884453bed53d74c9a16 (refs/bisect/skip-db7286b100b503ef80612884453bed53d74c9a16)
> >>>>>>     vhost: use batched version by default
> >>>>>> plus
> >>>>>> commit 6bd262d5eafcdf8cdfae491e2e748e4e434dcda6 (HEAD, refs/bisect/bad)
> >>>>>>     Revert "vhost/net: add an option to test new code"
> >>>>>> to make things compile (your next tree is not easily bisectable, can you fix that as well?).
> >>>>>
> >>>>> I'll try.
> >>>>>
> >>>>>>
> >>>>>> I get random crashes in my s390 KVM guests after reboot.
> >>>>>> Reverting both patches together with commit decd9b8 "vhost: use vhost_desc instead of vhost_log" to
> >>>>>> make it compile again) on top of linux-next-1218 makes the problem go away.
> >>>>>>
> >>>>>> Looks like the batched version is not yet ready for prime time. Can you drop these patches until
> >>>>>> we have fixed the issues?
> >>>>>>
> >>>>>> Christian
> >>>>>>
> >>>>>
> >>>>> Will do, thanks for letting me know.
> >>>>
> >>>> I have confirmed with the initial reporter (internal test team) that <driver name='qemu'/> 
> >>>> with a known to be broken linux next kernel also fixes the problem, so it is really the
> >>>> vhost changes.
> >>>
> >>> OK I'm back and trying to make it more bisectable.
> >>>
> >>> I pushed a new tag "batch-v2".
> >>> It's same code but with this bisect should get more information.
> >>
> >> I get the following with this tag
> >>
> >> drivers/vhost/net.c: In function ‘vhost_net_tx_get_vq_desc’:
> >> drivers/vhost/net.c:574:7: error: implicit declaration of function ‘vhost_get_vq_desc_batch’; did you mean ‘vhost_get_vq_desc’? [-Werror=implicit-function-declaration]
> >>   574 |   r = vhost_get_vq_desc_batch(tvq, tvq->iov, ARRAY_SIZE(tvq->iov),
> >>       |       ^~~~~~~~~~~~~~~~~~~~~~~
> >>       |       vhost_get_vq_desc
> >>
> > 
> > Not sure why but I pushed a wrong commit. Sorry. Should be good now.
> > 
> 
> during bisect:
> 
> drivers/vhost/vhost.c: In function ‘vhost_get_vq_desc_batch’:
> drivers/vhost/vhost.c:2634:8: error: ‘id’ undeclared (first use in this function); did you mean ‘i’?
>  2634 |  ret = id;
>       |        ^~
>       |        i
> 
> I changed that to i

Hmm no that's wrong I think. Sorry about all the errors. Let me push a
fixed v3.

> 
> The last step then gave me  (on commit 50297a8480b439efc5f3f23088cb2d90b799acef vhost: use batched version by default)
> net enc1: Unexpected TXQ (0) queue failure: -5
> in the guest.
> 
> bisect log so far:
> [cborntra@m83lp52 linux]$ git bisect log
> git bisect start
> # bad: [3131e79bb9e9892a5a6bd33513de9bc90b20e867] vhost: use vhost_desc instead of vhost_log
> git bisect bad 3131e79bb9e9892a5a6bd33513de9bc90b20e867
> # good: [d1281e3a562ec6a08f944a876481dd043ba739b9] virtio-blk: remove VIRTIO_BLK_F_SCSI support
> git bisect good d1281e3a562ec6a08f944a876481dd043ba739b9
> # good: [5b00aab5b6332a67e32dace1dcd3a198ab94ed56] vhost: option to fetch descriptors through an independent struct
> git bisect good 5b00aab5b6332a67e32dace1dcd3a198ab94ed56
> # good: [5b00aab5b6332a67e32dace1dcd3a198ab94ed56] vhost: option to fetch descriptors through an independent struct
> git bisect good 5b00aab5b6332a67e32dace1dcd3a198ab94ed56
> # bad: [1414d7ee3d10d2ec2bc4ee652d1d90ec91da1c79] vhost: batching fetches
> git bisect bad 1414d7ee3d10d2ec2bc4ee652d1d90ec91da1c79
> 
> 
> 

Re: vhost changes (batched) in linux-next after 12/13 trigger random crashes in KVM guests after reboot

[Michael S. Tsirkin]

On Tue, Jan 07, 2020 at 12:34:50PM +0100, Christian Borntraeger wrote:
> 
> 
> On 07.01.20 10:39, Michael S. Tsirkin wrote:
> > On Tue, Jan 07, 2020 at 09:59:16AM +0100, Christian Borntraeger wrote:
> >>
> >>
> >> On 06.01.20 11:50, Michael S. Tsirkin wrote:
> >>> On Wed, Dec 18, 2019 at 04:59:02PM +0100, Christian Borntraeger wrote:
> >>>> On 18.12.19 16:10, Michael S. Tsirkin wrote:
> >>>>> On Wed, Dec 18, 2019 at 03:43:43PM +0100, Christian Borntraeger wrote:
> >>>>>> Michael,
> >>>>>>
> >>>>>> with 
> >>>>>> commit db7286b100b503ef80612884453bed53d74c9a16 (refs/bisect/skip-db7286b100b503ef80612884453bed53d74c9a16)
> >>>>>>     vhost: use batched version by default
> >>>>>> plus
> >>>>>> commit 6bd262d5eafcdf8cdfae491e2e748e4e434dcda6 (HEAD, refs/bisect/bad)
> >>>>>>     Revert "vhost/net: add an option to test new code"
> >>>>>> to make things compile (your next tree is not easily bisectable, can you fix that as well?).
> >>>>>
> >>>>> I'll try.
> >>>>>
> >>>>>>
> >>>>>> I get random crashes in my s390 KVM guests after reboot.
> >>>>>> Reverting both patches together with commit decd9b8 "vhost: use vhost_desc instead of vhost_log" to
> >>>>>> make it compile again) on top of linux-next-1218 makes the problem go away.
> >>>>>>
> >>>>>> Looks like the batched version is not yet ready for prime time. Can you drop these patches until
> >>>>>> we have fixed the issues?
> >>>>>>
> >>>>>> Christian
> >>>>>>
> >>>>>
> >>>>> Will do, thanks for letting me know.
> >>>>
> >>>> I have confirmed with the initial reporter (internal test team) that <driver name='qemu'/> 
> >>>> with a known to be broken linux next kernel also fixes the problem, so it is really the
> >>>> vhost changes.
> >>>
> >>> OK I'm back and trying to make it more bisectable.
> >>>
> >>> I pushed a new tag "batch-v2".
> >>> It's same code but with this bisect should get more information.
> >>
> >> I get the following with this tag
> >>
> >> drivers/vhost/net.c: In function ‘vhost_net_tx_get_vq_desc’:
> >> drivers/vhost/net.c:574:7: error: implicit declaration of function ‘vhost_get_vq_desc_batch’; did you mean ‘vhost_get_vq_desc’? [-Werror=implicit-function-declaration]
> >>   574 |   r = vhost_get_vq_desc_batch(tvq, tvq->iov, ARRAY_SIZE(tvq->iov),
> >>       |       ^~~~~~~~~~~~~~~~~~~~~~~
> >>       |       vhost_get_vq_desc
> >>
> > 
> > Not sure why but I pushed a wrong commit. Sorry. Should be good now.
> > 
> 
> during bisect:
> 
> drivers/vhost/vhost.c: In function ‘vhost_get_vq_desc_batch’:
> drivers/vhost/vhost.c:2634:8: error: ‘id’ undeclared (first use in this function); did you mean ‘i’?
>  2634 |  ret = id;
>       |        ^~
>       |        i
> 
> I changed that to i
> 
> 
> The last step then gave me  (on commit 50297a8480b439efc5f3f23088cb2d90b799acef vhost: use batched version by default)
> net enc1: Unexpected TXQ (0) queue failure: -5
> in the guest.
> 
> bisect log so far:
> [cborntra@m83lp52 linux]$ git bisect log
> git bisect start
> # bad: [3131e79bb9e9892a5a6bd33513de9bc90b20e867] vhost: use vhost_desc instead of vhost_log
> git bisect bad 3131e79bb9e9892a5a6bd33513de9bc90b20e867
> # good: [d1281e3a562ec6a08f944a876481dd043ba739b9] virtio-blk: remove VIRTIO_BLK_F_SCSI support
> git bisect good d1281e3a562ec6a08f944a876481dd043ba739b9
> # good: [5b00aab5b6332a67e32dace1dcd3a198ab94ed56] vhost: option to fetch descriptors through an independent struct
> git bisect good 5b00aab5b6332a67e32dace1dcd3a198ab94ed56
> # good: [5b00aab5b6332a67e32dace1dcd3a198ab94ed56] vhost: option to fetch descriptors through an independent struct
> git bisect good 5b00aab5b6332a67e32dace1dcd3a198ab94ed56
> # bad: [1414d7ee3d10d2ec2bc4ee652d1d90ec91da1c79] vhost: batching fetches
> git bisect bad 1414d7ee3d10d2ec2bc4ee652d1d90ec91da1c79
> 
> 

I pushed batched-v3 - same head but bisect should work now.

-- 
MST

Re: vhost changes (batched) in linux-next after 12/13 trigger random crashes in KVM guests after reboot

[Christian Borntraeger]

On 07.01.20 12:55, Michael S. Tsirkin wrote:

> 
> I pushed batched-v3 - same head but bisect should work now.
> 

With 
commit 38ced0208491103b50f1056f0d1c8f28e2e13d08 (HEAD)
Author:     Michael S. Tsirkin <mst@redhat.com>
AuthorDate: Wed Dec 11 12:19:26 2019 -0500
Commit:     Michael S. Tsirkin <mst@redhat.com>
CommitDate: Tue Jan 7 06:52:42 2020 -0500

    vhost: use batched version by default


I have exactly one successful ping and then the network inside the guest is broken (no packet
anymore).

So you could consider this commit broken (but in a different way and also without any
guest reboot necessary).


bisect log:
git bisect start
# bad: [d2f6175f52062ee51ee69754a6925608213475d2] vhost: use vhost_desc instead of vhost_log
git bisect bad d2f6175f52062ee51ee69754a6925608213475d2
# good: [d1281e3a562ec6a08f944a876481dd043ba739b9] virtio-blk: remove VIRTIO_BLK_F_SCSI support
git bisect good d1281e3a562ec6a08f944a876481dd043ba739b9
# good: [fac7c0f46996e32d996f5c46121df24a6b95ec3b] vhost: option to fetch descriptors through an independent struct
git bisect good fac7c0f46996e32d996f5c46121df24a6b95ec3b
# bad: [539eb9d738f048cd7be61f404e8f9c7d9d2ff3cc] vhost: batching fetches
git bisect bad 539eb9d738f048cd7be61f404e8f9c7d9d2ff3cc

Re: vhost changes (batched) in linux-next after 12/13 trigger random crashes in KVM guests after reboot

[Michael S. Tsirkin]

On Tue, Jan 07, 2020 at 01:16:50PM +0100, Christian Borntraeger wrote:
> On 07.01.20 12:55, Michael S. Tsirkin wrote:
> 
> > 
> > I pushed batched-v3 - same head but bisect should work now.
> > 
> 
> With 
> commit 38ced0208491103b50f1056f0d1c8f28e2e13d08 (HEAD)
> Author:     Michael S. Tsirkin <mst@redhat.com>
> AuthorDate: Wed Dec 11 12:19:26 2019 -0500
> Commit:     Michael S. Tsirkin <mst@redhat.com>
> CommitDate: Tue Jan 7 06:52:42 2020 -0500
> 
>     vhost: use batched version by default
> 
> 
> I have exactly one successful ping and then the network inside the guest is broken (no packet
> anymore).

Does anything appear in host's dmesg when this happens?


> So you could consider this commit broken (but in a different way and also without any
> guest reboot necessary).
> 
> 
> bisect log:
> git bisect start
> # bad: [d2f6175f52062ee51ee69754a6925608213475d2] vhost: use vhost_desc instead of vhost_log
> git bisect bad d2f6175f52062ee51ee69754a6925608213475d2
> # good: [d1281e3a562ec6a08f944a876481dd043ba739b9] virtio-blk: remove VIRTIO_BLK_F_SCSI support
> git bisect good d1281e3a562ec6a08f944a876481dd043ba739b9
> # good: [fac7c0f46996e32d996f5c46121df24a6b95ec3b] vhost: option to fetch descriptors through an independent struct
> git bisect good fac7c0f46996e32d996f5c46121df24a6b95ec3b
> # bad: [539eb9d738f048cd7be61f404e8f9c7d9d2ff3cc] vhost: batching fetches
> git bisect bad 539eb9d738f048cd7be61f404e8f9c7d9d2ff3cc

Re: vhost changes (batched) in linux-next after 12/13 trigger random crashes in KVM guests after reboot

[Christian Borntraeger]

On 20.01.20 07:27, Michael S. Tsirkin wrote:
> On Tue, Jan 07, 2020 at 01:16:50PM +0100, Christian Borntraeger wrote:
>> On 07.01.20 12:55, Michael S. Tsirkin wrote:
>>
>>>
>>> I pushed batched-v3 - same head but bisect should work now.
>>>
>>
>> With 
>> commit 38ced0208491103b50f1056f0d1c8f28e2e13d08 (HEAD)
>> Author:     Michael S. Tsirkin <mst@redhat.com>
>> AuthorDate: Wed Dec 11 12:19:26 2019 -0500
>> Commit:     Michael S. Tsirkin <mst@redhat.com>
>> CommitDate: Tue Jan 7 06:52:42 2020 -0500
>>
>>     vhost: use batched version by default
>>
>>
>> I have exactly one successful ping and then the network inside the guest is broken (no packet
>> anymore).
> 
> Does anything appear in host's dmesg when this happens?

I think there was nothing, but I am not sure. I would need to redo the test if this is important to know.

> 
> 
>> So you could consider this commit broken (but in a different way and also without any
>> guest reboot necessary).
>>
>>
>> bisect log:
>> git bisect start
>> # bad: [d2f6175f52062ee51ee69754a6925608213475d2] vhost: use vhost_desc instead of vhost_log
>> git bisect bad d2f6175f52062ee51ee69754a6925608213475d2
>> # good: [d1281e3a562ec6a08f944a876481dd043ba739b9] virtio-blk: remove VIRTIO_BLK_F_SCSI support
>> git bisect good d1281e3a562ec6a08f944a876481dd043ba739b9
>> # good: [fac7c0f46996e32d996f5c46121df24a6b95ec3b] vhost: option to fetch descriptors through an independent struct
>> git bisect good fac7c0f46996e32d996f5c46121df24a6b95ec3b
>> # bad: [539eb9d738f048cd7be61f404e8f9c7d9d2ff3cc] vhost: batching fetches
>> git bisect bad 539eb9d738f048cd7be61f404e8f9c7d9d2ff3cc
> 

Re: vhost changes (batched) in linux-next after 12/13 trigger random crashes in KVM guests after reboot

[eperezma]

Hi Christian.

Could you try this patch on top of ("38ced0208491 vhost: use batched version by default")?

It will not solve your first random crash but it should help with the lost of network connectivity.

Please let me know how does it goes.

Thanks!

From 99f0f543f3939dbe803988c9153a95616ccccacd Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Eugenio=20P=C3=A9rez?= <eperezma@redhat.com>
Date: Thu, 6 Feb 2020 15:13:42 +0100
Subject: [PATCH] vhost: filter valid vhost descriptors flags
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

Previous commit copy _NEXT flag, and it complains if a copied descriptor
contains it.

Signed-off-by: Eugenio Pérez <eperezma@redhat.com>
---
 drivers/vhost/vhost.c | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)

diff --git a/drivers/vhost/vhost.c b/drivers/vhost/vhost.c
index 27ae5b4872a0..56c5253056ee 100644
--- a/drivers/vhost/vhost.c
+++ b/drivers/vhost/vhost.c
@@ -2125,6 +2125,8 @@ static void pop_split_desc(struct vhost_virtqueue *vq)
 	--vq->ndescs;
 }
 
+#define VHOST_DESC_FLAGS (VRING_DESC_F_INDIRECT | VRING_DESC_F_WRITE | \
+			  VRING_DESC_F_NEXT)
 static int push_split_desc(struct vhost_virtqueue *vq, struct vring_desc *desc, u16 id)
 {
 	struct vhost_desc *h;
@@ -2134,7 +2136,7 @@ static int push_split_desc(struct vhost_virtqueue *vq, struct vring_desc *desc,
 	h = &vq->descs[vq->ndescs++];
 	h->addr = vhost64_to_cpu(vq, desc->addr);
 	h->len = vhost32_to_cpu(vq, desc->len);
-	h->flags = vhost16_to_cpu(vq, desc->flags);
+	h->flags = vhost16_to_cpu(vq, desc->flags) & VHOST_DESC_FLAGS;
 	h->id = id;
 
 	return 0;
@@ -2343,7 +2345,7 @@ int vhost_get_vq_desc(struct vhost_virtqueue *vq,
 		struct vhost_desc *desc = &vq->descs[i];
 		int access;
 
-		if (desc->flags & ~(VRING_DESC_F_INDIRECT | VRING_DESC_F_WRITE)) {
+		if (desc->flags & ~VHOST_DESC_FLAGS) {
 			vq_err(vq, "Unexpected flags: 0x%x at descriptor id 0x%x\n",
 			       desc->flags, desc->id);
 			ret = -EINVAL;
-- 
2.18.1


On Wed, 2020-01-22 at 20:32 +0100, Christian Borntraeger wrote:
> 
> On 20.01.20 07:27, Michael S. Tsirkin wrote:
> > On Tue, Jan 07, 2020 at 01:16:50PM +0100, Christian Borntraeger wrote:
> > > On 07.01.20 12:55, Michael S. Tsirkin wrote:
> > > 
> > > > I pushed batched-v3 - same head but bisect should work now.
> > > > 
> > > 
> > > With 
> > > commit 38ced0208491103b50f1056f0d1c8f28e2e13d08 (HEAD)
> > > Author:     Michael S. Tsirkin <mst@redhat.com>
> > > AuthorDate: Wed Dec 11 12:19:26 2019 -0500
> > > Commit:     Michael S. Tsirkin <mst@redhat.com>
> > > CommitDate: Tue Jan 7 06:52:42 2020 -0500
> > > 
> > >     vhost: use batched version by default
> > > 
> > > 
> > > I have exactly one successful ping and then the network inside the guest is broken (no packet
> > > anymore).
> > 
> > Does anything appear in host's dmesg when this happens?
> 
> I think there was nothing, but I am not sure. I would need to redo the test if this is important to know.
> 
> > 
> > > So you could consider this commit broken (but in a different way and also without any
> > > guest reboot necessary).
> > > 
> > > 
> > > bisect log:
> > > git bisect start
> > > # bad: [d2f6175f52062ee51ee69754a6925608213475d2] vhost: use vhost_desc instead of vhost_log
> > > git bisect bad d2f6175f52062ee51ee69754a6925608213475d2
> > > # good: [d1281e3a562ec6a08f944a876481dd043ba739b9] virtio-blk: remove VIRTIO_BLK_F_SCSI support
> > > git bisect good d1281e3a562ec6a08f944a876481dd043ba739b9
> > > # good: [fac7c0f46996e32d996f5c46121df24a6b95ec3b] vhost: option to fetch descriptors through an independent
> > > struct
> > > git bisect good fac7c0f46996e32d996f5c46121df24a6b95ec3b
> > > # bad: [539eb9d738f048cd7be61f404e8f9c7d9d2ff3cc] vhost: batching fetches
> > > git bisect bad 539eb9d738f048cd7be61f404e8f9c7d9d2ff3cc

Re: vhost changes (batched) in linux-next after 12/13 trigger random crashes in KVM guests after reboot

[Christian Borntraeger]

On 06.02.20 15:22, eperezma@redhat.com wrote:
> Hi Christian.
> 
> Could you try this patch on top of ("38ced0208491 vhost: use batched version by default")?
> 
> It will not solve your first random crash but it should help with the lost of network connectivity.
> 
> Please let me know how does it goes.


38ced0208491 + this seem to be ok.

Not sure if you can make out anything of this (and the previous git bisect log)

Re: vhost changes (batched) in linux-next after 12/13 trigger random crashes in KVM guests after reboot

[Michael S. Tsirkin]

On Thu, Feb 06, 2020 at 03:22:39PM +0100, eperezma@redhat.com wrote:
> Hi Christian.
> 
> Could you try this patch on top of ("38ced0208491 vhost: use batched version by default")?
> 
> It will not solve your first random crash but it should help with the lost of network connectivity.
> 
> Please let me know how does it goes.
> 
> Thanks!
> 
> >From 99f0f543f3939dbe803988c9153a95616ccccacd Mon Sep 17 00:00:00 2001
> From: =?UTF-8?q?Eugenio=20P=C3=A9rez?= <eperezma@redhat.com>
> Date: Thu, 6 Feb 2020 15:13:42 +0100
> Subject: [PATCH] vhost: filter valid vhost descriptors flags
> MIME-Version: 1.0
> Content-Type: text/plain; charset=UTF-8
> Content-Transfer-Encoding: 8bit
> 
> Previous commit copy _NEXT flag, and it complains if a copied descriptor
> contains it.
> 
> Signed-off-by: Eugenio Pérez <eperezma@redhat.com>
> ---
>  drivers/vhost/vhost.c | 6 ++++--
>  1 file changed, 4 insertions(+), 2 deletions(-)
> 
> diff --git a/drivers/vhost/vhost.c b/drivers/vhost/vhost.c
> index 27ae5b4872a0..56c5253056ee 100644
> --- a/drivers/vhost/vhost.c
> +++ b/drivers/vhost/vhost.c
> @@ -2125,6 +2125,8 @@ static void pop_split_desc(struct vhost_virtqueue *vq)
>  	--vq->ndescs;
>  }
>  
> +#define VHOST_DESC_FLAGS (VRING_DESC_F_INDIRECT | VRING_DESC_F_WRITE | \
> +			  VRING_DESC_F_NEXT)
>  static int push_split_desc(struct vhost_virtqueue *vq, struct vring_desc *desc, u16 id)
>  {
>  	struct vhost_desc *h;
> @@ -2134,7 +2136,7 @@ static int push_split_desc(struct vhost_virtqueue *vq, struct vring_desc *desc,
>  	h = &vq->descs[vq->ndescs++];
>  	h->addr = vhost64_to_cpu(vq, desc->addr);
>  	h->len = vhost32_to_cpu(vq, desc->len);
> -	h->flags = vhost16_to_cpu(vq, desc->flags);
> +	h->flags = vhost16_to_cpu(vq, desc->flags) & VHOST_DESC_FLAGS;
>  	h->id = id;
>  
>  	return 0;



> @@ -2343,7 +2345,7 @@ int vhost_get_vq_desc(struct vhost_virtqueue *vq,
>  		struct vhost_desc *desc = &vq->descs[i];
>  		int access;
>  
> -		if (desc->flags & ~(VRING_DESC_F_INDIRECT | VRING_DESC_F_WRITE)) {
> +		if (desc->flags & ~VHOST_DESC_FLAGS) {
>  			vq_err(vq, "Unexpected flags: 0x%x at descriptor id 0x%x\n",
>  			       desc->flags, desc->id);
>  			ret = -EINVAL;
> -- 
> 2.18.1

Thanks for catching this!

Do we need the 1st chunk though?

It seems preferable to just muck with flags in 1 place, when we
validate them ...

> 
> On Wed, 2020-01-22 at 20:32 +0100, Christian Borntraeger wrote:
> > 
> > On 20.01.20 07:27, Michael S. Tsirkin wrote:
> > > On Tue, Jan 07, 2020 at 01:16:50PM +0100, Christian Borntraeger wrote:
> > > > On 07.01.20 12:55, Michael S. Tsirkin wrote:
> > > > 
> > > > > I pushed batched-v3 - same head but bisect should work now.
> > > > > 
> > > > 
> > > > With 
> > > > commit 38ced0208491103b50f1056f0d1c8f28e2e13d08 (HEAD)
> > > > Author:     Michael S. Tsirkin <mst@redhat.com>
> > > > AuthorDate: Wed Dec 11 12:19:26 2019 -0500
> > > > Commit:     Michael S. Tsirkin <mst@redhat.com>
> > > > CommitDate: Tue Jan 7 06:52:42 2020 -0500
> > > > 
> > > >     vhost: use batched version by default
> > > > 
> > > > 
> > > > I have exactly one successful ping and then the network inside the guest is broken (no packet
> > > > anymore).
> > > 
> > > Does anything appear in host's dmesg when this happens?
> > 
> > I think there was nothing, but I am not sure. I would need to redo the test if this is important to know.
> > 
> > > 
> > > > So you could consider this commit broken (but in a different way and also without any
> > > > guest reboot necessary).
> > > > 
> > > > 
> > > > bisect log:
> > > > git bisect start
> > > > # bad: [d2f6175f52062ee51ee69754a6925608213475d2] vhost: use vhost_desc instead of vhost_log
> > > > git bisect bad d2f6175f52062ee51ee69754a6925608213475d2
> > > > # good: [d1281e3a562ec6a08f944a876481dd043ba739b9] virtio-blk: remove VIRTIO_BLK_F_SCSI support
> > > > git bisect good d1281e3a562ec6a08f944a876481dd043ba739b9
> > > > # good: [fac7c0f46996e32d996f5c46121df24a6b95ec3b] vhost: option to fetch descriptors through an independent
> > > > struct
> > > > git bisect good fac7c0f46996e32d996f5c46121df24a6b95ec3b
> > > > # bad: [539eb9d738f048cd7be61f404e8f9c7d9d2ff3cc] vhost: batching fetches
> > > > git bisect bad 539eb9d738f048cd7be61f404e8f9c7d9d2ff3cc

Re: vhost changes (batched) in linux-next after 12/13 trigger random crashes in KVM guests after reboot

[Michael S. Tsirkin]

On Thu, Feb 06, 2020 at 04:12:21PM +0100, Christian Borntraeger wrote:
> 
> 
> On 06.02.20 15:22, eperezma@redhat.com wrote:
> > Hi Christian.
> > 
> > Could you try this patch on top of ("38ced0208491 vhost: use batched version by default")?
> > 
> > It will not solve your first random crash but it should help with the lost of network connectivity.
> > 
> > Please let me know how does it goes.
> 
> 
> 38ced0208491 + this seem to be ok.
> 
> Not sure if you can make out anything of this (and the previous git bisect log)

Yes it does - that this is just bad split-up of patches, and there's
still a real bug that caused worse crashes :)

So I just pushed batch-v4.
I expect that will fail, and bisect to give us
    vhost: batching fetches
Can you try that please?

Re: vhost changes (batched) in linux-next after 12/13 trigger random crashes in KVM guests after reboot

[Christian Borntraeger]

Also adding Cornelia.


On 06.02.20 23:17, Michael S. Tsirkin wrote:
> On Thu, Feb 06, 2020 at 04:12:21PM +0100, Christian Borntraeger wrote:
>>
>>
>> On 06.02.20 15:22, eperezma@redhat.com wrote:
>>> Hi Christian.
>>>
>>> Could you try this patch on top of ("38ced0208491 vhost: use batched version by default")?
>>>
>>> It will not solve your first random crash but it should help with the lost of network connectivity.
>>>
>>> Please let me know how does it goes.
>>
>>
>> 38ced0208491 + this seem to be ok.
>>
>> Not sure if you can make out anything of this (and the previous git bisect log)
> 
> Yes it does - that this is just bad split-up of patches, and there's
> still a real bug that caused worse crashes :)
> 
> So I just pushed batch-v4.
> I expect that will fail, and bisect to give us
>     vhost: batching fetches
> Can you try that please?
> 

yes.

eccb852f1fe6bede630e2e4f1a121a81e34354ab is the first bad commit
commit eccb852f1fe6bede630e2e4f1a121a81e34354ab
Author: Michael S. Tsirkin <mst@redhat.com>
Date:   Mon Oct 7 06:11:18 2019 -0400

    vhost: batching fetches
    
    With this patch applied, new and old code perform identically.
    
    Lots of extra optimizations are now possible, e.g.
    we can fetch multiple heads with copy_from/to_user now.
    We can get rid of maintaining the log array.  Etc etc.
    
    Signed-off-by: Michael S. Tsirkin <mst@redhat.com>

 drivers/vhost/test.c  |  2 +-
 drivers/vhost/vhost.c | 39 ++++++++++++++++++++++++++++++++++-----
 drivers/vhost/vhost.h |  4 +++-
 3 files changed, 38 insertions(+), 7 deletions(-)


> 
			

Re: vhost changes (batched) in linux-next after 12/13 trigger random crashes in KVM guests after reboot

[Michael S. Tsirkin]

On Fri, Feb 07, 2020 at 08:47:14AM +0100, Christian Borntraeger wrote:
> Also adding Cornelia.
> 
> 
> On 06.02.20 23:17, Michael S. Tsirkin wrote:
> > On Thu, Feb 06, 2020 at 04:12:21PM +0100, Christian Borntraeger wrote:
> >>
> >>
> >> On 06.02.20 15:22, eperezma@redhat.com wrote:
> >>> Hi Christian.
> >>>
> >>> Could you try this patch on top of ("38ced0208491 vhost: use batched version by default")?
> >>>
> >>> It will not solve your first random crash but it should help with the lost of network connectivity.
> >>>
> >>> Please let me know how does it goes.
> >>
> >>
> >> 38ced0208491 + this seem to be ok.
> >>
> >> Not sure if you can make out anything of this (and the previous git bisect log)
> > 
> > Yes it does - that this is just bad split-up of patches, and there's
> > still a real bug that caused worse crashes :)
> > 
> > So I just pushed batch-v4.
> > I expect that will fail, and bisect to give us
> >     vhost: batching fetches
> > Can you try that please?
> > 
> 
> yes.
> 
> eccb852f1fe6bede630e2e4f1a121a81e34354ab is the first bad commit
> commit eccb852f1fe6bede630e2e4f1a121a81e34354ab
> Author: Michael S. Tsirkin <mst@redhat.com>
> Date:   Mon Oct 7 06:11:18 2019 -0400
> 
>     vhost: batching fetches
>     
>     With this patch applied, new and old code perform identically.
>     
>     Lots of extra optimizations are now possible, e.g.
>     we can fetch multiple heads with copy_from/to_user now.
>     We can get rid of maintaining the log array.  Etc etc.
>     
>     Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
> 
>  drivers/vhost/test.c  |  2 +-
>  drivers/vhost/vhost.c | 39 ++++++++++++++++++++++++++++++++++-----
>  drivers/vhost/vhost.h |  4 +++-
>  3 files changed, 38 insertions(+), 7 deletions(-)
> 


And the symptom is still the same - random crashes
after a bit of traffic, right?

> > 
> 			

Re: vhost changes (batched) in linux-next after 12/13 trigger random crashes in KVM guests after reboot

[Christian Borntraeger]

On 07.02.20 08:58, Michael S. Tsirkin wrote:
> On Fri, Feb 07, 2020 at 08:47:14AM +0100, Christian Borntraeger wrote:
>> Also adding Cornelia.
>>
>>
>> On 06.02.20 23:17, Michael S. Tsirkin wrote:
>>> On Thu, Feb 06, 2020 at 04:12:21PM +0100, Christian Borntraeger wrote:
>>>>
>>>>
>>>> On 06.02.20 15:22, eperezma@redhat.com wrote:
>>>>> Hi Christian.
>>>>>
>>>>> Could you try this patch on top of ("38ced0208491 vhost: use batched version by default")?
>>>>>
>>>>> It will not solve your first random crash but it should help with the lost of network connectivity.
>>>>>
>>>>> Please let me know how does it goes.
>>>>
>>>>
>>>> 38ced0208491 + this seem to be ok.
>>>>
>>>> Not sure if you can make out anything of this (and the previous git bisect log)
>>>
>>> Yes it does - that this is just bad split-up of patches, and there's
>>> still a real bug that caused worse crashes :)
>>>
>>> So I just pushed batch-v4.
>>> I expect that will fail, and bisect to give us
>>>     vhost: batching fetches
>>> Can you try that please?
>>>
>>
>> yes.
>>
>> eccb852f1fe6bede630e2e4f1a121a81e34354ab is the first bad commit
>> commit eccb852f1fe6bede630e2e4f1a121a81e34354ab
>> Author: Michael S. Tsirkin <mst@redhat.com>
>> Date:   Mon Oct 7 06:11:18 2019 -0400
>>
>>     vhost: batching fetches
>>     
>>     With this patch applied, new and old code perform identically.
>>     
>>     Lots of extra optimizations are now possible, e.g.
>>     we can fetch multiple heads with copy_from/to_user now.
>>     We can get rid of maintaining the log array.  Etc etc.
>>     
>>     Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
>>
>>  drivers/vhost/test.c  |  2 +-
>>  drivers/vhost/vhost.c | 39 ++++++++++++++++++++++++++++++++++-----
>>  drivers/vhost/vhost.h |  4 +++-
>>  3 files changed, 38 insertions(+), 7 deletions(-)
>>
> 
> 
> And the symptom is still the same - random crashes
> after a bit of traffic, right?

random guest crashes after a reboot of the guests. As if vhost would still
write into now stale buffers.

Re: vhost changes (batched) in linux-next after 12/13 trigger random crashes in KVM guests after reboot

[Cornelia Huck]

On Fri, 7 Feb 2020 09:13:14 +0100
Christian Borntraeger <borntraeger@de.ibm.com> wrote:

> On 07.02.20 08:58, Michael S. Tsirkin wrote:
> > On Fri, Feb 07, 2020 at 08:47:14AM +0100, Christian Borntraeger wrote:  
> >> Also adding Cornelia.
> >>
> >>
> >> On 06.02.20 23:17, Michael S. Tsirkin wrote:  
> >>> On Thu, Feb 06, 2020 at 04:12:21PM +0100, Christian Borntraeger wrote:  
> >>>>
> >>>>
> >>>> On 06.02.20 15:22, eperezma@redhat.com wrote:  
> >>>>> Hi Christian.
> >>>>>
> >>>>> Could you try this patch on top of ("38ced0208491 vhost: use batched version by default")?
> >>>>>
> >>>>> It will not solve your first random crash but it should help with the lost of network connectivity.
> >>>>>
> >>>>> Please let me know how does it goes.  
> >>>>
> >>>>
> >>>> 38ced0208491 + this seem to be ok.
> >>>>
> >>>> Not sure if you can make out anything of this (and the previous git bisect log)  
> >>>
> >>> Yes it does - that this is just bad split-up of patches, and there's
> >>> still a real bug that caused worse crashes :)
> >>>
> >>> So I just pushed batch-v4.
> >>> I expect that will fail, and bisect to give us
> >>>     vhost: batching fetches
> >>> Can you try that please?
> >>>  
> >>
> >> yes.
> >>
> >> eccb852f1fe6bede630e2e4f1a121a81e34354ab is the first bad commit
> >> commit eccb852f1fe6bede630e2e4f1a121a81e34354ab
> >> Author: Michael S. Tsirkin <mst@redhat.com>
> >> Date:   Mon Oct 7 06:11:18 2019 -0400
> >>
> >>     vhost: batching fetches
> >>     
> >>     With this patch applied, new and old code perform identically.
> >>     
> >>     Lots of extra optimizations are now possible, e.g.
> >>     we can fetch multiple heads with copy_from/to_user now.
> >>     We can get rid of maintaining the log array.  Etc etc.
> >>     
> >>     Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
> >>
> >>  drivers/vhost/test.c  |  2 +-
> >>  drivers/vhost/vhost.c | 39 ++++++++++++++++++++++++++++++++++-----
> >>  drivers/vhost/vhost.h |  4 +++-
> >>  3 files changed, 38 insertions(+), 7 deletions(-)
> >>  
> > 
> > 
> > And the symptom is still the same - random crashes
> > after a bit of traffic, right?  
> 
> random guest crashes after a reboot of the guests. As if vhost would still
> write into now stale buffers.
> 

I'm late to the party; but where is that commit located? Or has it been
dropped again already?

Re: vhost changes (batched) in linux-next after 12/13 trigger random crashes in KVM guests after reboot

[Michael S. Tsirkin]

On Fri, Feb 07, 2020 at 09:53:53AM +0100, Cornelia Huck wrote:
> On Fri, 7 Feb 2020 09:13:14 +0100
> Christian Borntraeger <borntraeger@de.ibm.com> wrote:
> 
> > On 07.02.20 08:58, Michael S. Tsirkin wrote:
> > > On Fri, Feb 07, 2020 at 08:47:14AM +0100, Christian Borntraeger wrote:  
> > >> Also adding Cornelia.
> > >>
> > >>
> > >> On 06.02.20 23:17, Michael S. Tsirkin wrote:  
> > >>> On Thu, Feb 06, 2020 at 04:12:21PM +0100, Christian Borntraeger wrote:  
> > >>>>
> > >>>>
> > >>>> On 06.02.20 15:22, eperezma@redhat.com wrote:  
> > >>>>> Hi Christian.
> > >>>>>
> > >>>>> Could you try this patch on top of ("38ced0208491 vhost: use batched version by default")?
> > >>>>>
> > >>>>> It will not solve your first random crash but it should help with the lost of network connectivity.
> > >>>>>
> > >>>>> Please let me know how does it goes.  
> > >>>>
> > >>>>
> > >>>> 38ced0208491 + this seem to be ok.
> > >>>>
> > >>>> Not sure if you can make out anything of this (and the previous git bisect log)  
> > >>>
> > >>> Yes it does - that this is just bad split-up of patches, and there's
> > >>> still a real bug that caused worse crashes :)
> > >>>
> > >>> So I just pushed batch-v4.
> > >>> I expect that will fail, and bisect to give us
> > >>>     vhost: batching fetches
> > >>> Can you try that please?
> > >>>  
> > >>
> > >> yes.
> > >>
> > >> eccb852f1fe6bede630e2e4f1a121a81e34354ab is the first bad commit
> > >> commit eccb852f1fe6bede630e2e4f1a121a81e34354ab
> > >> Author: Michael S. Tsirkin <mst@redhat.com>
> > >> Date:   Mon Oct 7 06:11:18 2019 -0400
> > >>
> > >>     vhost: batching fetches
> > >>     
> > >>     With this patch applied, new and old code perform identically.
> > >>     
> > >>     Lots of extra optimizations are now possible, e.g.
> > >>     we can fetch multiple heads with copy_from/to_user now.
> > >>     We can get rid of maintaining the log array.  Etc etc.
> > >>     
> > >>     Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
> > >>
> > >>  drivers/vhost/test.c  |  2 +-
> > >>  drivers/vhost/vhost.c | 39 ++++++++++++++++++++++++++++++++++-----
> > >>  drivers/vhost/vhost.h |  4 +++-
> > >>  3 files changed, 38 insertions(+), 7 deletions(-)
> > >>  
> > > 
> > > 
> > > And the symptom is still the same - random crashes
> > > after a bit of traffic, right?  
> > 
> > random guest crashes after a reboot of the guests. As if vhost would still
> > write into now stale buffers.
> > 
> 
> I'm late to the party; but where is that commit located? Or has it been
> dropped again already?

my vhost tree. Tag batch-v4.

Re: vhost changes (batched) in linux-next after 12/13 trigger random crashes in KVM guests after reboot

[Christian Borntraeger]

On 10.02.20 10:40, Eugenio Perez Martin wrote:
> Hi Christian.
> 
> I'm not able to reproduce the failure with eccb852f1fe6bede630e2e4f1a121a81e34354ab commit. Could you add more data? Your configuration (libvirt or qemu line), and host's dmesg output if any?

I do the following in the guest:
ping -c 200 -f somevalidip; reboot
sometimes I need to do that multiple times and sometimes I do not get a guest crash but host dmesg like

Guest moved used index from 0 to 292

xml is pretty simple

    <interface type='direct'>
      <mac address='52:54:00:7c:2c:f3'/>
      <source dev='encbd00' mode='bridge'/>
      <model type='virtio'/>
      <driver name='vhost'/>
      <address type='ccw' cssid='0xfe' ssid='0x0' devno='0x0001'/>
    </interface>


Reverting this patch seems to make both problems go away.


> 
> Thanks!
> 
> On Fri, Feb 7, 2020 at 9:13 AM Christian Borntraeger <borntraeger@de.ibm.com <mailto:borntraeger@de.ibm.com>> wrote:
> 
> 
> 
>     On 07.02.20 08:58, Michael S. Tsirkin wrote:
>     > On Fri, Feb 07, 2020 at 08:47:14AM +0100, Christian Borntraeger wrote:
>     >> Also adding Cornelia.
>     >>
>     >>
>     >> On 06.02.20 23:17, Michael S. Tsirkin wrote:
>     >>> On Thu, Feb 06, 2020 at 04:12:21PM +0100, Christian Borntraeger wrote:
>     >>>>
>     >>>>
>     >>>> On 06.02.20 15:22, eperezma@redhat.com <mailto:eperezma@redhat.com> wrote:
>     >>>>> Hi Christian.
>     >>>>>
>     >>>>> Could you try this patch on top of ("38ced0208491 vhost: use batched version by default")?
>     >>>>>
>     >>>>> It will not solve your first random crash but it should help with the lost of network connectivity.
>     >>>>>
>     >>>>> Please let me know how does it goes.
>     >>>>
>     >>>>
>     >>>> 38ced0208491 + this seem to be ok.
>     >>>>
>     >>>> Not sure if you can make out anything of this (and the previous git bisect log)
>     >>>
>     >>> Yes it does - that this is just bad split-up of patches, and there's
>     >>> still a real bug that caused worse crashes :)
>     >>>
>     >>> So I just pushed batch-v4.
>     >>> I expect that will fail, and bisect to give us
>     >>>     vhost: batching fetches
>     >>> Can you try that please?
>     >>>
>     >>
>     >> yes.
>     >>
>     >> eccb852f1fe6bede630e2e4f1a121a81e34354ab is the first bad commit
>     >> commit eccb852f1fe6bede630e2e4f1a121a81e34354ab
>     >> Author: Michael S. Tsirkin <mst@redhat.com <mailto:mst@redhat.com>>
>     >> Date:   Mon Oct 7 06:11:18 2019 -0400
>     >>
>     >>     vhost: batching fetches
>     >>     
>     >>     With this patch applied, new and old code perform identically.
>     >>     
>     >>     Lots of extra optimizations are now possible, e.g.
>     >>     we can fetch multiple heads with copy_from/to_user now.
>     >>     We can get rid of maintaining the log array.  Etc etc.
>     >>     
>     >>     Signed-off-by: Michael S. Tsirkin <mst@redhat.com <mailto:mst@redhat.com>>
>     >>
>     >>  drivers/vhost/test.c  |  2 +-
>     >>  drivers/vhost/vhost.c | 39 ++++++++++++++++++++++++++++++++++-----
>     >>  drivers/vhost/vhost.h |  4 +++-
>     >>  3 files changed, 38 insertions(+), 7 deletions(-)
>     >>
>     >
>     >
>     > And the symptom is still the same - random crashes
>     > after a bit of traffic, right?
> 
>     random guest crashes after a reboot of the guests. As if vhost would still
>     write into now stale buffers.
> 

Re: vhost changes (batched) in linux-next after 12/13 trigger random crashes in KVM guests after reboot

[Christian Borntraeger]

On 10.02.20 10:47, Eugenio Perez Martin wrote:
> Hi Christian.
> 
> I'm not able to reproduce the failure with eccb852f1fe6bede630e2e4f1a121a81e34354ab commit. Could you add more data? Your configuration (libvirt or qemu line), and host's dmesg output if any?
> 
> Thanks!

If it was not obvious, this is on s390x, a big endian system.

Re: vhost changes (batched) in linux-next after 12/13 trigger random crashes in KVM guests after reboot

[Eugenio Pérez]

On Mon, 2020-02-10 at 12:01 +0100, Christian Borntraeger wrote:
> 
> On 10.02.20 10:47, Eugenio Perez Martin wrote:
> > Hi Christian.
> > 
> > I'm not able to reproduce the failure with eccb852f1fe6bede630e2e4f1a121a81e34354ab commit. Could you add more data?
> > Your configuration (libvirt or qemu line), and host's dmesg output if any?
> > 
> > Thanks!
> 
> If it was not obvious, this is on s390x, a big endian system.
> 

Hi Christian. Thank you very much for the hints.

Could we add some debug traces? Something like the inline patch should give us some clues.

Thanks!

From a8d65d5f0ae3d305443ee84b4842b7c712a1ac1d Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Eugenio=20P=C3=A9rez?= <eperezma@redhat.com>
Date: Tue, 11 Feb 2020 10:29:01 +0100
Subject: [PATCH] Add some traces

---
 drivers/vhost/vhost.c | 9 +++++++--
 1 file changed, 7 insertions(+), 2 deletions(-)

diff --git a/drivers/vhost/vhost.c b/drivers/vhost/vhost.c
index b5a51b1f2e79..60c048eebe4d 100644
--- a/drivers/vhost/vhost.c
+++ b/drivers/vhost/vhost.c
@@ -2239,8 +2239,8 @@ static int fetch_buf(struct vhost_virtqueue *vq)
 		vq->avail_idx = vhost16_to_cpu(vq, avail_idx);
 
 		if (unlikely((u16)(vq->avail_idx - last_avail_idx) > vq->num)) {
-			vq_err(vq, "Guest moved used index from %u to %u",
-				last_avail_idx, vq->avail_idx);
+			vq_err(vq, "Guest moved vq %p used index from %u to %u",
+				vq, last_avail_idx, vq->avail_idx);
 			return -EFAULT;
 		}
 
@@ -2336,6 +2336,8 @@ static int fetch_descs(struct vhost_virtqueue *vq)
 	while (!ret && vq->ndescs <= vq->batch_descs)
 		ret = fetch_buf(vq);
 
+	pr_debug("[vq=%p][vq->ndescs=%d]", vq, vq->ndescs);
+
 	return vq->ndescs ? 0 : ret;
 }
 
@@ -2416,6 +2418,8 @@ int vhost_get_vq_desc(struct vhost_virtqueue *vq,
 	}
 
 	vq->first_desc = i + 1;
+	pr_debug("[vq=%p][vq->ndescs=%d][vq->first_desc=%d]", vq, vq->ndescs,
+		 vq->first_desc);
 
 	return ret;
 
@@ -2459,6 +2463,7 @@ static int __vhost_add_used_n(struct vhost_virtqueue *vq,
 
 	start = vq->last_used_idx & (vq->num - 1);
 	used = vq->used->ring + start;
+	pr_debug("[vq=%p][start=%d][count=%u]", vq, start, count);
 	if (vhost_put_used(vq, heads, start, count)) {
 		vq_err(vq, "Failed to write used");
 		return -EFAULT;
-- 
2.18.1

Re: vhost changes (batched) in linux-next after 12/13 trigger random crashes in KVM guests after reboot

[Christian Borntraeger]

[-- Attachment #1: Type: text/plain, Size: 816 bytes --]



On 11.02.20 10:56, Christian Borntraeger wrote:
> 
> 
> On 11.02.20 10:33, Eugenio Pérez wrote:
>> On Mon, 2020-02-10 at 12:01 +0100, Christian Borntraeger wrote:
>>>
>>> On 10.02.20 10:47, Eugenio Perez Martin wrote:
>>>> Hi Christian.
>>>>
>>>> I'm not able to reproduce the failure with eccb852f1fe6bede630e2e4f1a121a81e34354ab commit. Could you add more data?
>>>> Your configuration (libvirt or qemu line), and host's dmesg output if any?
>>>>
>>>> Thanks!
>>>
>>> If it was not obvious, this is on s390x, a big endian system.
>>>
>>
>> Hi Christian. Thank you very much for the hints.
>>
>> Could we add some debug traces? Something like the inline patch should give us some clues.
> 
> attached. this time the guest crashed (and no guest moved message)
> 

And here with Guest moved vq after some reboots.

[-- Attachment #2: log2 --]
[-- Type: text/plain, Size: 72381 bytes --]

[   74.661450] [vq=000000008be108ff][vq->ndescs=0]
[  757.654136] [vq=00000000201c16ab][vq->ndescs=0]
[  757.654142] [vq=00000000465a3846][vq->ndescs=0]
[  758.283378] [vq=00000000201c16ab][vq->ndescs=65]
[  758.283381] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=1]
[  758.283385] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=2]
[  758.283386] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=3]
[  758.283388] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=4]
[  758.283390] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=5]
[  758.283391] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=6]
[  758.283393] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=7]
[  758.283394] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=8]
[  758.283397] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=9]
[  758.283399] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=10]
[  758.283401] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=11]
[  758.283403] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=12]
[  758.283406] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=13]
[  758.283407] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=14]
[  758.283409] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=15]
[  758.283410] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=16]
[  758.283413] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=17]
[  758.283414] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=18]
[  758.283417] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=19]
[  758.283418] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=20]
[  758.283420] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=21]
[  758.283422] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=22]
[  758.283423] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=23]
[  758.283425] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=24]
[  758.283426] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=25]
[  758.283428] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=26]
[  758.283429] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=27]
[  758.283431] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=28]
[  758.283433] [vq=00000000201c16ab][start=0][count=28]
[  758.344947] [vq=00000000465a3846][vq->ndescs=1]
[  758.344948] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  758.344951] [vq=00000000465a3846][vq->ndescs=0]
[  758.344962] [vq=00000000465a3846][start=0][count=1]
[  758.366414] [vq=00000000465a3846][vq->ndescs=1]
[  758.366415] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  758.366417] [vq=00000000465a3846][vq->ndescs=0]
[  758.366420] [vq=00000000465a3846][start=1][count=1]
[  758.366913] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=29]
[  758.366915] [vq=00000000201c16ab][start=28][count=1]
[  758.366917] [vq=00000000465a3846][vq->ndescs=1]
[  758.366918] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  758.366919] [vq=00000000465a3846][vq->ndescs=0]
[  758.366924] [vq=00000000465a3846][start=2][count=1]
[  758.366943] [vq=00000000465a3846][vq->ndescs=2]
[  758.366944] [vq=00000000465a3846][vq->ndescs=2][vq->first_desc=1]
[  758.366946] [vq=00000000465a3846][vq->ndescs=2][vq->first_desc=2]
[  758.366947] [vq=00000000465a3846][vq->ndescs=0]
[  758.366951] [vq=00000000465a3846][start=3][count=2]
[  758.369847] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=30]
[  758.369852] [vq=00000000201c16ab][start=29][count=1]
[  758.382143] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=31]
[  758.382145] [vq=00000000201c16ab][start=30][count=1]
[  758.421531] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=32]
[  758.421534] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=33]
[  758.421536] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=34]
[  758.421537] [vq=00000000201c16ab][start=31][count=3]
[  758.480414] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=35]
[  758.480418] [vq=00000000201c16ab][start=34][count=1]
[  758.584968] [vq=00000000465a3846][vq->ndescs=2]
[  758.584973] [vq=00000000465a3846][vq->ndescs=2][vq->first_desc=2]
[  758.584975] [vq=00000000465a3846][vq->ndescs=0]
[  758.584987] [vq=00000000465a3846][start=5][count=1]
[  758.629880] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=36]
[  758.629883] [vq=00000000201c16ab][start=35][count=1]
[  758.696982] [vq=00000000465a3846][vq->ndescs=1]
[  758.696985] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  758.696987] [vq=00000000465a3846][vq->ndescs=0]
[  758.696994] [vq=00000000465a3846][start=6][count=1]
[  758.788476] [vq=00000000465a3846][vq->ndescs=2]
[  758.788478] [vq=00000000465a3846][vq->ndescs=2][vq->first_desc=2]
[  758.788480] [vq=00000000465a3846][vq->ndescs=0]
[  758.788488] [vq=00000000465a3846][start=7][count=1]
[  758.818471] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=37]
[  758.818475] [vq=00000000201c16ab][start=36][count=1]
[  758.864048] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=38]
[  758.864051] [vq=00000000201c16ab][start=37][count=1]
[  758.912219] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=39]
[  758.912221] [vq=00000000201c16ab][start=38][count=1]
[  758.916937] [vq=00000000465a3846][vq->ndescs=1]
[  758.916939] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  758.916940] [vq=00000000465a3846][vq->ndescs=0]
[  758.916949] [vq=00000000465a3846][start=8][count=1]
[  758.992422] [vq=00000000465a3846][vq->ndescs=2]
[  758.992425] [vq=00000000465a3846][vq->ndescs=2][vq->first_desc=2]
[  758.992426] [vq=00000000465a3846][vq->ndescs=0]
[  758.992431] [vq=00000000465a3846][start=9][count=1]
[  759.026942] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=40]
[  759.026944] [vq=00000000201c16ab][start=39][count=1]
[  759.106290] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=41]
[  759.106293] [vq=00000000201c16ab][start=40][count=1]
[  759.176843] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=42]
[  759.176846] [vq=00000000201c16ab][start=41][count=1]
[  759.192881] [vq=00000000465a3846][vq->ndescs=2]
[  759.192884] [vq=00000000465a3846][vq->ndescs=2][vq->first_desc=2]
[  759.192886] [vq=00000000465a3846][vq->ndescs=0]
[  759.192896] [vq=00000000465a3846][start=10][count=1]
[  759.215142] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=43]
[  759.215144] [vq=00000000201c16ab][start=42][count=1]
[  759.277701] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=44]
[  759.277704] [vq=00000000201c16ab][start=43][count=1]
[  759.422515] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=45]
[  759.422518] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=46]
[  759.422520] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=47]
[  759.422521] [vq=00000000201c16ab][start=44][count=3]
[  759.422792] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=48]
[  759.422794] [vq=00000000201c16ab][start=47][count=1]
[  759.626994] [vq=00000000465a3846][vq->ndescs=1]
[  759.626997] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  759.626999] [vq=00000000465a3846][vq->ndescs=0]
[  759.627008] [vq=00000000465a3846][start=11][count=1]
[  759.904224] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=49]
[  759.904232] [vq=00000000201c16ab][start=48][count=1]
[  759.945942] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=50]
[  759.945944] [vq=00000000201c16ab][start=49][count=1]
[  759.951344] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=51]
[  759.951346] [vq=00000000201c16ab][start=50][count=1]
[  760.307838] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=52]
[  760.307841] [vq=00000000201c16ab][start=51][count=1]
[  760.344981] [vq=00000000465a3846][vq->ndescs=1]
[  760.344983] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  760.344985] [vq=00000000465a3846][vq->ndescs=0]
[  760.344994] [vq=00000000465a3846][start=12][count=1]
[  760.351941] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=53]
[  760.351943] [vq=00000000201c16ab][start=52][count=1]
[  760.423655] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=54]
[  760.423657] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=55]
[  760.423658] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=56]
[  760.423659] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=57]
[  760.423661] [vq=00000000201c16ab][start=53][count=4]
[  760.423903] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=58]
[  760.423905] [vq=00000000201c16ab][start=57][count=1]
[  760.633527] [vq=00000000465a3846][vq->ndescs=2]
[  760.633529] [vq=00000000465a3846][vq->ndescs=2][vq->first_desc=2]
[  760.633530] [vq=00000000465a3846][vq->ndescs=0]
[  760.633535] [vq=00000000465a3846][start=13][count=1]
[  760.666987] [vq=00000000465a3846][vq->ndescs=1]
[  760.666988] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  760.666990] [vq=00000000465a3846][vq->ndescs=0]
[  760.666993] [vq=00000000465a3846][start=14][count=1]
[  760.672771] [vq=00000000465a3846][vq->ndescs=1]
[  760.672772] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  760.672773] [vq=00000000465a3846][vq->ndescs=0]
[  760.672777] [vq=00000000465a3846][start=15][count=1]
[  760.673253] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=59]
[  760.673254] [vq=00000000201c16ab][start=58][count=1]
[  760.673277] [vq=00000000465a3846][vq->ndescs=1]
[  760.673278] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  760.673279] [vq=00000000465a3846][vq->ndescs=0]
[  760.673282] [vq=00000000465a3846][start=16][count=1]
[  760.673475] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=60]
[  760.673476] [vq=00000000201c16ab][start=59][count=1]
[  760.673592] [vq=00000000465a3846][vq->ndescs=1]
[  760.673593] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  760.673594] [vq=00000000465a3846][vq->ndescs=0]
[  760.673597] [vq=00000000465a3846][start=17][count=1]
[  760.673751] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=61]
[  760.673753] [vq=00000000201c16ab][start=60][count=1]
[  760.673800] [vq=00000000465a3846][vq->ndescs=1]
[  760.673801] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  760.673803] [vq=00000000465a3846][vq->ndescs=0]
[  760.673805] [vq=00000000465a3846][start=18][count=1]
[  760.673944] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=62]
[  760.673946] [vq=00000000201c16ab][start=61][count=1]
[  760.673986] [vq=00000000465a3846][vq->ndescs=1]
[  760.673987] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  760.673989] [vq=00000000465a3846][vq->ndescs=0]
[  760.673992] [vq=00000000465a3846][start=19][count=1]
[  760.674132] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=63]
[  760.674133] [vq=00000000201c16ab][start=62][count=1]
[  760.674175] [vq=00000000465a3846][vq->ndescs=1]
[  760.674176] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  760.674177] [vq=00000000465a3846][vq->ndescs=0]
[  760.674180] [vq=00000000465a3846][start=20][count=1]
[  760.674316] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=64]
[  760.674318] [vq=00000000201c16ab][start=63][count=1]
[  760.674358] [vq=00000000465a3846][vq->ndescs=1]
[  760.674359] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  760.674360] [vq=00000000465a3846][vq->ndescs=0]
[  760.674363] [vq=00000000465a3846][start=21][count=1]
[  760.674502] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=65]
[  760.674504] [vq=00000000201c16ab][start=64][count=1]
[  760.674548] [vq=00000000465a3846][vq->ndescs=1]
[  760.674549] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  760.674550] [vq=00000000465a3846][vq->ndescs=0]
[  760.674553] [vq=00000000465a3846][start=22][count=1]
[  760.674692] [vq=00000000201c16ab][vq->ndescs=65]
[  760.674693] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=1]
[  760.674694] [vq=00000000201c16ab][start=65][count=1]
[  760.674734] [vq=00000000465a3846][vq->ndescs=1]
[  760.674735] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  760.674737] [vq=00000000465a3846][vq->ndescs=0]
[  760.674739] [vq=00000000465a3846][start=23][count=1]
[  760.674877] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=2]
[  760.674878] [vq=00000000201c16ab][start=66][count=1]
[  760.674926] [vq=00000000465a3846][vq->ndescs=1]
[  760.674927] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  760.674929] [vq=00000000465a3846][vq->ndescs=0]
[  760.674931] [vq=00000000465a3846][start=24][count=1]
[  760.675082] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=3]
[  760.675083] [vq=00000000201c16ab][start=67][count=1]
[  760.675124] [vq=00000000465a3846][vq->ndescs=1]
[  760.675125] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  760.675127] [vq=00000000465a3846][vq->ndescs=0]
[  760.675129] [vq=00000000465a3846][start=25][count=1]
[  760.675266] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=4]
[  760.675268] [vq=00000000201c16ab][start=68][count=1]
[  760.675320] [vq=00000000465a3846][vq->ndescs=1]
[  760.675321] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  760.675322] [vq=00000000465a3846][vq->ndescs=0]
[  760.675325] [vq=00000000465a3846][start=26][count=1]
[  760.675465] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=5]
[  760.675466] [vq=00000000201c16ab][start=69][count=1]
[  760.675508] [vq=00000000465a3846][vq->ndescs=1]
[  760.675509] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  760.675510] [vq=00000000465a3846][vq->ndescs=0]
[  760.675513] [vq=00000000465a3846][start=27][count=1]
[  760.675659] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=6]
[  760.675661] [vq=00000000201c16ab][start=70][count=1]
[  760.675703] [vq=00000000465a3846][vq->ndescs=1]
[  760.675704] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  760.675706] [vq=00000000465a3846][vq->ndescs=0]
[  760.675708] [vq=00000000465a3846][start=28][count=1]
[  760.675845] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=7]
[  760.675846] [vq=00000000201c16ab][start=71][count=1]
[  760.675888] [vq=00000000465a3846][vq->ndescs=1]
[  760.675889] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  760.675890] [vq=00000000465a3846][vq->ndescs=0]
[  760.675893] [vq=00000000465a3846][start=29][count=1]
[  760.676031] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=8]
[  760.676032] [vq=00000000201c16ab][start=72][count=1]
[  760.676073] [vq=00000000465a3846][vq->ndescs=1]
[  760.676074] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  760.676076] [vq=00000000465a3846][vq->ndescs=0]
[  760.676079] [vq=00000000465a3846][start=30][count=1]
[  760.676220] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=9]
[  760.676221] [vq=00000000201c16ab][start=73][count=1]
[  760.676261] [vq=00000000465a3846][vq->ndescs=1]
[  760.676262] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  760.676263] [vq=00000000465a3846][vq->ndescs=0]
[  760.676266] [vq=00000000465a3846][start=31][count=1]
[  760.676406] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=10]
[  760.676407] [vq=00000000201c16ab][start=74][count=1]
[  760.676447] [vq=00000000465a3846][vq->ndescs=1]
[  760.676448] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  760.676450] [vq=00000000465a3846][vq->ndescs=0]
[  760.676452] [vq=00000000465a3846][start=32][count=1]
[  760.676590] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=11]
[  760.676591] [vq=00000000201c16ab][start=75][count=1]
[  760.676631] [vq=00000000465a3846][vq->ndescs=1]
[  760.676632] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  760.676634] [vq=00000000465a3846][vq->ndescs=0]
[  760.676636] [vq=00000000465a3846][start=33][count=1]
[  760.676775] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=12]
[  760.676777] [vq=00000000201c16ab][start=76][count=1]
[  760.676817] [vq=00000000465a3846][vq->ndescs=1]
[  760.676818] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  760.676819] [vq=00000000465a3846][vq->ndescs=0]
[  760.676822] [vq=00000000465a3846][start=34][count=1]
[  760.676961] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=13]
[  760.676963] [vq=00000000201c16ab][start=77][count=1]
[  760.677005] [vq=00000000465a3846][vq->ndescs=1]
[  760.677006] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  760.677007] [vq=00000000465a3846][vq->ndescs=0]
[  760.677011] [vq=00000000465a3846][start=35][count=1]
[  760.677152] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=14]
[  760.677153] [vq=00000000201c16ab][start=78][count=1]
[  760.678377] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=15]
[  760.678379] [vq=00000000201c16ab][start=79][count=1]
[  760.696916] [vq=00000000465a3846][vq->ndescs=1]
[  760.696918] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  760.696919] [vq=00000000465a3846][vq->ndescs=0]
[  760.696922] [vq=00000000465a3846][start=36][count=1]
[  760.731769] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=16]
[  760.731771] [vq=00000000201c16ab][start=80][count=1]
[  760.906976] [vq=00000000465a3846][vq->ndescs=1]
[  760.906980] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  760.906982] [vq=00000000465a3846][vq->ndescs=0]
[  760.906992] [vq=00000000465a3846][start=37][count=1]
[  761.001377] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=17]
[  761.001383] [vq=00000000201c16ab][start=81][count=1]
[  761.025300] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=18]
[  761.025304] [vq=00000000201c16ab][start=82][count=1]
[  761.088194] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=19]
[  761.088196] [vq=00000000201c16ab][start=83][count=1]
[  761.096984] [vq=00000000465a3846][vq->ndescs=1]
[  761.096985] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  761.096987] [vq=00000000465a3846][vq->ndescs=0]
[  761.096995] [vq=00000000465a3846][start=38][count=1]
[  761.347842] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=20]
[  761.347846] [vq=00000000201c16ab][start=84][count=1]
[  761.424800] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=21]
[  761.424803] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=22]
[  761.424805] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=23]
[  761.424807] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=24]
[  761.424809] [vq=00000000201c16ab][start=85][count=4]
[  761.848308] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=25]
[  761.848310] [vq=00000000201c16ab][start=89][count=1]
[  761.926827] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=26]
[  761.926829] [vq=00000000201c16ab][start=90][count=1]
[  761.960724] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=27]
[  761.960727] [vq=00000000201c16ab][start=91][count=1]
[  762.136662] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=28]
[  762.136664] [vq=00000000201c16ab][start=92][count=1]
[  762.387928] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=29]
[  762.387931] [vq=00000000201c16ab][start=93][count=1]
[  762.425545] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=30]
[  762.425547] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=31]
[  762.425548] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=32]
[  762.425550] [vq=00000000201c16ab][start=94][count=3]
[  762.425829] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=33]
[  762.425830] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=34]
[  762.425832] [vq=00000000201c16ab][start=97][count=2]
[  762.554053] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=35]
[  762.554055] [vq=00000000201c16ab][start=99][count=1]
[  762.804248] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=36]
[  762.804250] [vq=00000000201c16ab][start=100][count=1]
[  762.863934] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=37]
[  762.863936] [vq=00000000201c16ab][start=101][count=1]
[  763.176763] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=38]
[  763.176766] [vq=00000000201c16ab][start=102][count=1]
[  763.220580] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=39]
[  763.220582] [vq=00000000201c16ab][start=103][count=1]
[  763.371347] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=40]
[  763.371350] [vq=00000000201c16ab][start=104][count=1]
[  763.371376] [vq=00000000465a3846][vq->ndescs=1]
[  763.371379] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  763.371380] [vq=00000000465a3846][vq->ndescs=0]
[  763.371384] [vq=00000000465a3846][start=39][count=1]
[  763.426476] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=41]
[  763.426479] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=42]
[  763.426481] [vq=00000000201c16ab][start=105][count=2]
[  763.426783] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=43]
[  763.426785] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=44]
[  763.426786] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=45]
[  763.426788] [vq=00000000201c16ab][start=107][count=3]
[  763.784998] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=46]
[  763.785001] [vq=00000000201c16ab][start=110][count=1]
[  763.871337] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=47]
[  763.871340] [vq=00000000201c16ab][start=111][count=1]
[  763.903944] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=48]
[  763.903946] [vq=00000000201c16ab][start=112][count=1]
[  766.596032] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=49]
[  766.596036] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=50]
[  766.596038] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=51]
[  766.596040] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=52]
[  766.596041] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=53]
[  766.596043] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=54]
[  766.596044] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=55]
[  766.596046] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=56]
[  766.596047] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=57]
[  766.596049] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=58]
[  766.596051] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=59]
[  766.596053] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=60]
[  766.596054] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=61]
[  766.596056] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=62]
[  766.596058] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=63]
[  766.596059] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=64]
[  766.596061] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=65]
[  766.596063] [vq=00000000201c16ab][vq->ndescs=0]
[  766.596064] [vq=00000000201c16ab][start=0][count=17]
[  766.596067] [vq=00000000465a3846][vq->ndescs=0]
[  767.304752] [vq=00000000201c16ab][vq->ndescs=65]
[  767.304754] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=1]
[  767.304756] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=2]
[  767.304757] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=3]
[  767.304760] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=4]
[  767.304762] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=5]
[  767.304763] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=6]
[  767.304765] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=7]
[  767.304767] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=8]
[  767.304768] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=9]
[  767.304770] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=10]
[  767.304771] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=11]
[  767.304773] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=12]
[  767.304774] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=13]
[  767.304776] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=14]
[  767.304777] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=15]
[  767.304779] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=16]
[  767.304781] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=17]
[  767.304782] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=18]
[  767.304784] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=19]
[  767.304785] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=20]
[  767.304787] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=21]
[  767.304789] [vq=00000000201c16ab][start=17][count=21]
[  767.345024] [vq=00000000465a3846][vq->ndescs=1]
[  767.345025] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  767.345027] [vq=00000000465a3846][vq->ndescs=0]
[  767.345035] [vq=00000000465a3846][start=0][count=1]
[  767.364564] [vq=00000000465a3846][vq->ndescs=1]
[  767.364566] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  767.364567] [vq=00000000465a3846][vq->ndescs=0]
[  767.364571] [vq=00000000465a3846][start=1][count=1]
[  767.365202] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=22]
[  767.365209] [vq=00000000201c16ab][start=38][count=1]
[  767.365235] [vq=00000000465a3846][vq->ndescs=2]
[  767.365237] [vq=00000000465a3846][vq->ndescs=2][vq->first_desc=1]
[  767.365239] [vq=00000000465a3846][vq->ndescs=2][vq->first_desc=2]
[  767.365241] [vq=00000000465a3846][vq->ndescs=0]
[  767.365249] [vq=00000000465a3846][start=2][count=2]
[  767.365433] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=23]
[  767.365435] [vq=00000000201c16ab][start=39][count=1]
[  767.365448] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=24]
[  767.365449] [vq=00000000201c16ab][start=40][count=1]
[  767.370391] [vq=00000000465a3846][vq->ndescs=1]
[  767.370392] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  767.370394] [vq=00000000465a3846][vq->ndescs=0]
[  767.370398] [vq=00000000465a3846][start=4][count=1]
[  767.430616] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=25]
[  767.430620] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=26]
[  767.430622] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=27]
[  767.430624] [vq=00000000201c16ab][start=41][count=3]
[  767.430875] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=28]
[  767.430876] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=29]
[  767.430878] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=30]
[  767.430879] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=31]
[  767.430880] [vq=00000000201c16ab][start=44][count=4]
[  767.571839] [vq=00000000465a3846][vq->ndescs=2]
[  767.571842] [vq=00000000465a3846][vq->ndescs=2][vq->first_desc=2]
[  767.571844] [vq=00000000465a3846][vq->ndescs=0]
[  767.571851] [vq=00000000465a3846][start=5][count=1]
[  767.597747] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=32]
[  767.597750] [vq=00000000201c16ab][start=48][count=1]
[  767.650396] [vq=00000000465a3846][vq->ndescs=1]
[  767.650398] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  767.650399] [vq=00000000465a3846][vq->ndescs=0]
[  767.650406] [vq=00000000465a3846][start=6][count=1]
[  767.749947] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=33]
[  767.749952] [vq=00000000201c16ab][start=49][count=1]
[  767.773664] [vq=00000000465a3846][vq->ndescs=2]
[  767.773666] [vq=00000000465a3846][vq->ndescs=2][vq->first_desc=2]
[  767.773667] [vq=00000000465a3846][vq->ndescs=0]
[  767.773673] [vq=00000000465a3846][start=7][count=1]
[  767.807483] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=34]
[  767.807485] [vq=00000000201c16ab][start=50][count=1]
[  767.903813] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=35]
[  767.903816] [vq=00000000201c16ab][start=51][count=1]
[  767.975773] [vq=00000000465a3846][vq->ndescs=2]
[  767.975774] [vq=00000000465a3846][vq->ndescs=2][vq->first_desc=2]
[  767.975776] [vq=00000000465a3846][vq->ndescs=0]
[  767.975781] [vq=00000000465a3846][start=8][count=1]
[  768.007409] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=36]
[  768.007411] [vq=00000000201c16ab][start=52][count=1]
[  768.031512] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=37]
[  768.031514] [vq=00000000201c16ab][start=53][count=1]
[  768.067894] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=38]
[  768.067896] [vq=00000000201c16ab][start=54][count=1]
[  768.075648] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=39]
[  768.075650] [vq=00000000201c16ab][start=55][count=1]
[  768.090397] [vq=00000000465a3846][vq->ndescs=1]
[  768.090398] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  768.090400] [vq=00000000465a3846][vq->ndescs=0]
[  768.090406] [vq=00000000465a3846][start=9][count=1]
[  768.140369] [vq=00000000465a3846][vq->ndescs=1]
[  768.140371] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  768.140373] [vq=00000000465a3846][vq->ndescs=0]
[  768.140378] [vq=00000000465a3846][start=10][count=1]
[  768.176600] [vq=00000000465a3846][vq->ndescs=2]
[  768.176601] [vq=00000000465a3846][vq->ndescs=2][vq->first_desc=2]
[  768.176603] [vq=00000000465a3846][vq->ndescs=0]
[  768.176606] [vq=00000000465a3846][start=11][count=1]
[  768.217108] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=40]
[  768.217111] [vq=00000000201c16ab][start=56][count=1]
[  768.431553] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=41]
[  768.431560] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=42]
[  768.431563] [vq=00000000201c16ab][start=57][count=2]
[  768.431869] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=43]
[  768.431870] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=44]
[  768.431871] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=45]
[  768.431872] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=46]
[  768.431874] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=47]
[  768.431875] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=48]
[  768.431876] [vq=00000000201c16ab][start=59][count=6]
[  768.858404] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=49]
[  768.858412] [vq=00000000201c16ab][start=65][count=1]
[  769.065857] [vq=00000000465a3846][vq->ndescs=1]
[  769.065859] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  769.065862] [vq=00000000465a3846][vq->ndescs=0]
[  769.065871] [vq=00000000465a3846][start=12][count=1]
[  769.066299] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=50]
[  769.066301] [vq=00000000201c16ab][start=66][count=1]
[  769.066329] [vq=00000000465a3846][vq->ndescs=1]
[  769.066330] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  769.066331] [vq=00000000465a3846][vq->ndescs=0]
[  769.066335] [vq=00000000465a3846][start=13][count=1]
[  769.066514] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=51]
[  769.066515] [vq=00000000201c16ab][start=67][count=1]
[  769.066617] [vq=00000000465a3846][vq->ndescs=1]
[  769.066618] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  769.066619] [vq=00000000465a3846][vq->ndescs=0]
[  769.066622] [vq=00000000465a3846][start=14][count=1]
[  769.066765] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=52]
[  769.066769] [vq=00000000201c16ab][start=68][count=1]
[  769.066828] [vq=00000000465a3846][vq->ndescs=1]
[  769.066829] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  769.066831] [vq=00000000465a3846][vq->ndescs=0]
[  769.066835] [vq=00000000465a3846][start=15][count=1]
[  769.066967] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=53]
[  769.066968] [vq=00000000201c16ab][start=69][count=1]
[  769.067024] [vq=00000000465a3846][vq->ndescs=1]
[  769.067025] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  769.067026] [vq=00000000465a3846][vq->ndescs=0]
[  769.067029] [vq=00000000465a3846][start=16][count=1]
[  769.067170] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=54]
[  769.067172] [vq=00000000201c16ab][start=70][count=1]
[  769.067224] [vq=00000000465a3846][vq->ndescs=1]
[  769.067225] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  769.067226] [vq=00000000465a3846][vq->ndescs=0]
[  769.067229] [vq=00000000465a3846][start=17][count=1]
[  769.067365] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=55]
[  769.067366] [vq=00000000201c16ab][start=71][count=1]
[  769.067416] [vq=00000000465a3846][vq->ndescs=1]
[  769.067417] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  769.067418] [vq=00000000465a3846][vq->ndescs=0]
[  769.067421] [vq=00000000465a3846][start=18][count=1]
[  769.067557] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=56]
[  769.067558] [vq=00000000201c16ab][start=72][count=1]
[  769.067605] [vq=00000000465a3846][vq->ndescs=1]
[  769.067606] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  769.067607] [vq=00000000465a3846][vq->ndescs=0]
[  769.067610] [vq=00000000465a3846][start=19][count=1]
[  769.067758] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=57]
[  769.067759] [vq=00000000201c16ab][start=73][count=1]
[  769.067805] [vq=00000000465a3846][vq->ndescs=1]
[  769.067806] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  769.067807] [vq=00000000465a3846][vq->ndescs=0]
[  769.067811] [vq=00000000465a3846][start=20][count=1]
[  769.067959] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=58]
[  769.067961] [vq=00000000201c16ab][start=74][count=1]
[  769.068045] [vq=00000000465a3846][vq->ndescs=1]
[  769.068047] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  769.068049] [vq=00000000465a3846][vq->ndescs=0]
[  769.068052] [vq=00000000465a3846][start=21][count=1]
[  769.068181] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=59]
[  769.068186] [vq=00000000201c16ab][start=75][count=1]
[  769.068241] [vq=00000000465a3846][vq->ndescs=1]
[  769.068242] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  769.068244] [vq=00000000465a3846][vq->ndescs=0]
[  769.068250] [vq=00000000465a3846][start=22][count=1]
[  769.068379] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=60]
[  769.068381] [vq=00000000201c16ab][start=76][count=1]
[  769.068425] [vq=00000000465a3846][vq->ndescs=1]
[  769.068426] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  769.068428] [vq=00000000465a3846][vq->ndescs=0]
[  769.068432] [vq=00000000465a3846][start=23][count=1]
[  769.068555] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=61]
[  769.068557] [vq=00000000201c16ab][start=77][count=1]
[  769.068599] [vq=00000000465a3846][vq->ndescs=1]
[  769.068600] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  769.068605] [vq=00000000465a3846][vq->ndescs=0]
[  769.068608] [vq=00000000465a3846][start=24][count=1]
[  769.068737] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=62]
[  769.068739] [vq=00000000201c16ab][start=78][count=1]
[  769.068781] [vq=00000000465a3846][vq->ndescs=1]
[  769.068782] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  769.068784] [vq=00000000465a3846][vq->ndescs=0]
[  769.068787] [vq=00000000465a3846][start=25][count=1]
[  769.068914] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=63]
[  769.068916] [vq=00000000201c16ab][start=79][count=1]
[  769.068959] [vq=00000000465a3846][vq->ndescs=1]
[  769.068960] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  769.068962] [vq=00000000465a3846][vq->ndescs=0]
[  769.068965] [vq=00000000465a3846][start=26][count=1]
[  769.069094] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=64]
[  769.069097] [vq=00000000201c16ab][start=80][count=1]
[  769.069139] [vq=00000000465a3846][vq->ndescs=1]
[  769.069140] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  769.069143] [vq=00000000465a3846][vq->ndescs=0]
[  769.069145] [vq=00000000465a3846][start=27][count=1]
[  769.069271] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=65]
[  769.069274] [vq=00000000201c16ab][start=81][count=1]
[  769.069316] [vq=00000000465a3846][vq->ndescs=1]
[  769.069317] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  769.069319] [vq=00000000465a3846][vq->ndescs=0]
[  769.069322] [vq=00000000465a3846][start=28][count=1]
[  769.069452] [vq=00000000201c16ab][vq->ndescs=65]
[  769.069453] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=1]
[  769.069455] [vq=00000000201c16ab][start=82][count=1]
[  769.069499] [vq=00000000465a3846][vq->ndescs=1]
[  769.069500] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  769.069501] [vq=00000000465a3846][vq->ndescs=0]
[  769.069504] [vq=00000000465a3846][start=29][count=1]
[  769.069650] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=2]
[  769.069654] [vq=00000000201c16ab][start=83][count=1]
[  769.069711] [vq=00000000465a3846][vq->ndescs=1]
[  769.069712] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  769.069714] [vq=00000000465a3846][vq->ndescs=0]
[  769.069718] [vq=00000000465a3846][start=30][count=1]
[  769.069851] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=3]
[  769.069852] [vq=00000000201c16ab][start=84][count=1]
[  769.069905] [vq=00000000465a3846][vq->ndescs=1]
[  769.069906] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  769.069908] [vq=00000000465a3846][vq->ndescs=0]
[  769.069910] [vq=00000000465a3846][start=31][count=1]
[  769.070045] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=4]
[  769.070046] [vq=00000000201c16ab][start=85][count=1]
[  769.070115] [vq=00000000465a3846][vq->ndescs=1]
[  769.070116] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  769.070117] [vq=00000000465a3846][vq->ndescs=0]
[  769.070120] [vq=00000000465a3846][start=32][count=1]
[  769.070253] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=5]
[  769.070254] [vq=00000000201c16ab][start=86][count=1]
[  769.093873] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=6]
[  769.093878] [vq=00000000201c16ab][start=87][count=1]
[  769.107789] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=7]
[  769.107791] [vq=00000000201c16ab][start=88][count=1]
[  769.130423] [vq=00000000465a3846][vq->ndescs=1]
[  769.130424] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  769.130426] [vq=00000000465a3846][vq->ndescs=0]
[  769.130437] [vq=00000000465a3846][start=33][count=1]
[  769.133437] [vq=00000000465a3846][vq->ndescs=1]
[  769.133438] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  769.133440] [vq=00000000465a3846][vq->ndescs=0]
[  769.133443] [vq=00000000465a3846][start=34][count=1]
[  769.170381] [vq=00000000465a3846][vq->ndescs=1]
[  769.170383] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  769.170384] [vq=00000000465a3846][vq->ndescs=0]
[  769.170388] [vq=00000000465a3846][start=35][count=1]
[  769.370375] [vq=00000000465a3846][vq->ndescs=1]
[  769.370377] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  769.370380] [vq=00000000465a3846][vq->ndescs=0]
[  769.370386] [vq=00000000465a3846][start=36][count=1]
[  769.432770] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=8]
[  769.432773] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=9]
[  769.432775] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=10]
[  769.432776] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=11]
[  769.432778] [vq=00000000201c16ab][start=89][count=4]
[  769.432999] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=12]
[  769.433001] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=13]
[  769.433003] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=14]
[  769.433005] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=15]
[  769.433006] [vq=00000000201c16ab][start=93][count=4]
[  769.500358] [vq=00000000465a3846][vq->ndescs=1]
[  769.500361] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  769.500363] [vq=00000000465a3846][vq->ndescs=0]
[  769.500368] [vq=00000000465a3846][start=37][count=1]
[  769.855119] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=16]
[  769.855123] [vq=00000000201c16ab][start=97][count=1]
[  769.855886] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=17]
[  769.855888] [vq=00000000201c16ab][start=98][count=1]
[  770.136903] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=18]
[  770.136906] [vq=00000000201c16ab][start=99][count=1]
[  770.147613] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=19]
[  770.147616] [vq=00000000201c16ab][start=100][count=1]
[  770.433405] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=20]
[  770.433408] [vq=00000000201c16ab][start=101][count=1]
[  770.433716] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=21]
[  770.433718] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=22]
[  770.433720] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=23]
[  770.433721] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=24]
[  770.433723] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=25]
[  770.433724] [vq=00000000201c16ab][start=102][count=5]
[  770.433903] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=26]
[  770.433904] [vq=00000000201c16ab][start=107][count=1]
[  770.830869] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=27]
[  770.830874] [vq=00000000201c16ab][start=108][count=1]
[  770.863937] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=28]
[  770.863939] [vq=00000000201c16ab][start=109][count=1]
[  770.921316] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=29]
[  770.921319] [vq=00000000201c16ab][start=110][count=1]
[  771.176836] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=30]
[  771.176839] [vq=00000000201c16ab][start=111][count=1]
[  771.434412] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=31]
[  771.434415] [vq=00000000201c16ab][start=112][count=1]
[  771.434701] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=32]
[  771.434704] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=33]
[  771.434705] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=34]
[  771.434707] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=35]
[  771.434708] [vq=00000000201c16ab][start=113][count=4]
[  771.434885] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=36]
[  771.434887] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=37]
[  771.434888] [vq=00000000201c16ab][start=117][count=2]
[  771.904063] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=38]
[  771.904066] [vq=00000000201c16ab][start=119][count=1]
[  771.961232] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=39]
[  771.961234] [vq=00000000201c16ab][start=120][count=1]
[  772.435587] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=40]
[  772.435590] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=41]
[  772.435592] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=42]
[  772.435593] [vq=00000000201c16ab][start=121][count=3]
[  772.435862] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=43]
[  772.435864] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=44]
[  772.435865] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=45]
[  772.435867] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=46]
[  772.435868] [vq=00000000201c16ab][start=124][count=4]
[  775.057134] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=47]
[  775.057139] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=48]
[  775.057141] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=49]
[  775.057143] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=50]
[  775.057144] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=51]
[  775.057146] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=52]
[  775.057148] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=53]
[  775.057149] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=54]
[  775.057151] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=55]
[  775.057153] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=56]
[  775.057155] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=57]
[  775.057156] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=58]
[  775.057158] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=59]
[  775.057160] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=60]
[  775.057162] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=61]
[  775.057163] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=62]
[  775.057165] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=63]
[  775.057166] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=64]
[  775.057168] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=65]
[  775.057170] [vq=00000000201c16ab][vq->ndescs=0]
[  775.057172] [vq=00000000201c16ab][start=0][count=19]
[  775.057175] [vq=00000000465a3846][vq->ndescs=0]
[  775.746716] [vq=00000000201c16ab][vq->ndescs=65]
[  775.746717] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=1]
[  775.746719] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=2]
[  775.746721] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=3]
[  775.746722] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=4]
[  775.746723] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=5]
[  775.746725] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=6]
[  775.746726] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=7]
[  775.746727] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=8]
[  775.746729] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=9]
[  775.746730] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=10]
[  775.746731] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=11]
[  775.746732] [vq=00000000201c16ab][start=19][count=11]
[  775.779189] [vq=00000000465a3846][vq->ndescs=1]
[  775.779191] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  775.779193] [vq=00000000465a3846][vq->ndescs=0]
[  775.779201] [vq=00000000465a3846][start=0][count=1]
[  775.794833] [vq=00000000465a3846][vq->ndescs=1]
[  775.794835] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  775.794836] [vq=00000000465a3846][vq->ndescs=0]
[  775.794842] [vq=00000000465a3846][start=1][count=1]
[  775.803247] [vq=00000000465a3846][vq->ndescs=1]
[  775.803248] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  775.803249] [vq=00000000465a3846][vq->ndescs=0]
[  775.803253] [vq=00000000465a3846][start=2][count=1]
[  775.803932] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=12]
[  775.803938] [vq=00000000201c16ab][start=30][count=1]
[  775.803977] [vq=00000000465a3846][vq->ndescs=2]
[  775.803998] [vq=00000000465a3846][vq->ndescs=2][vq->first_desc=1]
[  775.804000] [vq=00000000465a3846][vq->ndescs=2][vq->first_desc=2]
[  775.804002] [vq=00000000465a3846][vq->ndescs=0]
[  775.804010] [vq=00000000465a3846][start=3][count=2]
[  775.804194] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=13]
[  775.804196] [vq=00000000201c16ab][start=31][count=1]
[  775.804205] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=14]
[  775.804207] [vq=00000000201c16ab][start=32][count=1]
[  775.913969] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=15]
[  775.913974] [vq=00000000201c16ab][start=33][count=1]
[  775.997471] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=16]
[  775.997474] [vq=00000000201c16ab][start=34][count=1]
[  776.006405] [vq=00000000465a3846][vq->ndescs=2]
[  776.006407] [vq=00000000465a3846][vq->ndescs=2][vq->first_desc=2]
[  776.006409] [vq=00000000465a3846][vq->ndescs=0]
[  776.006416] [vq=00000000465a3846][start=5][count=1]
[  776.036919] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=17]
[  776.036924] [vq=00000000201c16ab][start=35][count=1]
[  776.046852] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=18]
[  776.046856] [vq=00000000201c16ab][start=36][count=1]
[  776.061540] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=19]
[  776.061542] [vq=00000000201c16ab][start=37][count=1]
[  776.111415] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=20]
[  776.111419] [vq=00000000201c16ab][start=38][count=1]
[  776.154841] [vq=00000000465a3846][vq->ndescs=1]
[  776.154843] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  776.154845] [vq=00000000465a3846][vq->ndescs=0]
[  776.154850] [vq=00000000465a3846][start=6][count=1]
[  776.209078] [vq=00000000465a3846][vq->ndescs=2]
[  776.209079] [vq=00000000465a3846][vq->ndescs=2][vq->first_desc=2]
[  776.209080] [vq=00000000465a3846][vq->ndescs=0]
[  776.209083] [vq=00000000465a3846][start=7][count=1]
[  776.234725] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=21]
[  776.234727] [vq=00000000201c16ab][start=39][count=1]
[  776.306637] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=22]
[  776.306639] [vq=00000000201c16ab][start=40][count=1]
[  776.394831] [vq=00000000465a3846][vq->ndescs=1]
[  776.394832] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  776.394833] [vq=00000000465a3846][vq->ndescs=0]
[  776.394837] [vq=00000000465a3846][start=8][count=1]
[  776.411908] [vq=00000000465a3846][vq->ndescs=2]
[  776.411910] [vq=00000000465a3846][vq->ndescs=2][vq->first_desc=2]
[  776.411911] [vq=00000000465a3846][vq->ndescs=0]
[  776.411914] [vq=00000000465a3846][start=9][count=1]
[  776.439659] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=23]
[  776.439660] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=24]
[  776.439661] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=25]
[  776.439663] [vq=00000000201c16ab][start=41][count=3]
[  776.439929] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=26]
[  776.439931] [vq=00000000201c16ab][start=44][count=1]
[  776.448982] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=27]
[  776.448984] [vq=00000000201c16ab][start=45][count=1]
[  776.504845] [vq=00000000465a3846][vq->ndescs=1]
[  776.504846] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  776.504848] [vq=00000000465a3846][vq->ndescs=0]
[  776.504851] [vq=00000000465a3846][start=10][count=1]
[  776.615546] [vq=00000000465a3846][vq->ndescs=2]
[  776.615548] [vq=00000000465a3846][vq->ndescs=2][vq->first_desc=2]
[  776.615549] [vq=00000000465a3846][vq->ndescs=0]
[  776.615551] [vq=00000000465a3846][start=11][count=1]
[  776.650494] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=28]
[  776.650495] [vq=00000000201c16ab][start=46][count=1]
[  777.099061] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=29]
[  777.099068] [vq=00000000201c16ab][start=47][count=1]
[  777.151245] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=30]
[  777.151247] [vq=00000000201c16ab][start=48][count=1]
[  777.283424] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=31]
[  777.283427] [vq=00000000201c16ab][start=49][count=1]
[  777.337441] [vq=00000000465a3846][vq->ndescs=1]
[  777.337442] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  777.337443] [vq=00000000465a3846][vq->ndescs=0]
[  777.337447] [vq=00000000465a3846][start=12][count=1]
[  777.337944] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=32]
[  777.337948] [vq=00000000201c16ab][start=50][count=1]
[  777.337968] [vq=00000000465a3846][vq->ndescs=1]
[  777.337969] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  777.337971] [vq=00000000465a3846][vq->ndescs=0]
[  777.337975] [vq=00000000465a3846][start=13][count=1]
[  777.338197] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=33]
[  777.338198] [vq=00000000201c16ab][start=51][count=1]
[  777.338238] [vq=00000000465a3846][vq->ndescs=1]
[  777.338239] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  777.338241] [vq=00000000465a3846][vq->ndescs=0]
[  777.338244] [vq=00000000465a3846][start=14][count=1]
[  777.338382] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=34]
[  777.338383] [vq=00000000201c16ab][start=52][count=1]
[  777.338415] [vq=00000000465a3846][vq->ndescs=1]
[  777.338416] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  777.338417] [vq=00000000465a3846][vq->ndescs=0]
[  777.338419] [vq=00000000465a3846][start=15][count=1]
[  777.338556] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=35]
[  777.338557] [vq=00000000201c16ab][start=53][count=1]
[  777.338588] [vq=00000000465a3846][vq->ndescs=1]
[  777.338589] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  777.338600] [vq=00000000465a3846][vq->ndescs=0]
[  777.338603] [vq=00000000465a3846][start=16][count=1]
[  777.338760] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=36]
[  777.338762] [vq=00000000201c16ab][start=54][count=1]
[  777.338795] [vq=00000000465a3846][vq->ndescs=1]
[  777.338796] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  777.338797] [vq=00000000465a3846][vq->ndescs=0]
[  777.338799] [vq=00000000465a3846][start=17][count=1]
[  777.338939] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=37]
[  777.338940] [vq=00000000201c16ab][start=55][count=1]
[  777.338975] [vq=00000000465a3846][vq->ndescs=1]
[  777.338977] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  777.338979] [vq=00000000465a3846][vq->ndescs=0]
[  777.338983] [vq=00000000465a3846][start=18][count=1]
[  777.339124] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=38]
[  777.339126] [vq=00000000201c16ab][start=56][count=1]
[  777.339158] [vq=00000000465a3846][vq->ndescs=1]
[  777.339159] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  777.339160] [vq=00000000465a3846][vq->ndescs=0]
[  777.339163] [vq=00000000465a3846][start=19][count=1]
[  777.339314] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=39]
[  777.339316] [vq=00000000201c16ab][start=57][count=1]
[  777.339345] [vq=00000000465a3846][vq->ndescs=1]
[  777.339346] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  777.339348] [vq=00000000465a3846][vq->ndescs=0]
[  777.339350] [vq=00000000465a3846][start=20][count=1]
[  777.339489] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=40]
[  777.339490] [vq=00000000201c16ab][start=58][count=1]
[  777.339521] [vq=00000000465a3846][vq->ndescs=1]
[  777.339522] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  777.339524] [vq=00000000465a3846][vq->ndescs=0]
[  777.339526] [vq=00000000465a3846][start=21][count=1]
[  777.339662] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=41]
[  777.339664] [vq=00000000201c16ab][start=59][count=1]
[  777.339694] [vq=00000000465a3846][vq->ndescs=1]
[  777.339695] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  777.339696] [vq=00000000465a3846][vq->ndescs=0]
[  777.339699] [vq=00000000465a3846][start=22][count=1]
[  777.339842] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=42]
[  777.339843] [vq=00000000201c16ab][start=60][count=1]
[  777.339874] [vq=00000000465a3846][vq->ndescs=1]
[  777.339876] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  777.339877] [vq=00000000465a3846][vq->ndescs=0]
[  777.339879] [vq=00000000465a3846][start=23][count=1]
[  777.340015] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=43]
[  777.340016] [vq=00000000201c16ab][start=61][count=1]
[  777.340045] [vq=00000000465a3846][vq->ndescs=1]
[  777.340046] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  777.340048] [vq=00000000465a3846][vq->ndescs=0]
[  777.340050] [vq=00000000465a3846][start=24][count=1]
[  777.340187] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=44]
[  777.340188] [vq=00000000201c16ab][start=62][count=1]
[  777.340217] [vq=00000000465a3846][vq->ndescs=1]
[  777.340218] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  777.340220] [vq=00000000465a3846][vq->ndescs=0]
[  777.340222] [vq=00000000465a3846][start=25][count=1]
[  777.340358] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=45]
[  777.340360] [vq=00000000201c16ab][start=63][count=1]
[  777.340389] [vq=00000000465a3846][vq->ndescs=1]
[  777.340390] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  777.340391] [vq=00000000465a3846][vq->ndescs=0]
[  777.340393] [vq=00000000465a3846][start=26][count=1]
[  777.340528] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=46]
[  777.340529] [vq=00000000201c16ab][start=64][count=1]
[  777.340559] [vq=00000000465a3846][vq->ndescs=1]
[  777.340560] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  777.340561] [vq=00000000465a3846][vq->ndescs=0]
[  777.340564] [vq=00000000465a3846][start=27][count=1]
[  777.340697] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=47]
[  777.340698] [vq=00000000201c16ab][start=65][count=1]
[  777.340730] [vq=00000000465a3846][vq->ndescs=1]
[  777.340731] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  777.340732] [vq=00000000465a3846][vq->ndescs=0]
[  777.340734] [vq=00000000465a3846][start=28][count=1]
[  777.340870] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=48]
[  777.340871] [vq=00000000201c16ab][start=66][count=1]
[  777.340900] [vq=00000000465a3846][vq->ndescs=1]
[  777.340901] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  777.340902] [vq=00000000465a3846][vq->ndescs=0]
[  777.340904] [vq=00000000465a3846][start=29][count=1]
[  777.341038] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=49]
[  777.341039] [vq=00000000201c16ab][start=67][count=1]
[  777.341068] [vq=00000000465a3846][vq->ndescs=1]
[  777.341069] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  777.341070] [vq=00000000465a3846][vq->ndescs=0]
[  777.341073] [vq=00000000465a3846][start=30][count=1]
[  777.341209] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=50]
[  777.341210] [vq=00000000201c16ab][start=68][count=1]
[  777.341239] [vq=00000000465a3846][vq->ndescs=1]
[  777.341240] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  777.341242] [vq=00000000465a3846][vq->ndescs=0]
[  777.341244] [vq=00000000465a3846][start=31][count=1]
[  777.341379] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=51]
[  777.341381] [vq=00000000201c16ab][start=69][count=1]
[  777.341410] [vq=00000000465a3846][vq->ndescs=1]
[  777.341411] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  777.341412] [vq=00000000465a3846][vq->ndescs=0]
[  777.341415] [vq=00000000465a3846][start=32][count=1]
[  777.341550] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=52]
[  777.341551] [vq=00000000201c16ab][start=70][count=1]
[  777.347869] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=53]
[  777.347874] [vq=00000000201c16ab][start=71][count=1]
[  777.404861] [vq=00000000465a3846][vq->ndescs=1]
[  777.404863] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  777.404864] [vq=00000000465a3846][vq->ndescs=0]
[  777.404872] [vq=00000000465a3846][start=33][count=1]
[  777.434829] [vq=00000000465a3846][vq->ndescs=1]
[  777.434831] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  777.434832] [vq=00000000465a3846][vq->ndescs=0]
[  777.434836] [vq=00000000465a3846][start=34][count=1]
[  777.440676] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=54]
[  777.440678] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=55]
[  777.440679] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=56]
[  777.440681] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=57]
[  777.440682] [vq=00000000201c16ab][start=72][count=4]
[  777.440939] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=58]
[  777.440940] [vq=00000000201c16ab][start=76][count=1]
[  777.844890] [vq=00000000465a3846][vq->ndescs=1]
[  777.844893] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  777.844896] [vq=00000000465a3846][vq->ndescs=0]
[  777.844905] [vq=00000000465a3846][start=35][count=1]
[  777.861124] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=59]
[  777.861127] [vq=00000000201c16ab][start=77][count=1]
[  777.892947] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=60]
[  777.892950] [vq=00000000201c16ab][start=78][count=1]
[  778.136716] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=61]
[  778.136718] [vq=00000000201c16ab][start=79][count=1]
[  778.201376] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=62]
[  778.201378] [vq=00000000201c16ab][start=80][count=1]
[  778.204884] [vq=00000000465a3846][vq->ndescs=1]
[  778.204885] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  778.204887] [vq=00000000465a3846][vq->ndescs=0]
[  778.204892] [vq=00000000465a3846][start=36][count=1]
[  778.335800] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=63]
[  778.335803] [vq=00000000201c16ab][start=81][count=1]
[  778.387704] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=64]
[  778.387706] [vq=00000000201c16ab][start=82][count=1]
[  778.441538] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=65]
[  778.441543] [vq=00000000201c16ab][vq->ndescs=65]
[  778.441545] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=1]
[  778.441546] [vq=00000000201c16ab][start=83][count=2]
[  778.441804] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=2]
[  778.441805] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=3]
[  778.441807] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=4]
[  778.441808] [vq=00000000201c16ab][start=85][count=3]
[  778.808385] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=5]
[  778.808390] [vq=00000000201c16ab][start=88][count=1]
[  778.863990] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=6]
[  778.863992] [vq=00000000201c16ab][start=89][count=1]
[  778.907345] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=7]
[  778.907348] [vq=00000000201c16ab][start=90][count=1]
[  779.176815] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=8]
[  779.176819] [vq=00000000201c16ab][start=91][count=1]
[  779.358511] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=9]
[  779.358513] [vq=00000000201c16ab][start=92][count=1]
[  779.437933] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=10]
[  779.437935] [vq=00000000201c16ab][start=93][count=1]
[  779.442422] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=11]
[  779.442424] [vq=00000000201c16ab][start=94][count=1]
[  779.442665] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=12]
[  779.442667] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=13]
[  779.442668] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=14]
[  779.442669] [vq=00000000201c16ab][start=95][count=3]
[  779.904029] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=15]
[  779.904032] [vq=00000000201c16ab][start=98][count=1]
[  780.443688] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=16]
[  780.443691] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=17]
[  780.443692] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=18]
[  780.443694] [vq=00000000201c16ab][start=99][count=3]
[  780.443985] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=19]
[  780.443987] [vq=00000000201c16ab][start=102][count=1]
[  780.467631] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=20]
[  780.467633] [vq=00000000201c16ab][start=103][count=1]
[  780.651623] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=21]
[  780.651626] [vq=00000000201c16ab][start=104][count=1]
[  783.326504] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=22]
[  783.326510] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=23]
[  783.326512] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=24]
[  783.326514] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=25]
[  783.326515] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=26]
[  783.326517] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=27]
[  783.326518] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=28]
[  783.326520] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=29]
[  783.326522] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=30]
[  783.326523] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=31]
[  783.326524] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=32]
[  783.326526] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=33]
[  783.326528] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=34]
[  783.326530] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=35]
[  783.326532] [vq=00000000201c16ab][start=0][count=14]
[  783.326535] [vq=00000000465a3846][vq->ndescs=0]
[  783.446605] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=36]
[  783.446608] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=37]
[  783.446610] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=38]
[  783.446611] [vq=00000000201c16ab][start=14][count=3]
[  783.903893] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=39]
[  783.903896] [vq=00000000201c16ab][start=17][count=1]
[  784.022786] [vq=00000000465a3846][vq->ndescs=1]
[  784.022788] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  784.022790] [vq=00000000465a3846][vq->ndescs=0]
[  784.022797] [vq=00000000465a3846][start=0][count=1]
[  784.039300] [vq=00000000465a3846][vq->ndescs=1]
[  784.039301] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  784.039303] [vq=00000000465a3846][vq->ndescs=0]
[  784.039306] [vq=00000000465a3846][start=1][count=1]
[  784.039827] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=40]
[  784.039829] [vq=00000000201c16ab][start=18][count=1]
[  784.044823] [vq=00000000465a3846][vq->ndescs=1]
[  784.044824] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  784.044826] [vq=00000000465a3846][vq->ndescs=0]
[  784.044831] [vq=00000000465a3846][start=2][count=1]
[  784.167484] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=41]
[  784.167486] [vq=00000000201c16ab][start=19][count=1]
[  784.329413] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=42]
[  784.329416] [vq=00000000201c16ab][start=20][count=1]
[  784.414819] [vq=00000000465a3846][vq->ndescs=1]
[  784.414820] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  784.414822] [vq=00000000465a3846][vq->ndescs=0]
[  784.414826] [vq=00000000465a3846][start=3][count=1]
[  784.447615] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=43]
[  784.447618] [vq=00000000201c16ab][start=21][count=1]
[  784.447900] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=44]
[  784.447902] [vq=00000000201c16ab][start=22][count=1]
[  784.934838] [vq=00000000465a3846][vq->ndescs=1]
[  784.934840] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  784.934841] [vq=00000000465a3846][vq->ndescs=0]
[  784.934845] [vq=00000000465a3846][start=4][count=1]
[  785.100943] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=45]
[  785.100946] [vq=00000000201c16ab][start=23][count=1]
[  785.104788] [vq=00000000465a3846][vq->ndescs=1]
[  785.104789] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  785.104791] [vq=00000000465a3846][vq->ndescs=0]
[  785.104797] [vq=00000000465a3846][start=5][count=1]
[  785.105315] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=46]
[  785.105316] [vq=00000000201c16ab][start=24][count=1]
[  785.264810] [vq=00000000465a3846][vq->ndescs=1]
[  785.264812] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  785.264813] [vq=00000000465a3846][vq->ndescs=0]
[  785.264817] [vq=00000000465a3846][start=6][count=1]
[  785.448591] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=47]
[  785.448593] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=48]
[  785.448595] [vq=00000000201c16ab][start=25][count=2]
[  785.812091] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=49]
[  785.812093] [vq=00000000201c16ab][start=27][count=1]
[  785.865622] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=50]
[  785.865624] [vq=00000000201c16ab][start=28][count=1]
[  785.956012] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=51]
[  785.956014] [vq=00000000201c16ab][start=29][count=1]
[  785.984841] [vq=00000000465a3846][vq->ndescs=1]
[  785.984843] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  785.984844] [vq=00000000465a3846][vq->ndescs=0]
[  785.984849] [vq=00000000465a3846][start=7][count=1]
[  785.985751] [vq=00000000465a3846][vq->ndescs=1]
[  785.985752] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  785.985753] [vq=00000000465a3846][vq->ndescs=0]
[  785.985756] [vq=00000000465a3846][start=8][count=1]
[  786.022926] [vq=00000000465a3846][vq->ndescs=1]
[  786.022927] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  786.022928] [vq=00000000465a3846][vq->ndescs=0]
[  786.022931] [vq=00000000465a3846][start=9][count=1]
[  786.024797] [vq=00000000465a3846][vq->ndescs=1]
[  786.024798] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  786.024799] [vq=00000000465a3846][vq->ndescs=0]
[  786.024802] [vq=00000000465a3846][start=10][count=1]
[  786.054795] [vq=00000000465a3846][vq->ndescs=1]
[  786.054796] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  786.054798] [vq=00000000465a3846][vq->ndescs=0]
[  786.054802] [vq=00000000465a3846][start=11][count=1]
[  786.136759] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=52]
[  786.136761] [vq=00000000201c16ab][start=30][count=1]
[  786.144783] [vq=00000000465a3846][vq->ndescs=1]
[  786.144784] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  786.144786] [vq=00000000465a3846][vq->ndescs=0]
[  786.144790] [vq=00000000465a3846][start=12][count=1]
[  786.145388] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=53]
[  786.145389] [vq=00000000201c16ab][start=31][count=1]
[  786.166419] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=54]
[  786.166421] [vq=00000000201c16ab][start=32][count=1]
[  786.449605] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=55]
[  786.449608] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=56]
[  786.449610] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=57]
[  786.449611] [vq=00000000201c16ab][start=33][count=3]
[  786.831294] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=58]
[  786.831296] [vq=00000000201c16ab][start=36][count=1]
[  786.943969] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=59]
[  786.943971] [vq=00000000201c16ab][start=37][count=1]
[  786.944849] [vq=00000000465a3846][vq->ndescs=1]
[  786.944850] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  786.944853] [vq=00000000465a3846][vq->ndescs=0]
[  786.944858] [vq=00000000465a3846][start=13][count=1]
[  786.980009] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=60]
[  786.980011] [vq=00000000201c16ab][start=38][count=1]
[  787.155382] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=61]
[  787.155386] [vq=00000000201c16ab][start=39][count=1]
[  787.176736] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=62]
[  787.176738] [vq=00000000201c16ab][start=40][count=1]
[  787.185039] [vq=00000000465a3846][vq->ndescs=1]
[  787.185040] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  787.185042] [vq=00000000465a3846][vq->ndescs=0]
[  787.185050] [vq=00000000465a3846][start=14][count=1]
[  787.185639] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=63]
[  787.185641] [vq=00000000201c16ab][start=41][count=1]
[  787.450691] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=64]
[  787.450693] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=65]
[  787.450695] [vq=00000000201c16ab][start=42][count=2]
[  787.485512] Guest moved vq 00000000201c16ab used index from 0 to 289
[  787.485513] [vq=00000000201c16ab][vq->ndescs=0]
[  788.023034] [vq=00000000465a3846][vq->ndescs=1]
[  788.023036] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  788.023037] [vq=00000000465a3846][vq->ndescs=0]
[  788.023042] [vq=00000000465a3846][start=15][count=1]
[  788.224798] [vq=00000000465a3846][vq->ndescs=1]
[  788.224800] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  788.224802] [vq=00000000465a3846][vq->ndescs=0]
[  788.224806] [vq=00000000465a3846][start=16][count=1]
[  789.264807] [vq=00000000465a3846][vq->ndescs=1]
[  789.264809] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  789.264811] [vq=00000000465a3846][vq->ndescs=0]
[  789.264816] [vq=00000000465a3846][start=17][count=1]
[  789.885117] [vq=00000000465a3846][vq->ndescs=1]
[  789.885118] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  789.885120] [vq=00000000465a3846][vq->ndescs=0]
[  789.885124] [vq=00000000465a3846][start=18][count=1]
[  790.304957] [vq=00000000465a3846][vq->ndescs=1]
[  790.304959] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  790.304961] [vq=00000000465a3846][vq->ndescs=0]
[  790.304967] [vq=00000000465a3846][start=19][count=1]
[  791.344839] [vq=00000000465a3846][vq->ndescs=1]
[  791.344842] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  791.344843] [vq=00000000465a3846][vq->ndescs=0]
[  791.344850] [vq=00000000465a3846][start=20][count=1]
[  792.384804] [vq=00000000465a3846][vq->ndescs=1]
[  792.384807] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  792.384808] [vq=00000000465a3846][vq->ndescs=0]
[  792.384814] [vq=00000000465a3846][start=21][count=1]
[  793.424946] [vq=00000000465a3846][vq->ndescs=1]
[  793.424948] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  793.424950] [vq=00000000465a3846][vq->ndescs=0]
[  793.424955] [vq=00000000465a3846][start=22][count=1]
[  793.886441] [vq=00000000465a3846][vq->ndescs=1]
[  793.886442] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  793.886444] [vq=00000000465a3846][vq->ndescs=0]
[  793.886449] [vq=00000000465a3846][start=23][count=1]
[  794.464802] [vq=00000000465a3846][vq->ndescs=1]
[  794.464804] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  794.464806] [vq=00000000465a3846][vq->ndescs=0]
[  794.464811] [vq=00000000465a3846][start=24][count=1]
[  795.504831] [vq=00000000465a3846][vq->ndescs=1]
[  795.504835] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  795.504838] [vq=00000000465a3846][vq->ndescs=0]

Re: vhost changes (batched) in linux-next after 12/13 trigger random crashes in KVM guests after reboot

[Eugenio Pérez]

On Mon, 2020-02-10 at 12:01 +0100, Christian Borntraeger wrote:
> 
> On 10.02.20 10:47, Eugenio Perez Martin wrote:
> > Hi Christian.
> > 
> > I'm not able to reproduce the failure with eccb852f1fe6bede630e2e4f1a121a81e34354ab commit. Could you add more data?
> > Your configuration (libvirt or qemu line), and host's dmesg output if any?
> > 
> > Thanks!
> 
> If it was not obvious, this is on s390x, a big endian system.
> 

Hi Christian. Thank you very much for your fast responses.

Could you try this patch on top of eccb852f1fe6bede630e2e4f1a121a81e34354ab?

Thanks!

From 71d0f9108a18aa894cc0c0c1c7efbad39f465a27 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Eugenio=20P=C3=A9rez?= <
eperezma@redhat.com>
Date: Tue, 11 Feb 2020 13:19:10 +0100
Subject: [PATCH] vhost: fix return value of vhost_get_vq_desc

Before of the batch change, it was the chain's head. Need to keep that
way or we will not be able to free a chain of descriptors.

Fixes: eccb852f1fe6 ("vhost: batching fetches")
---
 drivers/vhost/vhost.c | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)

diff --git a/drivers/vhost/vhost.c b/drivers/vhost/vhost.c
index b5a51b1f2e79..fc422c3e5c08 100644
--- a/drivers/vhost/vhost.c
+++ b/drivers/vhost/vhost.c
@@ -2409,12 +2409,11 @@ int vhost_get_vq_desc(struct vhost_virtqueue *vq,
 			*out_num += ret;
 		}
 
-		ret = desc->id;
-
 		if (!(desc->flags & VRING_DESC_F_NEXT))
 			break;
 	}
 
+	ret = vq->descs[vq->first_desc].id;
 	vq->first_desc = i + 1;
 
 	return ret;
-- 
2.18.1

Re: vhost changes (batched) in linux-next after 12/13 trigger random crashes in KVM guests after reboot

[Christian Borntraeger]

On 11.02.20 14:04, Eugenio Pérez wrote:
> On Mon, 2020-02-10 at 12:01 +0100, Christian Borntraeger wrote:
>>
>> On 10.02.20 10:47, Eugenio Perez Martin wrote:
>>> Hi Christian.
>>>
>>> I'm not able to reproduce the failure with eccb852f1fe6bede630e2e4f1a121a81e34354ab commit. Could you add more data?
>>> Your configuration (libvirt or qemu line), and host's dmesg output if any?
>>>
>>> Thanks!
>>
>> If it was not obvious, this is on s390x, a big endian system.
>>
> 
> Hi Christian. Thank you very much for your fast responses.
> 
> Could you try this patch on top of eccb852f1fe6bede630e2e4f1a121a81e34354ab?

I still get 
[   43.665145] Guest moved used index from 0 to 289
after some reboots.


> 
> Thanks!
> 
> From 71d0f9108a18aa894cc0c0c1c7efbad39f465a27 Mon Sep 17 00:00:00 2001
> From: =?UTF-8?q?Eugenio=20P=C3=A9rez?= <
> eperezma@redhat.com>
> Date: Tue, 11 Feb 2020 13:19:10 +0100
> Subject: [PATCH] vhost: fix return value of vhost_get_vq_desc
> 
> Before of the batch change, it was the chain's head. Need to keep that
> way or we will not be able to free a chain of descriptors.
> 
> Fixes: eccb852f1fe6 ("vhost: batching fetches")
> ---
>  drivers/vhost/vhost.c | 3 +--
>  1 file changed, 1 insertion(+), 2 deletions(-)
> 
> diff --git a/drivers/vhost/vhost.c b/drivers/vhost/vhost.c
> index b5a51b1f2e79..fc422c3e5c08 100644
> --- a/drivers/vhost/vhost.c
> +++ b/drivers/vhost/vhost.c
> @@ -2409,12 +2409,11 @@ int vhost_get_vq_desc(struct vhost_virtqueue *vq,
>  			*out_num += ret;
>  		}
>  
> -		ret = desc->id;
> -
>  		if (!(desc->flags & VRING_DESC_F_NEXT))
>  			break;
>  	}
>  
> +	ret = vq->descs[vq->first_desc].id;
>  	vq->first_desc = i + 1;
>  
>  	return ret;
> 

Re: vhost changes (batched) in linux-next after 12/13 trigger random crashes in KVM guests after reboot

[Michael S. Tsirkin]

On Tue, Feb 11, 2020 at 02:04:54PM +0100, Eugenio Pérez wrote:
> On Mon, 2020-02-10 at 12:01 +0100, Christian Borntraeger wrote:
> > 
> > On 10.02.20 10:47, Eugenio Perez Martin wrote:
> > > Hi Christian.
> > > 
> > > I'm not able to reproduce the failure with eccb852f1fe6bede630e2e4f1a121a81e34354ab commit. Could you add more data?
> > > Your configuration (libvirt or qemu line), and host's dmesg output if any?
> > > 
> > > Thanks!
> > 
> > If it was not obvious, this is on s390x, a big endian system.
> > 
> 
> Hi Christian. Thank you very much for your fast responses.
> 
> Could you try this patch on top of eccb852f1fe6bede630e2e4f1a121a81e34354ab?
> 
> Thanks!
> 
> >From 71d0f9108a18aa894cc0c0c1c7efbad39f465a27 Mon Sep 17 00:00:00 2001
> From: =?UTF-8?q?Eugenio=20P=C3=A9rez?= <
> eperezma@redhat.com>
> Date: Tue, 11 Feb 2020 13:19:10 +0100
> Subject: [PATCH] vhost: fix return value of vhost_get_vq_desc
> 
> Before of the batch change, it was the chain's head. Need to keep that
> way or we will not be able to free a chain of descriptors.

I think it's cleaner to have all descriptors in the chain
have the same id.

> Fixes: eccb852f1fe6 ("vhost: batching fetches")
> ---
>  drivers/vhost/vhost.c | 3 +--
>  1 file changed, 1 insertion(+), 2 deletions(-)
> 
> diff --git a/drivers/vhost/vhost.c b/drivers/vhost/vhost.c
> index b5a51b1f2e79..fc422c3e5c08 100644
> --- a/drivers/vhost/vhost.c
> +++ b/drivers/vhost/vhost.c
> @@ -2409,12 +2409,11 @@ int vhost_get_vq_desc(struct vhost_virtqueue *vq,
>  			*out_num += ret;
>  		}
>  
> -		ret = desc->id;
> -
>  		if (!(desc->flags & VRING_DESC_F_NEXT))
>  			break;
>  	}
>  
> +	ret = vq->descs[vq->first_desc].id;
>  	vq->first_desc = i + 1;
>  
>  	return ret;
> -- 
> 2.18.1

Re: vhost changes (batched) in linux-next after 12/13 trigger random crashes in KVM guests after reboot

[Eugenio Pérez]

On Tue, 2020-02-11 at 14:13 +0100, Christian Borntraeger wrote:
> 
> On 11.02.20 14:04, Eugenio Pérez wrote:
> > On Mon, 2020-02-10 at 12:01 +0100, Christian Borntraeger wrote:
> > > On 10.02.20 10:47, Eugenio Perez Martin wrote:
> > > > Hi Christian.
> > > > 
> > > > I'm not able to reproduce the failure with eccb852f1fe6bede630e2e4f1a121a81e34354ab commit. Could you add more
> > > > data?
> > > > Your configuration (libvirt or qemu line), and host's dmesg output if any?
> > > > 
> > > > Thanks!
> > > 
> > > If it was not obvious, this is on s390x, a big endian system.
> > > 
> > 
> > Hi Christian. Thank you very much for your fast responses.
> > 
> > Could you try this patch on top of eccb852f1fe6bede630e2e4f1a121a81e34354ab?
> 
> I still get 
> [   43.665145] Guest moved used index from 0 to 289
> after some reboots.
> 
> 
> > Thanks!
> > 
> > From 71d0f9108a18aa894cc0c0c1c7efbad39f465a27 Mon Sep 17 00:00:00 2001
> > From: =?UTF-8?q?Eugenio=20P=C3=A9rez?= <
> > eperezma@redhat.com>
> > Date: Tue, 11 Feb 2020 13:19:10 +0100
> > Subject: [PATCH] vhost: fix return value of vhost_get_vq_desc
> > 
> > Before of the batch change, it was the chain's head. Need to keep that
> > way or we will not be able to free a chain of descriptors.
> > 
> > Fixes: eccb852f1fe6 ("vhost: batching fetches")
> > ---
> >  drivers/vhost/vhost.c | 3 +--
> >  1 file changed, 1 insertion(+), 2 deletions(-)
> > 
> > diff --git a/drivers/vhost/vhost.c b/drivers/vhost/vhost.c
> > index b5a51b1f2e79..fc422c3e5c08 100644
> > --- a/drivers/vhost/vhost.c
> > +++ b/drivers/vhost/vhost.c
> > @@ -2409,12 +2409,11 @@ int vhost_get_vq_desc(struct vhost_virtqueue *vq,
> >  			*out_num += ret;
> >  		}
> >  
> > -		ret = desc->id;
> > -
> >  		if (!(desc->flags & VRING_DESC_F_NEXT))
> >  			break;
> >  	}
> >  
> > +	ret = vq->descs[vq->first_desc].id;
> >  	vq->first_desc = i + 1;
> >  
> >  	return ret;
> > 

Sorry, still not able to reproduce the issue.

Could we try to disable all the vhost features?

Thanks!

diff --git a/drivers/vhost/vhost.h b/drivers/vhost/vhost.h
index 661088ae6dc7..08f6d2ccb697 100644
--- a/drivers/vhost/vhost.h
+++ b/drivers/vhost/vhost.h
@@ -250,11 +250,11 @@ int vhost_init_device_iotlb(struct vhost_dev *d, bool enabled);
        } while (0)
 
 enum {
-       VHOST_FEATURES = (1ULL << VIRTIO_F_NOTIFY_ON_EMPTY) |
-                        (1ULL << VIRTIO_RING_F_INDIRECT_DESC) |
-                        (1ULL << VIRTIO_RING_F_EVENT_IDX) |
-                        (1ULL << VHOST_F_LOG_ALL) |
-                        (1ULL << VIRTIO_F_ANY_LAYOUT) |
+       VHOST_FEATURES = /* (1ULL << VIRTIO_F_NOTIFY_ON_EMPTY) | */
+                        /* (1ULL << VIRTIO_RING_F_INDIRECT_DESC) | */
+                        /* (1ULL << VIRTIO_RING_F_EVENT_IDX) | */
+                        /* (1ULL << VHOST_F_LOG_ALL) | */
+                        /* (1ULL << VIRTIO_F_ANY_LAYOUT) | */
                         (1ULL << VIRTIO_F_VERSION_1)
 };

Re: vhost changes (batched) in linux-next after 12/13 trigger random crashes in KVM guests after reboot

[Christian Borntraeger]

On 12.02.20 17:34, Eugenio Pérez wrote:
> On Tue, 2020-02-11 at 14:13 +0100, Christian Borntraeger wrote:
>>
>> On 11.02.20 14:04, Eugenio Pérez wrote:
>>> On Mon, 2020-02-10 at 12:01 +0100, Christian Borntraeger wrote:
>>>> On 10.02.20 10:47, Eugenio Perez Martin wrote:
>>>>> Hi Christian.
>>>>>
>>>>> I'm not able to reproduce the failure with eccb852f1fe6bede630e2e4f1a121a81e34354ab commit. Could you add more
>>>>> data?
>>>>> Your configuration (libvirt or qemu line), and host's dmesg output if any?
>>>>>
>>>>> Thanks!
>>>>
>>>> If it was not obvious, this is on s390x, a big endian system.
>>>>
>>>
>>> Hi Christian. Thank you very much for your fast responses.
>>>
>>> Could you try this patch on top of eccb852f1fe6bede630e2e4f1a121a81e34354ab?
>>
>> I still get 
>> [   43.665145] Guest moved used index from 0 to 289
>> after some reboots.
>>
>>
>>> Thanks!
>>>
>>> From 71d0f9108a18aa894cc0c0c1c7efbad39f465a27 Mon Sep 17 00:00:00 2001
>>> From: =?UTF-8?q?Eugenio=20P=C3=A9rez?= <
>>> eperezma@redhat.com>
>>> Date: Tue, 11 Feb 2020 13:19:10 +0100
>>> Subject: [PATCH] vhost: fix return value of vhost_get_vq_desc
>>>
>>> Before of the batch change, it was the chain's head. Need to keep that
>>> way or we will not be able to free a chain of descriptors.
>>>
>>> Fixes: eccb852f1fe6 ("vhost: batching fetches")
>>> ---
>>>  drivers/vhost/vhost.c | 3 +--
>>>  1 file changed, 1 insertion(+), 2 deletions(-)
>>>
>>> diff --git a/drivers/vhost/vhost.c b/drivers/vhost/vhost.c
>>> index b5a51b1f2e79..fc422c3e5c08 100644
>>> --- a/drivers/vhost/vhost.c
>>> +++ b/drivers/vhost/vhost.c
>>> @@ -2409,12 +2409,11 @@ int vhost_get_vq_desc(struct vhost_virtqueue *vq,
>>>  			*out_num += ret;
>>>  		}
>>>  
>>> -		ret = desc->id;
>>> -
>>>  		if (!(desc->flags & VRING_DESC_F_NEXT))
>>>  			break;
>>>  	}
>>>  
>>> +	ret = vq->descs[vq->first_desc].id;
>>>  	vq->first_desc = i + 1;
>>>  
>>>  	return ret;
>>>
> 
> Sorry, still not able to reproduce the issue.
> 
> Could we try to disable all the vhost features?
> 
> Thanks!
> 
> diff --git a/drivers/vhost/vhost.h b/drivers/vhost/vhost.h
> index 661088ae6dc7..08f6d2ccb697 100644
> --- a/drivers/vhost/vhost.h
> +++ b/drivers/vhost/vhost.h
> @@ -250,11 +250,11 @@ int vhost_init_device_iotlb(struct vhost_dev *d, bool enabled);
>         } while (0)
>  
>  enum {
> -       VHOST_FEATURES = (1ULL << VIRTIO_F_NOTIFY_ON_EMPTY) |
> -                        (1ULL << VIRTIO_RING_F_INDIRECT_DESC) |
> -                        (1ULL << VIRTIO_RING_F_EVENT_IDX) |
> -                        (1ULL << VHOST_F_LOG_ALL) |
> -                        (1ULL << VIRTIO_F_ANY_LAYOUT) |
> +       VHOST_FEATURES = /* (1ULL << VIRTIO_F_NOTIFY_ON_EMPTY) | */
> +                        /* (1ULL << VIRTIO_RING_F_INDIRECT_DESC) | */
> +                        /* (1ULL << VIRTIO_RING_F_EVENT_IDX) | */
> +                        /* (1ULL << VHOST_F_LOG_ALL) | */
> +                        /* (1ULL << VIRTIO_F_ANY_LAYOUT) | */
>                          (1ULL << VIRTIO_F_VERSION_1)
>  };
> 


I still get  guest crashes with this on top of eccb852f1fe6. (The patch did not
apply, I had to manually comment out these things)

Re: vhost changes (batched) in linux-next after 12/13 trigger random crashes in KVM guests after reboot

[Eugenio Pérez]

On Thu, 2020-02-13 at 10:30 +0100, Christian Borntraeger wrote:
> 
> On 12.02.20 17:34, Eugenio Pérez wrote:
> > On Tue, 2020-02-11 at 14:13 +0100, Christian Borntraeger wrote:
> > > On 11.02.20 14:04, Eugenio Pérez wrote:
> > > > On Mon, 2020-02-10 at 12:01 +0100, Christian Borntraeger wrote:
> > > > > On 10.02.20 10:47, Eugenio Perez Martin wrote:
> > > > > > Hi Christian.
> > > > > > 
> > > > > > I'm not able to reproduce the failure with eccb852f1fe6bede630e2e4f1a121a81e34354ab commit. Could you add
> > > > > > more
> > > > > > data?
> > > > > > Your configuration (libvirt or qemu line), and host's dmesg output if any?
> > > > > > 
> > > > > > Thanks!
> > > > > 
> > > > > If it was not obvious, this is on s390x, a big endian system.
> > > > > 
> > > > 
> > > > Hi Christian. Thank you very much for your fast responses.
> > > > 
> > > > Could you try this patch on top of eccb852f1fe6bede630e2e4f1a121a81e34354ab?
> > > 
> > > I still get 
> > > [   43.665145] Guest moved used index from 0 to 289
> > > after some reboots.
> > > 
> > > 
> > > > Thanks!
> > > > 
> > > > From 71d0f9108a18aa894cc0c0c1c7efbad39f465a27 Mon Sep 17 00:00:00 2001
> > > > From: =?UTF-8?q?Eugenio=20P=C3=A9rez?= <
> > > > eperezma@redhat.com>
> > > > Date: Tue, 11 Feb 2020 13:19:10 +0100
> > > > Subject: [PATCH] vhost: fix return value of vhost_get_vq_desc
> > > > 
> > > > Before of the batch change, it was the chain's head. Need to keep that
> > > > way or we will not be able to free a chain of descriptors.
> > > > 
> > > > Fixes: eccb852f1fe6 ("vhost: batching fetches")
> > > > ---
> > > >  drivers/vhost/vhost.c | 3 +--
> > > >  1 file changed, 1 insertion(+), 2 deletions(-)
> > > > 
> > > > diff --git a/drivers/vhost/vhost.c b/drivers/vhost/vhost.c
> > > > index b5a51b1f2e79..fc422c3e5c08 100644
> > > > --- a/drivers/vhost/vhost.c
> > > > +++ b/drivers/vhost/vhost.c
> > > > @@ -2409,12 +2409,11 @@ int vhost_get_vq_desc(struct vhost_virtqueue *vq,
> > > >  			*out_num += ret;
> > > >  		}
> > > >  
> > > > -		ret = desc->id;
> > > > -
> > > >  		if (!(desc->flags & VRING_DESC_F_NEXT))
> > > >  			break;
> > > >  	}
> > > >  
> > > > +	ret = vq->descs[vq->first_desc].id;
> > > >  	vq->first_desc = i + 1;
> > > >  
> > > >  	return ret;
> > > > 
> > 
> > Sorry, still not able to reproduce the issue.
> > 
> > Could we try to disable all the vhost features?
> > 
> > Thanks!
> > 
> > diff --git a/drivers/vhost/vhost.h b/drivers/vhost/vhost.h
> > index 661088ae6dc7..08f6d2ccb697 100644
> > --- a/drivers/vhost/vhost.h
> > +++ b/drivers/vhost/vhost.h
> > @@ -250,11 +250,11 @@ int vhost_init_device_iotlb(struct vhost_dev *d, bool enabled);
> >         } while (0)
> >  
> >  enum {
> > -       VHOST_FEATURES = (1ULL << VIRTIO_F_NOTIFY_ON_EMPTY) |
> > -                        (1ULL << VIRTIO_RING_F_INDIRECT_DESC) |
> > -                        (1ULL << VIRTIO_RING_F_EVENT_IDX) |
> > -                        (1ULL << VHOST_F_LOG_ALL) |
> > -                        (1ULL << VIRTIO_F_ANY_LAYOUT) |
> > +       VHOST_FEATURES = /* (1ULL << VIRTIO_F_NOTIFY_ON_EMPTY) | */
> > +                        /* (1ULL << VIRTIO_RING_F_INDIRECT_DESC) | */
> > +                        /* (1ULL << VIRTIO_RING_F_EVENT_IDX) | */
> > +                        /* (1ULL << VHOST_F_LOG_ALL) | */
> > +                        /* (1ULL << VIRTIO_F_ANY_LAYOUT) | */
> >                          (1ULL << VIRTIO_F_VERSION_1)
> >  };
> > 
> 
> I still get  guest crashes with this on top of eccb852f1fe6. (The patch did not
> apply, I had to manually comment out these things)
> 

Sorry about that, I C&P transformed tabs to spaces.

Can we try tracing last_avail_idx with the attached patch? Can you enable also line and thread id (dyndbg='+plt')?

Thanks!

From f7012e8b9db711b12d36e6e97411e7afa34bf768 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Eugenio=20P=C3=A9rez?= <eperezma@redhat.com>
Date: Thu, 13 Feb 2020 11:26:06 +0100
Subject: [PATCH] vhost: disable all features and trace last_avail_idx

---
 drivers/vhost/vhost.c | 13 +++++++++++--
 drivers/vhost/vhost.h | 10 +++++-----
 2 files changed, 16 insertions(+), 7 deletions(-)

diff --git a/drivers/vhost/vhost.c b/drivers/vhost/vhost.c
index fc422c3e5c08..021d70bed015 100644
--- a/drivers/vhost/vhost.c
+++ b/drivers/vhost/vhost.c
@@ -1645,6 +1645,9 @@ long vhost_vring_ioctl(struct vhost_dev *d, unsigned int ioctl, void __user *arg
 		vq->last_avail_idx = s.num;
 		/* Forget the cached index value. */
 		vq->avail_idx = vq->last_avail_idx;
+		pr_debug(
+			"VHOST_SET_VRING_BASE [vq=%p][vq->last_avail_idx=%u][vq->avail_idx=%u]",
+			vq, vq->last_avail_idx, vq->avail_idx);
 		break;
 	case VHOST_GET_VRING_BASE:
 		s.index = idx;
@@ -2239,8 +2242,8 @@ static int fetch_buf(struct vhost_virtqueue *vq)
 		vq->avail_idx = vhost16_to_cpu(vq, avail_idx);
 
 		if (unlikely((u16)(vq->avail_idx - last_avail_idx) > vq->num)) {
-			vq_err(vq, "Guest moved used index from %u to %u",
-				last_avail_idx, vq->avail_idx);
+			vq_err(vq, "Guest moved vq %p used index from %u to %u",
+				vq, last_avail_idx, vq->avail_idx);
 			return -EFAULT;
 		}
 
@@ -2316,6 +2319,9 @@ static int fetch_buf(struct vhost_virtqueue *vq)
 	BUG_ON(!(vq->used_flags & VRING_USED_F_NO_NOTIFY));
 
 	/* On success, increment avail index. */
+	pr_debug(
+		"[vq=%p][vq->last_avail_idx=%u][vq->avail_idx=%u][vq->ndescs=%d][vq->first_desc=%d]",
+		vq, vq->last_avail_idx, vq->avail_idx, vq->ndescs, vq->first_desc);
 	vq->last_avail_idx++;
 
 	return 0;
@@ -2431,6 +2437,9 @@ EXPORT_SYMBOL_GPL(vhost_get_vq_desc);
 /* Reverse the effect of vhost_get_vq_desc. Useful for error handling. */
 void vhost_discard_vq_desc(struct vhost_virtqueue *vq, int n)
 {
+	pr_debug(
+		"DISCARD [vq=%p][vq->last_avail_idx=%u][vq->avail_idx=%u][n=%d]",
+		vq, vq->last_avail_idx, vq->avail_idx, n);
 	vq->last_avail_idx -= n;
 }
 EXPORT_SYMBOL_GPL(vhost_discard_vq_desc);
diff --git a/drivers/vhost/vhost.h b/drivers/vhost/vhost.h
index 661088ae6dc7..08f6d2ccb697 100644
--- a/drivers/vhost/vhost.h
+++ b/drivers/vhost/vhost.h
@@ -250,11 +250,11 @@ int vhost_init_device_iotlb(struct vhost_dev *d, bool enabled);
 	} while (0)
 
 enum {
-	VHOST_FEATURES = (1ULL << VIRTIO_F_NOTIFY_ON_EMPTY) |
-			 (1ULL << VIRTIO_RING_F_INDIRECT_DESC) |
-			 (1ULL << VIRTIO_RING_F_EVENT_IDX) |
-			 (1ULL << VHOST_F_LOG_ALL) |
-			 (1ULL << VIRTIO_F_ANY_LAYOUT) |
+	VHOST_FEATURES = /* (1ULL << VIRTIO_F_NOTIFY_ON_EMPTY) | */
+			 /* (1ULL << VIRTIO_RING_F_INDIRECT_DESC) | */
+			 /* (1ULL << VIRTIO_RING_F_EVENT_IDX) | */
+			 /* (1ULL << VHOST_F_LOG_ALL) | */
+			 /* (1ULL << VIRTIO_F_ANY_LAYOUT) | */
 			 (1ULL << VIRTIO_F_VERSION_1)
 };
 
-- 
2.18.1

Re: vhost changes (batched) in linux-next after 12/13 trigger random crashes in KVM guests after reboot

[Christian Borntraeger]

[-- Attachment #1: Type: text/plain, Size: 169 bytes --]



On 13.02.20 11:47, Eugenio Pérez wrote:
> 
> Can we try tracing last_avail_idx with the attached patch? Can you enable also line and thread id (dyndbg='+plt')?







[-- Attachment #2: log --]
[-- Type: text/plain, Size: 23398 bytes --]

[  326.584446] [2127] 1648: VHOST_SET_VRING_BASE [vq=0000000036fdfcb7][vq->last_avail_idx=0][vq->avail_idx=0]
[  326.584457] [2127] 1648: VHOST_SET_VRING_BASE [vq=0000000060d7302b][vq->last_avail_idx=0][vq->avail_idx=0]
[  326.584474] [2124] 2441: DISCARD [vq=0000000036fdfcb7][vq->last_avail_idx=0][vq->avail_idx=0][n=0]
[  327.281901] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=0][vq->avail_idx=256][vq->ndescs=1][vq->first_desc=0]
[  327.281902] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=1][vq->avail_idx=256][vq->ndescs=2][vq->first_desc=0]
[  327.281903] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=2][vq->avail_idx=256][vq->ndescs=3][vq->first_desc=0]
[  327.281904] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=3][vq->avail_idx=256][vq->ndescs=4][vq->first_desc=0]
[  327.281905] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=4][vq->avail_idx=256][vq->ndescs=5][vq->first_desc=0]
[  327.281906] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=5][vq->avail_idx=256][vq->ndescs=6][vq->first_desc=0]
[  327.281907] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=6][vq->avail_idx=256][vq->ndescs=7][vq->first_desc=0]
[  327.281908] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=7][vq->avail_idx=256][vq->ndescs=8][vq->first_desc=0]
[  327.281909] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=8][vq->avail_idx=256][vq->ndescs=9][vq->first_desc=0]
[  327.281910] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=9][vq->avail_idx=256][vq->ndescs=10][vq->first_desc=0]
[  327.281911] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=10][vq->avail_idx=256][vq->ndescs=11][vq->first_desc=0]
[  327.281911] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=11][vq->avail_idx=256][vq->ndescs=12][vq->first_desc=0]
[  327.281912] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=12][vq->avail_idx=256][vq->ndescs=13][vq->first_desc=0]
[  327.281913] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=13][vq->avail_idx=256][vq->ndescs=14][vq->first_desc=0]
[  327.281914] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=14][vq->avail_idx=256][vq->ndescs=15][vq->first_desc=0]
[  327.281915] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=15][vq->avail_idx=256][vq->ndescs=16][vq->first_desc=0]
[  327.281916] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=16][vq->avail_idx=256][vq->ndescs=17][vq->first_desc=0]
[  327.281916] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=17][vq->avail_idx=256][vq->ndescs=18][vq->first_desc=0]
[  327.281917] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=18][vq->avail_idx=256][vq->ndescs=19][vq->first_desc=0]
[  327.281918] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=19][vq->avail_idx=256][vq->ndescs=20][vq->first_desc=0]
[  327.281919] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=20][vq->avail_idx=256][vq->ndescs=21][vq->first_desc=0]
[  327.281920] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=21][vq->avail_idx=256][vq->ndescs=22][vq->first_desc=0]
[  327.281921] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=22][vq->avail_idx=256][vq->ndescs=23][vq->first_desc=0]
[  327.281921] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=23][vq->avail_idx=256][vq->ndescs=24][vq->first_desc=0]
[  327.281922] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=24][vq->avail_idx=256][vq->ndescs=25][vq->first_desc=0]
[  327.281923] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=25][vq->avail_idx=256][vq->ndescs=26][vq->first_desc=0]
[  327.281924] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=26][vq->avail_idx=256][vq->ndescs=27][vq->first_desc=0]
[  327.281925] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=27][vq->avail_idx=256][vq->ndescs=28][vq->first_desc=0]
[  327.281926] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=28][vq->avail_idx=256][vq->ndescs=29][vq->first_desc=0]
[  327.281926] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=29][vq->avail_idx=256][vq->ndescs=30][vq->first_desc=0]
[  327.281927] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=30][vq->avail_idx=256][vq->ndescs=31][vq->first_desc=0]
[  327.281928] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=31][vq->avail_idx=256][vq->ndescs=32][vq->first_desc=0]
[  327.281929] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=32][vq->avail_idx=256][vq->ndescs=33][vq->first_desc=0]
[  327.281930] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=33][vq->avail_idx=256][vq->ndescs=34][vq->first_desc=0]
[  327.281931] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=34][vq->avail_idx=256][vq->ndescs=35][vq->first_desc=0]
[  327.281932] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=35][vq->avail_idx=256][vq->ndescs=36][vq->first_desc=0]
[  327.281933] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=36][vq->avail_idx=256][vq->ndescs=37][vq->first_desc=0]
[  327.281934] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=37][vq->avail_idx=256][vq->ndescs=38][vq->first_desc=0]
[  327.281934] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=38][vq->avail_idx=256][vq->ndescs=39][vq->first_desc=0]
[  327.281935] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=39][vq->avail_idx=256][vq->ndescs=40][vq->first_desc=0]
[  327.281936] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=40][vq->avail_idx=256][vq->ndescs=41][vq->first_desc=0]
[  327.281937] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=41][vq->avail_idx=256][vq->ndescs=42][vq->first_desc=0]
[  327.281938] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=42][vq->avail_idx=256][vq->ndescs=43][vq->first_desc=0]
[  327.281939] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=43][vq->avail_idx=256][vq->ndescs=44][vq->first_desc=0]
[  327.281939] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=44][vq->avail_idx=256][vq->ndescs=45][vq->first_desc=0]
[  327.281940] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=45][vq->avail_idx=256][vq->ndescs=46][vq->first_desc=0]
[  327.281941] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=46][vq->avail_idx=256][vq->ndescs=47][vq->first_desc=0]
[  327.281942] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=47][vq->avail_idx=256][vq->ndescs=48][vq->first_desc=0]
[  327.281943] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=48][vq->avail_idx=256][vq->ndescs=49][vq->first_desc=0]
[  327.281944] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=49][vq->avail_idx=256][vq->ndescs=50][vq->first_desc=0]
[  327.281945] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=50][vq->avail_idx=256][vq->ndescs=51][vq->first_desc=0]
[  327.281946] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=51][vq->avail_idx=256][vq->ndescs=52][vq->first_desc=0]
[  327.281947] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=52][vq->avail_idx=256][vq->ndescs=53][vq->first_desc=0]
[  327.281948] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=53][vq->avail_idx=256][vq->ndescs=54][vq->first_desc=0]
[  327.281948] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=54][vq->avail_idx=256][vq->ndescs=55][vq->first_desc=0]
[  327.281949] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=55][vq->avail_idx=256][vq->ndescs=56][vq->first_desc=0]
[  327.281950] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=56][vq->avail_idx=256][vq->ndescs=57][vq->first_desc=0]
[  327.281951] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=57][vq->avail_idx=256][vq->ndescs=58][vq->first_desc=0]
[  327.281952] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=58][vq->avail_idx=256][vq->ndescs=59][vq->first_desc=0]
[  327.281953] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=59][vq->avail_idx=256][vq->ndescs=60][vq->first_desc=0]
[  327.281954] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=60][vq->avail_idx=256][vq->ndescs=61][vq->first_desc=0]
[  327.281955] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=61][vq->avail_idx=256][vq->ndescs=62][vq->first_desc=0]
[  327.281955] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=62][vq->avail_idx=256][vq->ndescs=63][vq->first_desc=0]
[  327.281956] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=63][vq->avail_idx=256][vq->ndescs=64][vq->first_desc=0]
[  327.281958] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=64][vq->avail_idx=256][vq->ndescs=65][vq->first_desc=0]
[  327.295289] [2124] 2322: [vq=0000000060d7302b][vq->last_avail_idx=0][vq->avail_idx=1][vq->ndescs=1][vq->first_desc=0]
[  327.313733] [2124] 2322: [vq=0000000060d7302b][vq->last_avail_idx=1][vq->avail_idx=2][vq->ndescs=1][vq->first_desc=0]
[  327.328259] [2124] 2322: [vq=0000000060d7302b][vq->last_avail_idx=2][vq->avail_idx=3][vq->ndescs=1][vq->first_desc=0]
[  327.328816] [2124] 2322: [vq=0000000060d7302b][vq->last_avail_idx=3][vq->avail_idx=5][vq->ndescs=1][vq->first_desc=0]
[  327.328818] [2124] 2322: [vq=0000000060d7302b][vq->last_avail_idx=4][vq->avail_idx=5][vq->ndescs=2][vq->first_desc=0]
[  327.553778] [2124] 2322: [vq=0000000060d7302b][vq->last_avail_idx=5][vq->avail_idx=6][vq->ndescs=1][vq->first_desc=0]
[  327.663779] [2124] 2322: [vq=0000000060d7302b][vq->last_avail_idx=6][vq->avail_idx=7][vq->ndescs=1][vq->first_desc=0]
[  328.433781] [2124] 2322: [vq=0000000060d7302b][vq->last_avail_idx=7][vq->avail_idx=8][vq->ndescs=1][vq->first_desc=0]
[  328.522897] [2124] 2322: [vq=0000000060d7302b][vq->last_avail_idx=8][vq->avail_idx=9][vq->ndescs=2][vq->first_desc=0]
[  328.593777] [2124] 2322: [vq=0000000060d7302b][vq->last_avail_idx=9][vq->avail_idx=10][vq->ndescs=1][vq->first_desc=0]
[  328.723911] [2124] 2322: [vq=0000000060d7302b][vq->last_avail_idx=10][vq->avail_idx=11][vq->ndescs=2][vq->first_desc=0]
[  328.926482] [2124] 2322: [vq=0000000060d7302b][vq->last_avail_idx=11][vq->avail_idx=12][vq->ndescs=2][vq->first_desc=0]
[  329.129059] [2124] 2322: [vq=0000000060d7302b][vq->last_avail_idx=12][vq->avail_idx=13][vq->ndescs=2][vq->first_desc=0]
[  329.295360] [2124] 2322: [vq=0000000060d7302b][vq->last_avail_idx=13][vq->avail_idx=14][vq->ndescs=1][vq->first_desc=0]
[  329.633801] [2124] 2322: [vq=0000000060d7302b][vq->last_avail_idx=14][vq->avail_idx=15][vq->ndescs=1][vq->first_desc=0]
[  329.663784] [2124] 2322: [vq=0000000060d7302b][vq->last_avail_idx=15][vq->avail_idx=16][vq->ndescs=1][vq->first_desc=0]
[  330.053043] [2124] 2322: [vq=0000000060d7302b][vq->last_avail_idx=16][vq->avail_idx=17][vq->ndescs=1][vq->first_desc=0]
[  330.111522] [2124] 2322: [vq=0000000060d7302b][vq->last_avail_idx=17][vq->avail_idx=18][vq->ndescs=1][vq->first_desc=0]
[  330.112176] [2124] 2322: [vq=0000000060d7302b][vq->last_avail_idx=18][vq->avail_idx=19][vq->ndescs=1][vq->first_desc=0]
[  330.112449] [2124] 2322: [vq=0000000060d7302b][vq->last_avail_idx=19][vq->avail_idx=20][vq->ndescs=1][vq->first_desc=0]
[  330.112619] [2124] 2322: [vq=0000000060d7302b][vq->last_avail_idx=20][vq->avail_idx=21][vq->ndescs=1][vq->first_desc=0]
[  330.112793] [2124] 2322: [vq=0000000060d7302b][vq->last_avail_idx=21][vq->avail_idx=22][vq->ndescs=1][vq->first_desc=0]
[  330.113028] [2124] 2322: [vq=0000000060d7302b][vq->last_avail_idx=22][vq->avail_idx=23][vq->ndescs=1][vq->first_desc=0]
[  330.113203] [2124] 2322: [vq=0000000060d7302b][vq->last_avail_idx=23][vq->avail_idx=24][vq->ndescs=1][vq->first_desc=0]
[  330.113366] [2124] 2322: [vq=0000000060d7302b][vq->last_avail_idx=24][vq->avail_idx=25][vq->ndescs=1][vq->first_desc=0]
[  330.113527] [2124] 2322: [vq=0000000060d7302b][vq->last_avail_idx=25][vq->avail_idx=26][vq->ndescs=1][vq->first_desc=0]
[  330.113686] [2124] 2322: [vq=0000000060d7302b][vq->last_avail_idx=26][vq->avail_idx=27][vq->ndescs=1][vq->first_desc=0]
[  330.113895] [2124] 2322: [vq=0000000060d7302b][vq->last_avail_idx=27][vq->avail_idx=28][vq->ndescs=1][vq->first_desc=0]
[  330.114064] [2124] 2322: [vq=0000000060d7302b][vq->last_avail_idx=28][vq->avail_idx=29][vq->ndescs=1][vq->first_desc=0]
[  330.114227] [2124] 2322: [vq=0000000060d7302b][vq->last_avail_idx=29][vq->avail_idx=30][vq->ndescs=1][vq->first_desc=0]
[  330.114389] [2124] 2322: [vq=0000000060d7302b][vq->last_avail_idx=30][vq->avail_idx=31][vq->ndescs=1][vq->first_desc=0]
[  330.114549] [2124] 2322: [vq=0000000060d7302b][vq->last_avail_idx=31][vq->avail_idx=32][vq->ndescs=1][vq->first_desc=0]
[  330.114708] [2124] 2322: [vq=0000000060d7302b][vq->last_avail_idx=32][vq->avail_idx=33][vq->ndescs=1][vq->first_desc=0]
[  330.114869] [2124] 2322: [vq=0000000060d7302b][vq->last_avail_idx=33][vq->avail_idx=34][vq->ndescs=1][vq->first_desc=0]
[  330.115033] [2124] 2322: [vq=0000000060d7302b][vq->last_avail_idx=34][vq->avail_idx=35][vq->ndescs=1][vq->first_desc=0]
[  330.115194] [2124] 2322: [vq=0000000060d7302b][vq->last_avail_idx=35][vq->avail_idx=36][vq->ndescs=1][vq->first_desc=0]
[  330.115355] [2124] 2322: [vq=0000000060d7302b][vq->last_avail_idx=36][vq->avail_idx=37][vq->ndescs=1][vq->first_desc=0]
[  330.115515] [2124] 2322: [vq=0000000060d7302b][vq->last_avail_idx=37][vq->avail_idx=38][vq->ndescs=1][vq->first_desc=0]
[  330.433757] [2124] 2322: [vq=0000000060d7302b][vq->last_avail_idx=38][vq->avail_idx=39][vq->ndescs=1][vq->first_desc=0]
[  330.593759] [2124] 2322: [vq=0000000060d7302b][vq->last_avail_idx=39][vq->avail_idx=40][vq->ndescs=1][vq->first_desc=0]
[  332.511365] [2124] 2322: [vq=0000000060d7302b][vq->last_avail_idx=40][vq->avail_idx=41][vq->ndescs=1][vq->first_desc=0]
[  333.245392] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=65][vq->avail_idx=256][vq->ndescs=1][vq->first_desc=0]
[  333.245393] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=66][vq->avail_idx=256][vq->ndescs=2][vq->first_desc=0]
[  333.245394] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=67][vq->avail_idx=256][vq->ndescs=3][vq->first_desc=0]
[  333.245395] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=68][vq->avail_idx=256][vq->ndescs=4][vq->first_desc=0]
[  333.245396] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=69][vq->avail_idx=256][vq->ndescs=5][vq->first_desc=0]
[  333.245397] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=70][vq->avail_idx=256][vq->ndescs=6][vq->first_desc=0]
[  333.245398] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=71][vq->avail_idx=256][vq->ndescs=7][vq->first_desc=0]
[  333.245398] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=72][vq->avail_idx=256][vq->ndescs=8][vq->first_desc=0]
[  333.245399] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=73][vq->avail_idx=256][vq->ndescs=9][vq->first_desc=0]
[  333.245400] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=74][vq->avail_idx=256][vq->ndescs=10][vq->first_desc=0]
[  333.245401] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=75][vq->avail_idx=256][vq->ndescs=11][vq->first_desc=0]
[  333.245402] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=76][vq->avail_idx=256][vq->ndescs=12][vq->first_desc=0]
[  333.245403] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=77][vq->avail_idx=256][vq->ndescs=13][vq->first_desc=0]
[  333.245404] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=78][vq->avail_idx=256][vq->ndescs=14][vq->first_desc=0]
[  333.245405] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=79][vq->avail_idx=256][vq->ndescs=15][vq->first_desc=0]
[  333.245406] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=80][vq->avail_idx=256][vq->ndescs=16][vq->first_desc=0]
[  333.245407] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=81][vq->avail_idx=256][vq->ndescs=17][vq->first_desc=0]
[  333.245408] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=82][vq->avail_idx=256][vq->ndescs=18][vq->first_desc=0]
[  333.245408] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=83][vq->avail_idx=256][vq->ndescs=19][vq->first_desc=0]
[  333.245409] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=84][vq->avail_idx=256][vq->ndescs=20][vq->first_desc=0]
[  333.245410] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=85][vq->avail_idx=256][vq->ndescs=21][vq->first_desc=0]
[  333.245411] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=86][vq->avail_idx=256][vq->ndescs=22][vq->first_desc=0]
[  333.245412] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=87][vq->avail_idx=256][vq->ndescs=23][vq->first_desc=0]
[  333.245413] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=88][vq->avail_idx=256][vq->ndescs=24][vq->first_desc=0]
[  333.245414] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=89][vq->avail_idx=256][vq->ndescs=25][vq->first_desc=0]
[  333.245414] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=90][vq->avail_idx=256][vq->ndescs=26][vq->first_desc=0]
[  333.245415] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=91][vq->avail_idx=256][vq->ndescs=27][vq->first_desc=0]
[  333.245416] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=92][vq->avail_idx=256][vq->ndescs=28][vq->first_desc=0]
[  333.245417] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=93][vq->avail_idx=256][vq->ndescs=29][vq->first_desc=0]
[  333.245418] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=94][vq->avail_idx=256][vq->ndescs=30][vq->first_desc=0]
[  333.245419] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=95][vq->avail_idx=256][vq->ndescs=31][vq->first_desc=0]
[  333.245420] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=96][vq->avail_idx=256][vq->ndescs=32][vq->first_desc=0]
[  333.245421] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=97][vq->avail_idx=256][vq->ndescs=33][vq->first_desc=0]
[  333.245422] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=98][vq->avail_idx=256][vq->ndescs=34][vq->first_desc=0]
[  333.245422] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=99][vq->avail_idx=256][vq->ndescs=35][vq->first_desc=0]
[  333.245423] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=100][vq->avail_idx=256][vq->ndescs=36][vq->first_desc=0]
[  333.245424] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=101][vq->avail_idx=256][vq->ndescs=37][vq->first_desc=0]
[  333.245425] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=102][vq->avail_idx=256][vq->ndescs=38][vq->first_desc=0]
[  333.245426] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=103][vq->avail_idx=256][vq->ndescs=39][vq->first_desc=0]
[  333.245427] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=104][vq->avail_idx=256][vq->ndescs=40][vq->first_desc=0]
[  333.245427] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=105][vq->avail_idx=256][vq->ndescs=41][vq->first_desc=0]
[  333.245428] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=106][vq->avail_idx=256][vq->ndescs=42][vq->first_desc=0]
[  333.245429] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=107][vq->avail_idx=256][vq->ndescs=43][vq->first_desc=0]
[  333.245430] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=108][vq->avail_idx=256][vq->ndescs=44][vq->first_desc=0]
[  333.245431] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=109][vq->avail_idx=256][vq->ndescs=45][vq->first_desc=0]
[  333.245431] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=110][vq->avail_idx=256][vq->ndescs=46][vq->first_desc=0]
[  333.245432] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=111][vq->avail_idx=256][vq->ndescs=47][vq->first_desc=0]
[  333.245433] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=112][vq->avail_idx=256][vq->ndescs=48][vq->first_desc=0]
[  333.245434] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=113][vq->avail_idx=256][vq->ndescs=49][vq->first_desc=0]
[  333.245435] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=114][vq->avail_idx=256][vq->ndescs=50][vq->first_desc=0]
[  333.245436] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=115][vq->avail_idx=256][vq->ndescs=51][vq->first_desc=0]
[  333.245437] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=116][vq->avail_idx=256][vq->ndescs=52][vq->first_desc=0]
[  333.245438] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=117][vq->avail_idx=256][vq->ndescs=53][vq->first_desc=0]
[  333.245438] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=118][vq->avail_idx=256][vq->ndescs=54][vq->first_desc=0]
[  333.245439] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=119][vq->avail_idx=256][vq->ndescs=55][vq->first_desc=0]
[  333.245440] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=120][vq->avail_idx=256][vq->ndescs=56][vq->first_desc=0]
[  333.245441] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=121][vq->avail_idx=256][vq->ndescs=57][vq->first_desc=0]
[  333.245442] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=122][vq->avail_idx=256][vq->ndescs=58][vq->first_desc=0]
[  333.245443] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=123][vq->avail_idx=256][vq->ndescs=59][vq->first_desc=0]
[  333.245443] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=124][vq->avail_idx=256][vq->ndescs=60][vq->first_desc=0]
[  333.245444] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=125][vq->avail_idx=256][vq->ndescs=61][vq->first_desc=0]
[  333.245445] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=126][vq->avail_idx=256][vq->ndescs=62][vq->first_desc=0]
[  333.245446] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=127][vq->avail_idx=256][vq->ndescs=63][vq->first_desc=0]
[  333.245447] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=128][vq->avail_idx=256][vq->ndescs=64][vq->first_desc=0]
[  333.245448] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=129][vq->avail_idx=256][vq->ndescs=65][vq->first_desc=0]
[  336.121638] [2126] 1648: VHOST_SET_VRING_BASE [vq=0000000036fdfcb7][vq->last_avail_idx=0][vq->avail_idx=0]
[  336.121654] [2126] 1648: VHOST_SET_VRING_BASE [vq=0000000060d7302b][vq->last_avail_idx=0][vq->avail_idx=0]
[  336.943877] [2124] 2322: [vq=0000000060d7302b][vq->last_avail_idx=0][vq->avail_idx=1][vq->ndescs=1][vq->first_desc=0]
[  336.957322] [2124] 2322: [vq=0000000060d7302b][vq->last_avail_idx=1][vq->avail_idx=2][vq->ndescs=1][vq->first_desc=0]
[  336.973438] [2124] 2322: [vq=0000000060d7302b][vq->last_avail_idx=2][vq->avail_idx=3][vq->ndescs=1][vq->first_desc=0]
[  337.223453] [2124] 2322: [vq=0000000060d7302b][vq->last_avail_idx=3][vq->avail_idx=4][vq->ndescs=1][vq->first_desc=0]
[  337.383453] [2124] 2322: [vq=0000000060d7302b][vq->last_avail_idx=4][vq->avail_idx=5][vq->ndescs=1][vq->first_desc=0]
[  337.473429] [2124] 2322: [vq=0000000060d7302b][vq->last_avail_idx=5][vq->avail_idx=6][vq->ndescs=1][vq->first_desc=0]
[  337.993434] [2124] 2322: [vq=0000000060d7302b][vq->last_avail_idx=6][vq->avail_idx=7][vq->ndescs=1][vq->first_desc=0]
[  338.393446] [2124] 2322: [vq=0000000060d7302b][vq->last_avail_idx=7][vq->avail_idx=8][vq->ndescs=1][vq->first_desc=0]
[  338.423446] [2124] 2322: [vq=0000000060d7302b][vq->last_avail_idx=8][vq->avail_idx=9][vq->ndescs=1][vq->first_desc=0]
[  338.693357] [2124] 2322: [vq=0000000060d7302b][vq->last_avail_idx=9][vq->avail_idx=10][vq->ndescs=1][vq->first_desc=0]
[  338.843477] [2124] 2322: [vq=0000000060d7302b][vq->last_avail_idx=10][vq->avail_idx=11][vq->ndescs=1][vq->first_desc=0]
[  338.943915] [2124] 2322: [vq=0000000060d7302b][vq->last_avail_idx=11][vq->avail_idx=12][vq->ndescs=1][vq->first_desc=0]
[  339.033436] [2124] 2322: [vq=0000000060d7302b][vq->last_avail_idx=12][vq->avail_idx=13][vq->ndescs=1][vq->first_desc=0]

Re: vhost changes (batched) in linux-next after 12/13 trigger random crashes in KVM guests after reboot

[Eugenio Pérez]

Can we try with this traces?

From b793b4106085ab1970bdedb340e49f37843ed585 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Eugenio=20P=C3=A9rez?= <eperezma@redhat.com>
Date: Thu, 13 Feb 2020 17:27:05 +0100
Subject: [PATCH] vhost: Add debug in ioctl calls

---
 drivers/vhost/net.c   | 20 +++++++++++++++++---
 drivers/vhost/vhost.c | 16 ++++++++++++++--
 2 files changed, 31 insertions(+), 5 deletions(-)

diff --git a/drivers/vhost/net.c b/drivers/vhost/net.c
index e158159671fa..e4d5f843f9c0 100644
--- a/drivers/vhost/net.c
+++ b/drivers/vhost/net.c
@@ -1505,10 +1505,13 @@ static long vhost_net_set_backend(struct vhost_net *n, unsigned index, int fd)
 
 	mutex_lock(&n->dev.mutex);
 	r = vhost_dev_check_owner(&n->dev);
-	if (r)
+	if (r) {
+		pr_debug("vhost_dev_check_owner index=%u fd=%d rc r=%d", index, fd, r);
 		goto err;
+	}
 
 	if (index >= VHOST_NET_VQ_MAX) {
+		pr_debug("vhost_dev_check_owner index=%u fd=%d MAX=%d", index, fd, VHOST_NET_VQ_MAX);
 		r = -ENOBUFS;
 		goto err;
 	}
@@ -1518,22 +1521,26 @@ static long vhost_net_set_backend(struct vhost_net *n, unsigned index, int fd)
 
 	/* Verify that ring has been setup correctly. */
 	if (!vhost_vq_access_ok(vq)) {
+		pr_debug("vhost_net_set_backend index=%u fd=%d !vhost_vq_access_ok", index, fd);
 		r = -EFAULT;
 		goto err_vq;
 	}
 	sock = get_socket(fd);
 	if (IS_ERR(sock)) {
 		r = PTR_ERR(sock);
+		pr_debug("vhost_net_set_backend index=%u fd=%d get_socket err r=%d", index, fd, r);
 		goto err_vq;
 	}
 
 	/* start polling new socket */
 	oldsock = vq->private_data;
 	if (sock != oldsock) {
+		pr_debug("sock=%p != oldsock=%p index=%u fd=%d vq=%p", sock, oldsock, index, fd, vq);
 		ubufs = vhost_net_ubuf_alloc(vq,
 					     sock && vhost_sock_zcopy(sock));
 		if (IS_ERR(ubufs)) {
 			r = PTR_ERR(ubufs);
+			pr_debug("ubufs index=%u fd=%d err r=%d vq=%p", index, fd, r, vq);
 			goto err_ubufs;
 		}
 
@@ -1541,11 +1548,15 @@ static long vhost_net_set_backend(struct vhost_net *n, unsigned index, int fd)
 		vq->private_data = sock;
 		vhost_net_buf_unproduce(nvq);
 		r = vhost_vq_init_access(vq);
-		if (r)
+		if (r) {
+			pr_debug("init_access index=%u fd=%d r=%d vq=%p", index, fd, r, vq);
 			goto err_used;
+		}
 		r = vhost_net_enable_vq(n, vq);
-		if (r)
+		if (r) {
+			pr_debug("enable_vq index=%u fd=%d r=%d vq=%p", index, fd, r, vq);
 			goto err_used;
+		}
 		if (index == VHOST_NET_VQ_RX)
 			nvq->rx_ring = get_tap_ptr_ring(fd);
 
@@ -1559,6 +1570,8 @@ static long vhost_net_set_backend(struct vhost_net *n, unsigned index, int fd)
 
 	mutex_unlock(&vq->mutex);
 
+	pr_debug("sock=%p", sock);
+
 	if (oldubufs) {
 		vhost_net_ubuf_put_wait_and_free(oldubufs);
 		mutex_lock(&vq->mutex);
@@ -1710,6 +1723,7 @@ static long vhost_net_ioctl(struct file *f, unsigned int ioctl,
 
 	switch (ioctl) {
 	case VHOST_NET_SET_BACKEND:
+		pr_debug("VHOST_NET_SET_BACKEND");
 		if (copy_from_user(&backend, argp, sizeof backend))
 			return -EFAULT;
 		return vhost_net_set_backend(n, backend.index, backend.fd);
diff --git a/drivers/vhost/vhost.c b/drivers/vhost/vhost.c
index 021d70bed015..7f4848f9cec3 100644
--- a/drivers/vhost/vhost.c
+++ b/drivers/vhost/vhost.c
@@ -1642,18 +1642,30 @@ long vhost_vring_ioctl(struct vhost_dev *d, unsigned int ioctl, void __user *arg
 			r = -EINVAL;
 			break;
 		}
+
+		if (vq->last_avail_idx || vq->avail_idx) {
+			pr_debug(
+				"strange VHOST_SET_VRING_BASE [vq=%p][s.index=%u][s.num=%u]",
+				vq, s.index, s.num);
+			dump_stack();
+			r = 0;
+			break;
+		}
 		vq->last_avail_idx = s.num;
 		/* Forget the cached index value. */
 		vq->avail_idx = vq->last_avail_idx;
 		pr_debug(
-			"VHOST_SET_VRING_BASE [vq=%p][vq->last_avail_idx=%u][vq->avail_idx=%u]",
-			vq, vq->last_avail_idx, vq->avail_idx);
+			"VHOST_SET_VRING_BASE [vq=%p][vq->last_avail_idx=%u][vq->avail_idx=%u][s.index=%u][s.num=%u]",
+			vq, vq->last_avail_idx, vq->avail_idx, s.index, s.num);
 		break;
 	case VHOST_GET_VRING_BASE:
 		s.index = idx;
 		s.num = vq->last_avail_idx;
 		if (copy_to_user(argp, &s, sizeof s))
 			r = -EFAULT;
+		pr_debug(
+			"VHOST_GET_VRING_BASE [vq=%p][vq->last_avail_idx=%u][vq->avail_idx=%u][s.index=%u][s.num=%u]",
+			vq, vq->last_avail_idx, vq->avail_idx, s.index, s.num);
 		break;
 	case VHOST_SET_VRING_KICK:
 		if (copy_from_user(&f, argp, sizeof f)) {
-- 
2.18.1

Re: vhost changes (batched) in linux-next after 12/13 trigger random crashes in KVM guests after reboot

[Christian Borntraeger]

On 13.02.20 17:29, Eugenio Pérez wrote:
> Can we try with this traces?

Does not apply on eccb852f1fe6bede630e2e4f1a121a81e34354ab, can you double check?

> 
> From b793b4106085ab1970bdedb340e49f37843ed585 Mon Sep 17 00:00:00 2001
> From: =?UTF-8?q?Eugenio=20P=C3=A9rez?= <eperezma@redhat.com>
> Date: Thu, 13 Feb 2020 17:27:05 +0100
> Subject: [PATCH] vhost: Add debug in ioctl calls
> 
> ---
>  drivers/vhost/net.c   | 20 +++++++++++++++++---
>  drivers/vhost/vhost.c | 16 ++++++++++++++--
>  2 files changed, 31 insertions(+), 5 deletions(-)
> 
> diff --git a/drivers/vhost/net.c b/drivers/vhost/net.c
> index e158159671fa..e4d5f843f9c0 100644
> --- a/drivers/vhost/net.c
> +++ b/drivers/vhost/net.c
> @@ -1505,10 +1505,13 @@ static long vhost_net_set_backend(struct vhost_net *n, unsigned index, int fd)
>  
>  	mutex_lock(&n->dev.mutex);
>  	r = vhost_dev_check_owner(&n->dev);
> -	if (r)
> +	if (r) {
> +		pr_debug("vhost_dev_check_owner index=%u fd=%d rc r=%d", index, fd, r);
>  		goto err;
> +	}
>  
>  	if (index >= VHOST_NET_VQ_MAX) {
> +		pr_debug("vhost_dev_check_owner index=%u fd=%d MAX=%d", index, fd, VHOST_NET_VQ_MAX);
>  		r = -ENOBUFS;
>  		goto err;
>  	}
> @@ -1518,22 +1521,26 @@ static long vhost_net_set_backend(struct vhost_net *n, unsigned index, int fd)
>  
>  	/* Verify that ring has been setup correctly. */
>  	if (!vhost_vq_access_ok(vq)) {
> +		pr_debug("vhost_net_set_backend index=%u fd=%d !vhost_vq_access_ok", index, fd);
>  		r = -EFAULT;
>  		goto err_vq;
>  	}
>  	sock = get_socket(fd);
>  	if (IS_ERR(sock)) {
>  		r = PTR_ERR(sock);
> +		pr_debug("vhost_net_set_backend index=%u fd=%d get_socket err r=%d", index, fd, r);
>  		goto err_vq;
>  	}
>  
>  	/* start polling new socket */
>  	oldsock = vq->private_data;
>  	if (sock != oldsock) {
> +		pr_debug("sock=%p != oldsock=%p index=%u fd=%d vq=%p", sock, oldsock, index, fd, vq);
>  		ubufs = vhost_net_ubuf_alloc(vq,
>  					     sock && vhost_sock_zcopy(sock));
>  		if (IS_ERR(ubufs)) {
>  			r = PTR_ERR(ubufs);
> +			pr_debug("ubufs index=%u fd=%d err r=%d vq=%p", index, fd, r, vq);
>  			goto err_ubufs;
>  		}
>  
> @@ -1541,11 +1548,15 @@ static long vhost_net_set_backend(struct vhost_net *n, unsigned index, int fd)
>  		vq->private_data = sock;
>  		vhost_net_buf_unproduce(nvq);
>  		r = vhost_vq_init_access(vq);
> -		if (r)
> +		if (r) {
> +			pr_debug("init_access index=%u fd=%d r=%d vq=%p", index, fd, r, vq);
>  			goto err_used;
> +		}
>  		r = vhost_net_enable_vq(n, vq);
> -		if (r)
> +		if (r) {
> +			pr_debug("enable_vq index=%u fd=%d r=%d vq=%p", index, fd, r, vq);
>  			goto err_used;
> +		}
>  		if (index == VHOST_NET_VQ_RX)
>  			nvq->rx_ring = get_tap_ptr_ring(fd);
>  
> @@ -1559,6 +1570,8 @@ static long vhost_net_set_backend(struct vhost_net *n, unsigned index, int fd)
>  
>  	mutex_unlock(&vq->mutex);
>  
> +	pr_debug("sock=%p", sock);
> +
>  	if (oldubufs) {
>  		vhost_net_ubuf_put_wait_and_free(oldubufs);
>  		mutex_lock(&vq->mutex);
> @@ -1710,6 +1723,7 @@ static long vhost_net_ioctl(struct file *f, unsigned int ioctl,
>  
>  	switch (ioctl) {
>  	case VHOST_NET_SET_BACKEND:
> +		pr_debug("VHOST_NET_SET_BACKEND");
>  		if (copy_from_user(&backend, argp, sizeof backend))
>  			return -EFAULT;
>  		return vhost_net_set_backend(n, backend.index, backend.fd);
> diff --git a/drivers/vhost/vhost.c b/drivers/vhost/vhost.c
> index 021d70bed015..7f4848f9cec3 100644
> --- a/drivers/vhost/vhost.c
> +++ b/drivers/vhost/vhost.c
> @@ -1642,18 +1642,30 @@ long vhost_vring_ioctl(struct vhost_dev *d, unsigned int ioctl, void __user *arg
>  			r = -EINVAL;
>  			break;
>  		}
> +
> +		if (vq->last_avail_idx || vq->avail_idx) {
> +			pr_debug(
> +				"strange VHOST_SET_VRING_BASE [vq=%p][s.index=%u][s.num=%u]",
> +				vq, s.index, s.num);
> +			dump_stack();
> +			r = 0;
> +			break;
> +		}
>  		vq->last_avail_idx = s.num;
>  		/* Forget the cached index value. */
>  		vq->avail_idx = vq->last_avail_idx;
>  		pr_debug(
> -			"VHOST_SET_VRING_BASE [vq=%p][vq->last_avail_idx=%u][vq->avail_idx=%u]",
> -			vq, vq->last_avail_idx, vq->avail_idx);
> +			"VHOST_SET_VRING_BASE [vq=%p][vq->last_avail_idx=%u][vq->avail_idx=%u][s.index=%u][s.num=%u]",
> +			vq, vq->last_avail_idx, vq->avail_idx, s.index, s.num);
>  		break;
>  	case VHOST_GET_VRING_BASE:
>  		s.index = idx;
>  		s.num = vq->last_avail_idx;
>  		if (copy_to_user(argp, &s, sizeof s))
>  			r = -EFAULT;
> +		pr_debug(
> +			"VHOST_GET_VRING_BASE [vq=%p][vq->last_avail_idx=%u][vq->avail_idx=%u][s.index=%u][s.num=%u]",
> +			vq, vq->last_avail_idx, vq->avail_idx, s.index, s.num);
>  		break;
>  	case VHOST_SET_VRING_KICK:
>  		if (copy_from_user(&f, argp, sizeof f)) {
> 

Re: vhost changes (batched) in linux-next after 12/13 trigger random crashes in KVM guests after reboot

[Eugenio Pérez]

Hi Christian.

Sorry, that was meant to be applied over previous debug patch.

Here I inline the one meant to be applied over eccb852f1fe6bede630e2e4f1a121a81e34354ab.

Thanks!

From d978ace99e4844b49b794d768385db3d128a4cc0 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Eugenio=20P=C3=A9rez?= <eperezma@redhat.com>
Date: Fri, 14 Feb 2020 08:02:26 +0100
Subject: [PATCH] vhost: disable all features and trace last_avail_idx and
 ioctl calls

---
 drivers/vhost/net.c   | 20 +++++++++++++++++---
 drivers/vhost/vhost.c | 25 +++++++++++++++++++++++--
 drivers/vhost/vhost.h | 10 +++++-----
 3 files changed, 45 insertions(+), 10 deletions(-)

diff --git a/drivers/vhost/net.c b/drivers/vhost/net.c
index e158159671fa..e4d5f843f9c0 100644
--- a/drivers/vhost/net.c
+++ b/drivers/vhost/net.c
@@ -1505,10 +1505,13 @@ static long vhost_net_set_backend(struct vhost_net *n, unsigned index, int fd)
 
 	mutex_lock(&n->dev.mutex);
 	r = vhost_dev_check_owner(&n->dev);
-	if (r)
+	if (r) {
+		pr_debug("vhost_dev_check_owner index=%u fd=%d rc r=%d", index, fd, r);
 		goto err;
+	}
 
 	if (index >= VHOST_NET_VQ_MAX) {
+		pr_debug("vhost_dev_check_owner index=%u fd=%d MAX=%d", index, fd, VHOST_NET_VQ_MAX);
 		r = -ENOBUFS;
 		goto err;
 	}
@@ -1518,22 +1521,26 @@ static long vhost_net_set_backend(struct vhost_net *n, unsigned index, int fd)
 
 	/* Verify that ring has been setup correctly. */
 	if (!vhost_vq_access_ok(vq)) {
+		pr_debug("vhost_net_set_backend index=%u fd=%d !vhost_vq_access_ok", index, fd);
 		r = -EFAULT;
 		goto err_vq;
 	}
 	sock = get_socket(fd);
 	if (IS_ERR(sock)) {
 		r = PTR_ERR(sock);
+		pr_debug("vhost_net_set_backend index=%u fd=%d get_socket err r=%d", index, fd, r);
 		goto err_vq;
 	}
 
 	/* start polling new socket */
 	oldsock = vq->private_data;
 	if (sock != oldsock) {
+		pr_debug("sock=%p != oldsock=%p index=%u fd=%d vq=%p", sock, oldsock, index, fd, vq);
 		ubufs = vhost_net_ubuf_alloc(vq,
 					     sock && vhost_sock_zcopy(sock));
 		if (IS_ERR(ubufs)) {
 			r = PTR_ERR(ubufs);
+			pr_debug("ubufs index=%u fd=%d err r=%d vq=%p", index, fd, r, vq);
 			goto err_ubufs;
 		}
 
@@ -1541,11 +1548,15 @@ static long vhost_net_set_backend(struct vhost_net *n, unsigned index, int fd)
 		vq->private_data = sock;
 		vhost_net_buf_unproduce(nvq);
 		r = vhost_vq_init_access(vq);
-		if (r)
+		if (r) {
+			pr_debug("init_access index=%u fd=%d r=%d vq=%p", index, fd, r, vq);
 			goto err_used;
+		}
 		r = vhost_net_enable_vq(n, vq);
-		if (r)
+		if (r) {
+			pr_debug("enable_vq index=%u fd=%d r=%d vq=%p", index, fd, r, vq);
 			goto err_used;
+		}
 		if (index == VHOST_NET_VQ_RX)
 			nvq->rx_ring = get_tap_ptr_ring(fd);
 
@@ -1559,6 +1570,8 @@ static long vhost_net_set_backend(struct vhost_net *n, unsigned index, int fd)
 
 	mutex_unlock(&vq->mutex);
 
+	pr_debug("sock=%p", sock);
+
 	if (oldubufs) {
 		vhost_net_ubuf_put_wait_and_free(oldubufs);
 		mutex_lock(&vq->mutex);
@@ -1710,6 +1723,7 @@ static long vhost_net_ioctl(struct file *f, unsigned int ioctl,
 
 	switch (ioctl) {
 	case VHOST_NET_SET_BACKEND:
+		pr_debug("VHOST_NET_SET_BACKEND");
 		if (copy_from_user(&backend, argp, sizeof backend))
 			return -EFAULT;
 		return vhost_net_set_backend(n, backend.index, backend.fd);
diff --git a/drivers/vhost/vhost.c b/drivers/vhost/vhost.c
index b5a51b1f2e79..ec25ba32fe81 100644
--- a/drivers/vhost/vhost.c
+++ b/drivers/vhost/vhost.c
@@ -1642,15 +1642,30 @@ long vhost_vring_ioctl(struct vhost_dev *d, unsigned int ioctl, void __user *arg
 			r = -EINVAL;
 			break;
 		}
+
+		if (vq->last_avail_idx || vq->avail_idx) {
+			pr_debug(
+				"strange VHOST_SET_VRING_BASE [vq=%p][s.index=%u][s.num=%u]",
+				vq, s.index, s.num);
+			dump_stack();
+			r = 0;
+			break;
+		}
 		vq->last_avail_idx = s.num;
 		/* Forget the cached index value. */
 		vq->avail_idx = vq->last_avail_idx;
+		pr_debug(
+			"VHOST_SET_VRING_BASE [vq=%p][vq->last_avail_idx=%u][vq->avail_idx=%u][s.index=%u][s.num=%u]",
+			vq, vq->last_avail_idx, vq->avail_idx, s.index, s.num);
 		break;
 	case VHOST_GET_VRING_BASE:
 		s.index = idx;
 		s.num = vq->last_avail_idx;
 		if (copy_to_user(argp, &s, sizeof s))
 			r = -EFAULT;
+		pr_debug(
+			"VHOST_GET_VRING_BASE [vq=%p][vq->last_avail_idx=%u][vq->avail_idx=%u][s.index=%u][s.num=%u]",
+			vq, vq->last_avail_idx, vq->avail_idx, s.index, s.num);
 		break;
 	case VHOST_SET_VRING_KICK:
 		if (copy_from_user(&f, argp, sizeof f)) {
@@ -2239,8 +2254,8 @@ static int fetch_buf(struct vhost_virtqueue *vq)
 		vq->avail_idx = vhost16_to_cpu(vq, avail_idx);
 
 		if (unlikely((u16)(vq->avail_idx - last_avail_idx) > vq->num)) {
-			vq_err(vq, "Guest moved used index from %u to %u",
-				last_avail_idx, vq->avail_idx);
+			vq_err(vq, "Guest moved vq %p used index from %u to %u",
+				vq, last_avail_idx, vq->avail_idx);
 			return -EFAULT;
 		}
 
@@ -2316,6 +2331,9 @@ static int fetch_buf(struct vhost_virtqueue *vq)
 	BUG_ON(!(vq->used_flags & VRING_USED_F_NO_NOTIFY));
 
 	/* On success, increment avail index. */
+	pr_debug(
+		"[vq=%p][vq->last_avail_idx=%u][vq->avail_idx=%u][vq->ndescs=%d][vq->first_desc=%d]",
+		vq, vq->last_avail_idx, vq->avail_idx, vq->ndescs, vq->first_desc);
 	vq->last_avail_idx++;
 
 	return 0;
@@ -2432,6 +2450,9 @@ EXPORT_SYMBOL_GPL(vhost_get_vq_desc);
 /* Reverse the effect of vhost_get_vq_desc. Useful for error handling. */
 void vhost_discard_vq_desc(struct vhost_virtqueue *vq, int n)
 {
+	pr_debug(
+		"DISCARD [vq=%p][vq->last_avail_idx=%u][vq->avail_idx=%u][n=%d]",
+		vq, vq->last_avail_idx, vq->avail_idx, n);
 	vq->last_avail_idx -= n;
 }
 EXPORT_SYMBOL_GPL(vhost_discard_vq_desc);
diff --git a/drivers/vhost/vhost.h b/drivers/vhost/vhost.h
index 661088ae6dc7..08f6d2ccb697 100644
--- a/drivers/vhost/vhost.h
+++ b/drivers/vhost/vhost.h
@@ -250,11 +250,11 @@ int vhost_init_device_iotlb(struct vhost_dev *d, bool enabled);
 	} while (0)
 
 enum {
-	VHOST_FEATURES = (1ULL << VIRTIO_F_NOTIFY_ON_EMPTY) |
-			 (1ULL << VIRTIO_RING_F_INDIRECT_DESC) |
-			 (1ULL << VIRTIO_RING_F_EVENT_IDX) |
-			 (1ULL << VHOST_F_LOG_ALL) |
-			 (1ULL << VIRTIO_F_ANY_LAYOUT) |
+	VHOST_FEATURES = /* (1ULL << VIRTIO_F_NOTIFY_ON_EMPTY) | */
+			 /* (1ULL << VIRTIO_RING_F_INDIRECT_DESC) | */
+			 /* (1ULL << VIRTIO_RING_F_EVENT_IDX) | */
+			 /* (1ULL << VHOST_F_LOG_ALL) | */
+			 /* (1ULL << VIRTIO_F_ANY_LAYOUT) | */
 			 (1ULL << VIRTIO_F_VERSION_1)
 };
 
-- 
2.18.1

Re: vhost changes (batched) in linux-next after 12/13 trigger random crashes in KVM guests after reboot

[Christian Borntraeger]

I did 
ping -c 20 -f ... ; reboot 
twice

The ping after the first reboot showed .......E 

this was on the host console

[   55.951885] CPU: 34 PID: 1908 Comm: CPU 0/KVM Not tainted 5.5.0+ #21
[   55.951891] Hardware name: IBM 3906 M04 704 (LPAR)
[   55.951892] Call Trace:
[   55.951902]  [<0000001ede114132>] show_stack+0x8a/0xd0 
[   55.951906]  [<0000001edeb0672a>] dump_stack+0x8a/0xb8 
[   55.951915]  [<000003ff803736a6>] vhost_vring_ioctl+0x6fe/0x858 [vhost] 
[   55.951919]  [<000003ff8042a608>] vhost_net_ioctl+0x510/0x570 [vhost_net] 
[   55.951924]  [<0000001ede3c4dd8>] do_vfs_ioctl+0x430/0x6f8 
[   55.951926]  [<0000001ede3c5124>] ksys_ioctl+0x84/0xb0 
[   55.951927]  [<0000001ede3c51ba>] __s390x_sys_ioctl+0x2a/0x38 
[   55.951931]  [<0000001edeb27f72>] system_call+0x2a6/0x2c8 
[   55.951949] CPU: 34 PID: 1908 Comm: CPU 0/KVM Not tainted 5.5.0+ #21
[   55.951950] Hardware name: IBM 3906 M04 704 (LPAR)
[   55.951951] Call Trace:
[   55.951952]  [<0000001ede114132>] show_stack+0x8a/0xd0 
[   55.951954]  [<0000001edeb0672a>] dump_stack+0x8a/0xb8 
[   55.951956]  [<000003ff803736a6>] vhost_vring_ioctl+0x6fe/0x858 [vhost] 
[   55.951958]  [<000003ff8042a608>] vhost_net_ioctl+0x510/0x570 [vhost_net] 
[   55.951959]  [<0000001ede3c4dd8>] do_vfs_ioctl+0x430/0x6f8 
[   55.951961]  [<0000001ede3c5124>] ksys_ioctl+0x84/0xb0 
[   55.951962]  [<0000001ede3c51ba>] __s390x_sys_ioctl+0x2a/0x38 
[   55.951964]  [<0000001edeb27f72>] system_call+0x2a6/0x2c8 
[   55.951997] Guest moved vq 0000000063d896c6 used index from 44 to 0
[   56.609831] unexpected descriptor format for RX: out 0, in 0
[   86.540460] CPU: 6 PID: 1908 Comm: CPU 0/KVM Not tainted 5.5.0+ #21
[   86.540464] Hardware name: IBM 3906 M04 704 (LPAR)
[   86.540466] Call Trace:
[   86.540473]  [<0000001ede114132>] show_stack+0x8a/0xd0 
[   86.540477]  [<0000001edeb0672a>] dump_stack+0x8a/0xb8 
[   86.540486]  [<000003ff803736a6>] vhost_vring_ioctl+0x6fe/0x858 [vhost] 
[   86.540490]  [<000003ff8042a608>] vhost_net_ioctl+0x510/0x570 [vhost_net] 
[   86.540494]  [<0000001ede3c4dd8>] do_vfs_ioctl+0x430/0x6f8 
[   86.540496]  [<0000001ede3c5124>] ksys_ioctl+0x84/0xb0 
[   86.540498]  [<0000001ede3c51ba>] __s390x_sys_ioctl+0x2a/0x38 
[   86.540501]  [<0000001edeb27f72>] system_call+0x2a6/0x2c8 
[   86.540524] CPU: 6 PID: 1908 Comm: CPU 0/KVM Not tainted 5.5.0+ #21
[   86.540525] Hardware name: IBM 3906 M04 704 (LPAR)
[   86.540526] Call Trace:
[   86.540527]  [<0000001ede114132>] show_stack+0x8a/0xd0 
[   86.540528]  [<0000001edeb0672a>] dump_stack+0x8a/0xb8 
[   86.540531]  [<000003ff803736a6>] vhost_vring_ioctl+0x6fe/0x858 [vhost] 
[   86.540532]  [<000003ff8042a608>] vhost_net_ioctl+0x510/0x570 [vhost_net] 
[   86.540534]  [<0000001ede3c4dd8>] do_vfs_ioctl+0x430/0x6f8 
[   86.540536]  [<0000001ede3c5124>] ksys_ioctl+0x84/0xb0 
[   86.540537]  [<0000001ede3c51ba>] __s390x_sys_ioctl+0x2a/0x38 
[   86.540538]  [<0000001edeb27f72>] system_call+0x2a6/0x2c8 
[   86.540570] unexpected descriptor format for RX: out 0, in 0
[   86.540577] Unexpected header len for TX: 0 expected 0


On 14.02.20 08:06, Eugenio Pérez wrote:
> Hi Christian.
> 
> Sorry, that was meant to be applied over previous debug patch.
> 
> Here I inline the one meant to be applied over eccb852f1fe6bede630e2e4f1a121a81e34354ab.
> 
> Thanks!
> 
> From d978ace99e4844b49b794d768385db3d128a4cc0 Mon Sep 17 00:00:00 2001
> From: =?UTF-8?q?Eugenio=20P=C3=A9rez?= <eperezma@redhat.com>
> Date: Fri, 14 Feb 2020 08:02:26 +0100
> Subject: [PATCH] vhost: disable all features and trace last_avail_idx and
>  ioctl calls
> 
> ---
>  drivers/vhost/net.c   | 20 +++++++++++++++++---
>  drivers/vhost/vhost.c | 25 +++++++++++++++++++++++--
>  drivers/vhost/vhost.h | 10 +++++-----
>  3 files changed, 45 insertions(+), 10 deletions(-)
> 
> diff --git a/drivers/vhost/net.c b/drivers/vhost/net.c
> index e158159671fa..e4d5f843f9c0 100644
> --- a/drivers/vhost/net.c
> +++ b/drivers/vhost/net.c
> @@ -1505,10 +1505,13 @@ static long vhost_net_set_backend(struct vhost_net *n, unsigned index, int fd)
>  
>  	mutex_lock(&n->dev.mutex);
>  	r = vhost_dev_check_owner(&n->dev);
> -	if (r)
> +	if (r) {
> +		pr_debug("vhost_dev_check_owner index=%u fd=%d rc r=%d", index, fd, r);
>  		goto err;
> +	}
>  
>  	if (index >= VHOST_NET_VQ_MAX) {
> +		pr_debug("vhost_dev_check_owner index=%u fd=%d MAX=%d", index, fd, VHOST_NET_VQ_MAX);
>  		r = -ENOBUFS;
>  		goto err;
>  	}
> @@ -1518,22 +1521,26 @@ static long vhost_net_set_backend(struct vhost_net *n, unsigned index, int fd)
>  
>  	/* Verify that ring has been setup correctly. */
>  	if (!vhost_vq_access_ok(vq)) {
> +		pr_debug("vhost_net_set_backend index=%u fd=%d !vhost_vq_access_ok", index, fd);
>  		r = -EFAULT;
>  		goto err_vq;
>  	}
>  	sock = get_socket(fd);
>  	if (IS_ERR(sock)) {
>  		r = PTR_ERR(sock);
> +		pr_debug("vhost_net_set_backend index=%u fd=%d get_socket err r=%d", index, fd, r);
>  		goto err_vq;
>  	}
>  
>  	/* start polling new socket */
>  	oldsock = vq->private_data;
>  	if (sock != oldsock) {
> +		pr_debug("sock=%p != oldsock=%p index=%u fd=%d vq=%p", sock, oldsock, index, fd, vq);
>  		ubufs = vhost_net_ubuf_alloc(vq,
>  					     sock && vhost_sock_zcopy(sock));
>  		if (IS_ERR(ubufs)) {
>  			r = PTR_ERR(ubufs);
> +			pr_debug("ubufs index=%u fd=%d err r=%d vq=%p", index, fd, r, vq);
>  			goto err_ubufs;
>  		}
>  
> @@ -1541,11 +1548,15 @@ static long vhost_net_set_backend(struct vhost_net *n, unsigned index, int fd)
>  		vq->private_data = sock;
>  		vhost_net_buf_unproduce(nvq);
>  		r = vhost_vq_init_access(vq);
> -		if (r)
> +		if (r) {
> +			pr_debug("init_access index=%u fd=%d r=%d vq=%p", index, fd, r, vq);
>  			goto err_used;
> +		}
>  		r = vhost_net_enable_vq(n, vq);
> -		if (r)
> +		if (r) {
> +			pr_debug("enable_vq index=%u fd=%d r=%d vq=%p", index, fd, r, vq);
>  			goto err_used;
> +		}
>  		if (index == VHOST_NET_VQ_RX)
>  			nvq->rx_ring = get_tap_ptr_ring(fd);
>  
> @@ -1559,6 +1570,8 @@ static long vhost_net_set_backend(struct vhost_net *n, unsigned index, int fd)
>  
>  	mutex_unlock(&vq->mutex);
>  
> +	pr_debug("sock=%p", sock);
> +
>  	if (oldubufs) {
>  		vhost_net_ubuf_put_wait_and_free(oldubufs);
>  		mutex_lock(&vq->mutex);
> @@ -1710,6 +1723,7 @@ static long vhost_net_ioctl(struct file *f, unsigned int ioctl,
>  
>  	switch (ioctl) {
>  	case VHOST_NET_SET_BACKEND:
> +		pr_debug("VHOST_NET_SET_BACKEND");
>  		if (copy_from_user(&backend, argp, sizeof backend))
>  			return -EFAULT;
>  		return vhost_net_set_backend(n, backend.index, backend.fd);
> diff --git a/drivers/vhost/vhost.c b/drivers/vhost/vhost.c
> index b5a51b1f2e79..ec25ba32fe81 100644
> --- a/drivers/vhost/vhost.c
> +++ b/drivers/vhost/vhost.c
> @@ -1642,15 +1642,30 @@ long vhost_vring_ioctl(struct vhost_dev *d, unsigned int ioctl, void __user *arg
>  			r = -EINVAL;
>  			break;
>  		}
> +
> +		if (vq->last_avail_idx || vq->avail_idx) {
> +			pr_debug(
> +				"strange VHOST_SET_VRING_BASE [vq=%p][s.index=%u][s.num=%u]",
> +				vq, s.index, s.num);
> +			dump_stack();
> +			r = 0;
> +			break;
> +		}
>  		vq->last_avail_idx = s.num;
>  		/* Forget the cached index value. */
>  		vq->avail_idx = vq->last_avail_idx;
> +		pr_debug(
> +			"VHOST_SET_VRING_BASE [vq=%p][vq->last_avail_idx=%u][vq->avail_idx=%u][s.index=%u][s.num=%u]",
> +			vq, vq->last_avail_idx, vq->avail_idx, s.index, s.num);
>  		break;
>  	case VHOST_GET_VRING_BASE:
>  		s.index = idx;
>  		s.num = vq->last_avail_idx;
>  		if (copy_to_user(argp, &s, sizeof s))
>  			r = -EFAULT;
> +		pr_debug(
> +			"VHOST_GET_VRING_BASE [vq=%p][vq->last_avail_idx=%u][vq->avail_idx=%u][s.index=%u][s.num=%u]",
> +			vq, vq->last_avail_idx, vq->avail_idx, s.index, s.num);
>  		break;
>  	case VHOST_SET_VRING_KICK:
>  		if (copy_from_user(&f, argp, sizeof f)) {
> @@ -2239,8 +2254,8 @@ static int fetch_buf(struct vhost_virtqueue *vq)
>  		vq->avail_idx = vhost16_to_cpu(vq, avail_idx);
>  
>  		if (unlikely((u16)(vq->avail_idx - last_avail_idx) > vq->num)) {
> -			vq_err(vq, "Guest moved used index from %u to %u",
> -				last_avail_idx, vq->avail_idx);
> +			vq_err(vq, "Guest moved vq %p used index from %u to %u",
> +				vq, last_avail_idx, vq->avail_idx);
>  			return -EFAULT;
>  		}
>  
> @@ -2316,6 +2331,9 @@ static int fetch_buf(struct vhost_virtqueue *vq)
>  	BUG_ON(!(vq->used_flags & VRING_USED_F_NO_NOTIFY));
>  
>  	/* On success, increment avail index. */
> +	pr_debug(
> +		"[vq=%p][vq->last_avail_idx=%u][vq->avail_idx=%u][vq->ndescs=%d][vq->first_desc=%d]",
> +		vq, vq->last_avail_idx, vq->avail_idx, vq->ndescs, vq->first_desc);
>  	vq->last_avail_idx++;
>  
>  	return 0;
> @@ -2432,6 +2450,9 @@ EXPORT_SYMBOL_GPL(vhost_get_vq_desc);
>  /* Reverse the effect of vhost_get_vq_desc. Useful for error handling. */
>  void vhost_discard_vq_desc(struct vhost_virtqueue *vq, int n)
>  {
> +	pr_debug(
> +		"DISCARD [vq=%p][vq->last_avail_idx=%u][vq->avail_idx=%u][n=%d]",
> +		vq, vq->last_avail_idx, vq->avail_idx, n);
>  	vq->last_avail_idx -= n;
>  }
>  EXPORT_SYMBOL_GPL(vhost_discard_vq_desc);
> diff --git a/drivers/vhost/vhost.h b/drivers/vhost/vhost.h
> index 661088ae6dc7..08f6d2ccb697 100644
> --- a/drivers/vhost/vhost.h
> +++ b/drivers/vhost/vhost.h
> @@ -250,11 +250,11 @@ int vhost_init_device_iotlb(struct vhost_dev *d, bool enabled);
>  	} while (0)
>  
>  enum {
> -	VHOST_FEATURES = (1ULL << VIRTIO_F_NOTIFY_ON_EMPTY) |
> -			 (1ULL << VIRTIO_RING_F_INDIRECT_DESC) |
> -			 (1ULL << VIRTIO_RING_F_EVENT_IDX) |
> -			 (1ULL << VHOST_F_LOG_ALL) |
> -			 (1ULL << VIRTIO_F_ANY_LAYOUT) |
> +	VHOST_FEATURES = /* (1ULL << VIRTIO_F_NOTIFY_ON_EMPTY) | */
> +			 /* (1ULL << VIRTIO_RING_F_INDIRECT_DESC) | */
> +			 /* (1ULL << VIRTIO_RING_F_EVENT_IDX) | */
> +			 /* (1ULL << VHOST_F_LOG_ALL) | */
> +			 /* (1ULL << VIRTIO_F_ANY_LAYOUT) | */
>  			 (1ULL << VIRTIO_F_VERSION_1)
>  };
>  
> 

Re: vhost changes (batched) in linux-next after 12/13 trigger random crashes in KVM guests after reboot

[Eugenio Perez Martin]

Hi.

Were the vhost and vhost_net modules loaded with dyndbg='+plt'? I miss
all the others regular debug traces on that one.

Thanks!

On Fri, Feb 14, 2020 at 8:34 AM Christian Borntraeger
<borntraeger@de.ibm.com> wrote:
>
> I did
> ping -c 20 -f ... ; reboot
> twice
>
> The ping after the first reboot showed .......E
>
> this was on the host console
>
> [   55.951885] CPU: 34 PID: 1908 Comm: CPU 0/KVM Not tainted 5.5.0+ #21
> [   55.951891] Hardware name: IBM 3906 M04 704 (LPAR)
> [   55.951892] Call Trace:
> [   55.951902]  [<0000001ede114132>] show_stack+0x8a/0xd0
> [   55.951906]  [<0000001edeb0672a>] dump_stack+0x8a/0xb8
> [   55.951915]  [<000003ff803736a6>] vhost_vring_ioctl+0x6fe/0x858 [vhost]
> [   55.951919]  [<000003ff8042a608>] vhost_net_ioctl+0x510/0x570 [vhost_net]
> [   55.951924]  [<0000001ede3c4dd8>] do_vfs_ioctl+0x430/0x6f8
> [   55.951926]  [<0000001ede3c5124>] ksys_ioctl+0x84/0xb0
> [   55.951927]  [<0000001ede3c51ba>] __s390x_sys_ioctl+0x2a/0x38
> [   55.951931]  [<0000001edeb27f72>] system_call+0x2a6/0x2c8
> [   55.951949] CPU: 34 PID: 1908 Comm: CPU 0/KVM Not tainted 5.5.0+ #21
> [   55.951950] Hardware name: IBM 3906 M04 704 (LPAR)
> [   55.951951] Call Trace:
> [   55.951952]  [<0000001ede114132>] show_stack+0x8a/0xd0
> [   55.951954]  [<0000001edeb0672a>] dump_stack+0x8a/0xb8
> [   55.951956]  [<000003ff803736a6>] vhost_vring_ioctl+0x6fe/0x858 [vhost]
> [   55.951958]  [<000003ff8042a608>] vhost_net_ioctl+0x510/0x570 [vhost_net]
> [   55.951959]  [<0000001ede3c4dd8>] do_vfs_ioctl+0x430/0x6f8
> [   55.951961]  [<0000001ede3c5124>] ksys_ioctl+0x84/0xb0
> [   55.951962]  [<0000001ede3c51ba>] __s390x_sys_ioctl+0x2a/0x38
> [   55.951964]  [<0000001edeb27f72>] system_call+0x2a6/0x2c8
> [   55.951997] Guest moved vq 0000000063d896c6 used index from 44 to 0
> [   56.609831] unexpected descriptor format for RX: out 0, in 0
> [   86.540460] CPU: 6 PID: 1908 Comm: CPU 0/KVM Not tainted 5.5.0+ #21
> [   86.540464] Hardware name: IBM 3906 M04 704 (LPAR)
> [   86.540466] Call Trace:
> [   86.540473]  [<0000001ede114132>] show_stack+0x8a/0xd0
> [   86.540477]  [<0000001edeb0672a>] dump_stack+0x8a/0xb8
> [   86.540486]  [<000003ff803736a6>] vhost_vring_ioctl+0x6fe/0x858 [vhost]
> [   86.540490]  [<000003ff8042a608>] vhost_net_ioctl+0x510/0x570 [vhost_net]
> [   86.540494]  [<0000001ede3c4dd8>] do_vfs_ioctl+0x430/0x6f8
> [   86.540496]  [<0000001ede3c5124>] ksys_ioctl+0x84/0xb0
> [   86.540498]  [<0000001ede3c51ba>] __s390x_sys_ioctl+0x2a/0x38
> [   86.540501]  [<0000001edeb27f72>] system_call+0x2a6/0x2c8
> [   86.540524] CPU: 6 PID: 1908 Comm: CPU 0/KVM Not tainted 5.5.0+ #21
> [   86.540525] Hardware name: IBM 3906 M04 704 (LPAR)
> [   86.540526] Call Trace:
> [   86.540527]  [<0000001ede114132>] show_stack+0x8a/0xd0
> [   86.540528]  [<0000001edeb0672a>] dump_stack+0x8a/0xb8
> [   86.540531]  [<000003ff803736a6>] vhost_vring_ioctl+0x6fe/0x858 [vhost]
> [   86.540532]  [<000003ff8042a608>] vhost_net_ioctl+0x510/0x570 [vhost_net]
> [   86.540534]  [<0000001ede3c4dd8>] do_vfs_ioctl+0x430/0x6f8
> [   86.540536]  [<0000001ede3c5124>] ksys_ioctl+0x84/0xb0
> [   86.540537]  [<0000001ede3c51ba>] __s390x_sys_ioctl+0x2a/0x38
> [   86.540538]  [<0000001edeb27f72>] system_call+0x2a6/0x2c8
> [   86.540570] unexpected descriptor format for RX: out 0, in 0
> [   86.540577] Unexpected header len for TX: 0 expected 0
>
>
> On 14.02.20 08:06, Eugenio Pérez wrote:
> > Hi Christian.
> >
> > Sorry, that was meant to be applied over previous debug patch.
> >
> > Here I inline the one meant to be applied over eccb852f1fe6bede630e2e4f1a121a81e34354ab.
> >
> > Thanks!
> >
> > From d978ace99e4844b49b794d768385db3d128a4cc0 Mon Sep 17 00:00:00 2001
> > From: =?UTF-8?q?Eugenio=20P=C3=A9rez?= <eperezma@redhat.com>
> > Date: Fri, 14 Feb 2020 08:02:26 +0100
> > Subject: [PATCH] vhost: disable all features and trace last_avail_idx and
> >  ioctl calls
> >
> > ---
> >  drivers/vhost/net.c   | 20 +++++++++++++++++---
> >  drivers/vhost/vhost.c | 25 +++++++++++++++++++++++--
> >  drivers/vhost/vhost.h | 10 +++++-----
> >  3 files changed, 45 insertions(+), 10 deletions(-)
> >
> > diff --git a/drivers/vhost/net.c b/drivers/vhost/net.c
> > index e158159671fa..e4d5f843f9c0 100644
> > --- a/drivers/vhost/net.c
> > +++ b/drivers/vhost/net.c
> > @@ -1505,10 +1505,13 @@ static long vhost_net_set_backend(struct vhost_net *n, unsigned index, int fd)
> >
> >       mutex_lock(&n->dev.mutex);
> >       r = vhost_dev_check_owner(&n->dev);
> > -     if (r)
> > +     if (r) {
> > +             pr_debug("vhost_dev_check_owner index=%u fd=%d rc r=%d", index, fd, r);
> >               goto err;
> > +     }
> >
> >       if (index >= VHOST_NET_VQ_MAX) {
> > +             pr_debug("vhost_dev_check_owner index=%u fd=%d MAX=%d", index, fd, VHOST_NET_VQ_MAX);
> >               r = -ENOBUFS;
> >               goto err;
> >       }
> > @@ -1518,22 +1521,26 @@ static long vhost_net_set_backend(struct vhost_net *n, unsigned index, int fd)
> >
> >       /* Verify that ring has been setup correctly. */
> >       if (!vhost_vq_access_ok(vq)) {
> > +             pr_debug("vhost_net_set_backend index=%u fd=%d !vhost_vq_access_ok", index, fd);
> >               r = -EFAULT;
> >               goto err_vq;
> >       }
> >       sock = get_socket(fd);
> >       if (IS_ERR(sock)) {
> >               r = PTR_ERR(sock);
> > +             pr_debug("vhost_net_set_backend index=%u fd=%d get_socket err r=%d", index, fd, r);
> >               goto err_vq;
> >       }
> >
> >       /* start polling new socket */
> >       oldsock = vq->private_data;
> >       if (sock != oldsock) {
> > +             pr_debug("sock=%p != oldsock=%p index=%u fd=%d vq=%p", sock, oldsock, index, fd, vq);
> >               ubufs = vhost_net_ubuf_alloc(vq,
> >                                            sock && vhost_sock_zcopy(sock));
> >               if (IS_ERR(ubufs)) {
> >                       r = PTR_ERR(ubufs);
> > +                     pr_debug("ubufs index=%u fd=%d err r=%d vq=%p", index, fd, r, vq);
> >                       goto err_ubufs;
> >               }
> >
> > @@ -1541,11 +1548,15 @@ static long vhost_net_set_backend(struct vhost_net *n, unsigned index, int fd)
> >               vq->private_data = sock;
> >               vhost_net_buf_unproduce(nvq);
> >               r = vhost_vq_init_access(vq);
> > -             if (r)
> > +             if (r) {
> > +                     pr_debug("init_access index=%u fd=%d r=%d vq=%p", index, fd, r, vq);
> >                       goto err_used;
> > +             }
> >               r = vhost_net_enable_vq(n, vq);
> > -             if (r)
> > +             if (r) {
> > +                     pr_debug("enable_vq index=%u fd=%d r=%d vq=%p", index, fd, r, vq);
> >                       goto err_used;
> > +             }
> >               if (index == VHOST_NET_VQ_RX)
> >                       nvq->rx_ring = get_tap_ptr_ring(fd);
> >
> > @@ -1559,6 +1570,8 @@ static long vhost_net_set_backend(struct vhost_net *n, unsigned index, int fd)
> >
> >       mutex_unlock(&vq->mutex);
> >
> > +     pr_debug("sock=%p", sock);
> > +
> >       if (oldubufs) {
> >               vhost_net_ubuf_put_wait_and_free(oldubufs);
> >               mutex_lock(&vq->mutex);
> > @@ -1710,6 +1723,7 @@ static long vhost_net_ioctl(struct file *f, unsigned int ioctl,
> >
> >       switch (ioctl) {
> >       case VHOST_NET_SET_BACKEND:
> > +             pr_debug("VHOST_NET_SET_BACKEND");
> >               if (copy_from_user(&backend, argp, sizeof backend))
> >                       return -EFAULT;
> >               return vhost_net_set_backend(n, backend.index, backend.fd);
> > diff --git a/drivers/vhost/vhost.c b/drivers/vhost/vhost.c
> > index b5a51b1f2e79..ec25ba32fe81 100644
> > --- a/drivers/vhost/vhost.c
> > +++ b/drivers/vhost/vhost.c
> > @@ -1642,15 +1642,30 @@ long vhost_vring_ioctl(struct vhost_dev *d, unsigned int ioctl, void __user *arg
> >                       r = -EINVAL;
> >                       break;
> >               }
> > +
> > +             if (vq->last_avail_idx || vq->avail_idx) {
> > +                     pr_debug(
> > +                             "strange VHOST_SET_VRING_BASE [vq=%p][s.index=%u][s.num=%u]",
> > +                             vq, s.index, s.num);
> > +                     dump_stack();
> > +                     r = 0;
> > +                     break;
> > +             }
> >               vq->last_avail_idx = s.num;
> >               /* Forget the cached index value. */
> >               vq->avail_idx = vq->last_avail_idx;
> > +             pr_debug(
> > +                     "VHOST_SET_VRING_BASE [vq=%p][vq->last_avail_idx=%u][vq->avail_idx=%u][s.index=%u][s.num=%u]",
> > +                     vq, vq->last_avail_idx, vq->avail_idx, s.index, s.num);
> >               break;
> >       case VHOST_GET_VRING_BASE:
> >               s.index = idx;
> >               s.num = vq->last_avail_idx;
> >               if (copy_to_user(argp, &s, sizeof s))
> >                       r = -EFAULT;
> > +             pr_debug(
> > +                     "VHOST_GET_VRING_BASE [vq=%p][vq->last_avail_idx=%u][vq->avail_idx=%u][s.index=%u][s.num=%u]",
> > +                     vq, vq->last_avail_idx, vq->avail_idx, s.index, s.num);
> >               break;
> >       case VHOST_SET_VRING_KICK:
> >               if (copy_from_user(&f, argp, sizeof f)) {
> > @@ -2239,8 +2254,8 @@ static int fetch_buf(struct vhost_virtqueue *vq)
> >               vq->avail_idx = vhost16_to_cpu(vq, avail_idx);
> >
> >               if (unlikely((u16)(vq->avail_idx - last_avail_idx) > vq->num)) {
> > -                     vq_err(vq, "Guest moved used index from %u to %u",
> > -                             last_avail_idx, vq->avail_idx);
> > +                     vq_err(vq, "Guest moved vq %p used index from %u to %u",
> > +                             vq, last_avail_idx, vq->avail_idx);
> >                       return -EFAULT;
> >               }
> >
> > @@ -2316,6 +2331,9 @@ static int fetch_buf(struct vhost_virtqueue *vq)
> >       BUG_ON(!(vq->used_flags & VRING_USED_F_NO_NOTIFY));
> >
> >       /* On success, increment avail index. */
> > +     pr_debug(
> > +             "[vq=%p][vq->last_avail_idx=%u][vq->avail_idx=%u][vq->ndescs=%d][vq->first_desc=%d]",
> > +             vq, vq->last_avail_idx, vq->avail_idx, vq->ndescs, vq->first_desc);
> >       vq->last_avail_idx++;
> >
> >       return 0;
> > @@ -2432,6 +2450,9 @@ EXPORT_SYMBOL_GPL(vhost_get_vq_desc);
> >  /* Reverse the effect of vhost_get_vq_desc. Useful for error handling. */
> >  void vhost_discard_vq_desc(struct vhost_virtqueue *vq, int n)
> >  {
> > +     pr_debug(
> > +             "DISCARD [vq=%p][vq->last_avail_idx=%u][vq->avail_idx=%u][n=%d]",
> > +             vq, vq->last_avail_idx, vq->avail_idx, n);
> >       vq->last_avail_idx -= n;
> >  }
> >  EXPORT_SYMBOL_GPL(vhost_discard_vq_desc);
> > diff --git a/drivers/vhost/vhost.h b/drivers/vhost/vhost.h
> > index 661088ae6dc7..08f6d2ccb697 100644
> > --- a/drivers/vhost/vhost.h
> > +++ b/drivers/vhost/vhost.h
> > @@ -250,11 +250,11 @@ int vhost_init_device_iotlb(struct vhost_dev *d, bool enabled);
> >       } while (0)
> >
> >  enum {
> > -     VHOST_FEATURES = (1ULL << VIRTIO_F_NOTIFY_ON_EMPTY) |
> > -                      (1ULL << VIRTIO_RING_F_INDIRECT_DESC) |
> > -                      (1ULL << VIRTIO_RING_F_EVENT_IDX) |
> > -                      (1ULL << VHOST_F_LOG_ALL) |
> > -                      (1ULL << VIRTIO_F_ANY_LAYOUT) |
> > +     VHOST_FEATURES = /* (1ULL << VIRTIO_F_NOTIFY_ON_EMPTY) | */
> > +                      /* (1ULL << VIRTIO_RING_F_INDIRECT_DESC) | */
> > +                      /* (1ULL << VIRTIO_RING_F_EVENT_IDX) | */
> > +                      /* (1ULL << VHOST_F_LOG_ALL) | */
> > +                      /* (1ULL << VIRTIO_F_ANY_LAYOUT) | */
> >                        (1ULL << VIRTIO_F_VERSION_1)
> >  };
> >
> >
>

Re: vhost changes (batched) in linux-next after 12/13 trigger random crashes in KVM guests after reboot

[Christian Borntraeger]

On 14.02.20 08:40, Eugenio Perez Martin wrote:
> Hi.
> 
> Were the vhost and vhost_net modules loaded with dyndbg='+plt'? I miss
> all the others regular debug traces on that one.

I did 

 echo -n 'file drivers/vhost/vhost.c +plt' > control
and
echo -n 'file drivers/vhost/net.c +plt'  > control

but apparently it did not work...me hates dynamic debug.

> 
> Thanks!
> 
> On Fri, Feb 14, 2020 at 8:34 AM Christian Borntraeger
> <borntraeger@de.ibm.com> wrote:
>>
>> I did
>> ping -c 20 -f ... ; reboot
>> twice
>>
>> The ping after the first reboot showed .......E
>>
>> this was on the host console
>>
>> [   55.951885] CPU: 34 PID: 1908 Comm: CPU 0/KVM Not tainted 5.5.0+ #21
>> [   55.951891] Hardware name: IBM 3906 M04 704 (LPAR)
>> [   55.951892] Call Trace:
>> [   55.951902]  [<0000001ede114132>] show_stack+0x8a/0xd0
>> [   55.951906]  [<0000001edeb0672a>] dump_stack+0x8a/0xb8
>> [   55.951915]  [<000003ff803736a6>] vhost_vring_ioctl+0x6fe/0x858 [vhost]
>> [   55.951919]  [<000003ff8042a608>] vhost_net_ioctl+0x510/0x570 [vhost_net]
>> [   55.951924]  [<0000001ede3c4dd8>] do_vfs_ioctl+0x430/0x6f8
>> [   55.951926]  [<0000001ede3c5124>] ksys_ioctl+0x84/0xb0
>> [   55.951927]  [<0000001ede3c51ba>] __s390x_sys_ioctl+0x2a/0x38
>> [   55.951931]  [<0000001edeb27f72>] system_call+0x2a6/0x2c8
>> [   55.951949] CPU: 34 PID: 1908 Comm: CPU 0/KVM Not tainted 5.5.0+ #21
>> [   55.951950] Hardware name: IBM 3906 M04 704 (LPAR)
>> [   55.951951] Call Trace:
>> [   55.951952]  [<0000001ede114132>] show_stack+0x8a/0xd0
>> [   55.951954]  [<0000001edeb0672a>] dump_stack+0x8a/0xb8
>> [   55.951956]  [<000003ff803736a6>] vhost_vring_ioctl+0x6fe/0x858 [vhost]
>> [   55.951958]  [<000003ff8042a608>] vhost_net_ioctl+0x510/0x570 [vhost_net]
>> [   55.951959]  [<0000001ede3c4dd8>] do_vfs_ioctl+0x430/0x6f8
>> [   55.951961]  [<0000001ede3c5124>] ksys_ioctl+0x84/0xb0
>> [   55.951962]  [<0000001ede3c51ba>] __s390x_sys_ioctl+0x2a/0x38
>> [   55.951964]  [<0000001edeb27f72>] system_call+0x2a6/0x2c8
>> [   55.951997] Guest moved vq 0000000063d896c6 used index from 44 to 0
>> [   56.609831] unexpected descriptor format for RX: out 0, in 0
>> [   86.540460] CPU: 6 PID: 1908 Comm: CPU 0/KVM Not tainted 5.5.0+ #21
>> [   86.540464] Hardware name: IBM 3906 M04 704 (LPAR)
>> [   86.540466] Call Trace:
>> [   86.540473]  [<0000001ede114132>] show_stack+0x8a/0xd0
>> [   86.540477]  [<0000001edeb0672a>] dump_stack+0x8a/0xb8
>> [   86.540486]  [<000003ff803736a6>] vhost_vring_ioctl+0x6fe/0x858 [vhost]
>> [   86.540490]  [<000003ff8042a608>] vhost_net_ioctl+0x510/0x570 [vhost_net]
>> [   86.540494]  [<0000001ede3c4dd8>] do_vfs_ioctl+0x430/0x6f8
>> [   86.540496]  [<0000001ede3c5124>] ksys_ioctl+0x84/0xb0
>> [   86.540498]  [<0000001ede3c51ba>] __s390x_sys_ioctl+0x2a/0x38
>> [   86.540501]  [<0000001edeb27f72>] system_call+0x2a6/0x2c8
>> [   86.540524] CPU: 6 PID: 1908 Comm: CPU 0/KVM Not tainted 5.5.0+ #21
>> [   86.540525] Hardware name: IBM 3906 M04 704 (LPAR)
>> [   86.540526] Call Trace:
>> [   86.540527]  [<0000001ede114132>] show_stack+0x8a/0xd0
>> [   86.540528]  [<0000001edeb0672a>] dump_stack+0x8a/0xb8
>> [   86.540531]  [<000003ff803736a6>] vhost_vring_ioctl+0x6fe/0x858 [vhost]
>> [   86.540532]  [<000003ff8042a608>] vhost_net_ioctl+0x510/0x570 [vhost_net]
>> [   86.540534]  [<0000001ede3c4dd8>] do_vfs_ioctl+0x430/0x6f8
>> [   86.540536]  [<0000001ede3c5124>] ksys_ioctl+0x84/0xb0
>> [   86.540537]  [<0000001ede3c51ba>] __s390x_sys_ioctl+0x2a/0x38
>> [   86.540538]  [<0000001edeb27f72>] system_call+0x2a6/0x2c8
>> [   86.540570] unexpected descriptor format for RX: out 0, in 0
>> [   86.540577] Unexpected header len for TX: 0 expected 0
>>
>>
>> On 14.02.20 08:06, Eugenio Pérez wrote:
>>> Hi Christian.
>>>
>>> Sorry, that was meant to be applied over previous debug patch.
>>>
>>> Here I inline the one meant to be applied over eccb852f1fe6bede630e2e4f1a121a81e34354ab.
>>>
>>> Thanks!
>>>
>>> From d978ace99e4844b49b794d768385db3d128a4cc0 Mon Sep 17 00:00:00 2001
>>> From: =?UTF-8?q?Eugenio=20P=C3=A9rez?= <eperezma@redhat.com>
>>> Date: Fri, 14 Feb 2020 08:02:26 +0100
>>> Subject: [PATCH] vhost: disable all features and trace last_avail_idx and
>>>  ioctl calls
>>>
>>> ---
>>>  drivers/vhost/net.c   | 20 +++++++++++++++++---
>>>  drivers/vhost/vhost.c | 25 +++++++++++++++++++++++--
>>>  drivers/vhost/vhost.h | 10 +++++-----
>>>  3 files changed, 45 insertions(+), 10 deletions(-)
>>>
>>> diff --git a/drivers/vhost/net.c b/drivers/vhost/net.c
>>> index e158159671fa..e4d5f843f9c0 100644
>>> --- a/drivers/vhost/net.c
>>> +++ b/drivers/vhost/net.c
>>> @@ -1505,10 +1505,13 @@ static long vhost_net_set_backend(struct vhost_net *n, unsigned index, int fd)
>>>
>>>       mutex_lock(&n->dev.mutex);
>>>       r = vhost_dev_check_owner(&n->dev);
>>> -     if (r)
>>> +     if (r) {
>>> +             pr_debug("vhost_dev_check_owner index=%u fd=%d rc r=%d", index, fd, r);
>>>               goto err;
>>> +     }
>>>
>>>       if (index >= VHOST_NET_VQ_MAX) {
>>> +             pr_debug("vhost_dev_check_owner index=%u fd=%d MAX=%d", index, fd, VHOST_NET_VQ_MAX);
>>>               r = -ENOBUFS;
>>>               goto err;
>>>       }
>>> @@ -1518,22 +1521,26 @@ static long vhost_net_set_backend(struct vhost_net *n, unsigned index, int fd)
>>>
>>>       /* Verify that ring has been setup correctly. */
>>>       if (!vhost_vq_access_ok(vq)) {
>>> +             pr_debug("vhost_net_set_backend index=%u fd=%d !vhost_vq_access_ok", index, fd);
>>>               r = -EFAULT;
>>>               goto err_vq;
>>>       }
>>>       sock = get_socket(fd);
>>>       if (IS_ERR(sock)) {
>>>               r = PTR_ERR(sock);
>>> +             pr_debug("vhost_net_set_backend index=%u fd=%d get_socket err r=%d", index, fd, r);
>>>               goto err_vq;
>>>       }
>>>
>>>       /* start polling new socket */
>>>       oldsock = vq->private_data;
>>>       if (sock != oldsock) {
>>> +             pr_debug("sock=%p != oldsock=%p index=%u fd=%d vq=%p", sock, oldsock, index, fd, vq);
>>>               ubufs = vhost_net_ubuf_alloc(vq,
>>>                                            sock && vhost_sock_zcopy(sock));
>>>               if (IS_ERR(ubufs)) {
>>>                       r = PTR_ERR(ubufs);
>>> +                     pr_debug("ubufs index=%u fd=%d err r=%d vq=%p", index, fd, r, vq);
>>>                       goto err_ubufs;
>>>               }
>>>
>>> @@ -1541,11 +1548,15 @@ static long vhost_net_set_backend(struct vhost_net *n, unsigned index, int fd)
>>>               vq->private_data = sock;
>>>               vhost_net_buf_unproduce(nvq);
>>>               r = vhost_vq_init_access(vq);
>>> -             if (r)
>>> +             if (r) {
>>> +                     pr_debug("init_access index=%u fd=%d r=%d vq=%p", index, fd, r, vq);
>>>                       goto err_used;
>>> +             }
>>>               r = vhost_net_enable_vq(n, vq);
>>> -             if (r)
>>> +             if (r) {
>>> +                     pr_debug("enable_vq index=%u fd=%d r=%d vq=%p", index, fd, r, vq);
>>>                       goto err_used;
>>> +             }
>>>               if (index == VHOST_NET_VQ_RX)
>>>                       nvq->rx_ring = get_tap_ptr_ring(fd);
>>>
>>> @@ -1559,6 +1570,8 @@ static long vhost_net_set_backend(struct vhost_net *n, unsigned index, int fd)
>>>
>>>       mutex_unlock(&vq->mutex);
>>>
>>> +     pr_debug("sock=%p", sock);
>>> +
>>>       if (oldubufs) {
>>>               vhost_net_ubuf_put_wait_and_free(oldubufs);
>>>               mutex_lock(&vq->mutex);
>>> @@ -1710,6 +1723,7 @@ static long vhost_net_ioctl(struct file *f, unsigned int ioctl,
>>>
>>>       switch (ioctl) {
>>>       case VHOST_NET_SET_BACKEND:
>>> +             pr_debug("VHOST_NET_SET_BACKEND");
>>>               if (copy_from_user(&backend, argp, sizeof backend))
>>>                       return -EFAULT;
>>>               return vhost_net_set_backend(n, backend.index, backend.fd);
>>> diff --git a/drivers/vhost/vhost.c b/drivers/vhost/vhost.c
>>> index b5a51b1f2e79..ec25ba32fe81 100644
>>> --- a/drivers/vhost/vhost.c
>>> +++ b/drivers/vhost/vhost.c
>>> @@ -1642,15 +1642,30 @@ long vhost_vring_ioctl(struct vhost_dev *d, unsigned int ioctl, void __user *arg
>>>                       r = -EINVAL;
>>>                       break;
>>>               }
>>> +
>>> +             if (vq->last_avail_idx || vq->avail_idx) {
>>> +                     pr_debug(
>>> +                             "strange VHOST_SET_VRING_BASE [vq=%p][s.index=%u][s.num=%u]",
>>> +                             vq, s.index, s.num);
>>> +                     dump_stack();
>>> +                     r = 0;
>>> +                     break;
>>> +             }
>>>               vq->last_avail_idx = s.num;
>>>               /* Forget the cached index value. */
>>>               vq->avail_idx = vq->last_avail_idx;
>>> +             pr_debug(
>>> +                     "VHOST_SET_VRING_BASE [vq=%p][vq->last_avail_idx=%u][vq->avail_idx=%u][s.index=%u][s.num=%u]",
>>> +                     vq, vq->last_avail_idx, vq->avail_idx, s.index, s.num);
>>>               break;
>>>       case VHOST_GET_VRING_BASE:
>>>               s.index = idx;
>>>               s.num = vq->last_avail_idx;
>>>               if (copy_to_user(argp, &s, sizeof s))
>>>                       r = -EFAULT;
>>> +             pr_debug(
>>> +                     "VHOST_GET_VRING_BASE [vq=%p][vq->last_avail_idx=%u][vq->avail_idx=%u][s.index=%u][s.num=%u]",
>>> +                     vq, vq->last_avail_idx, vq->avail_idx, s.index, s.num);
>>>               break;
>>>       case VHOST_SET_VRING_KICK:
>>>               if (copy_from_user(&f, argp, sizeof f)) {
>>> @@ -2239,8 +2254,8 @@ static int fetch_buf(struct vhost_virtqueue *vq)
>>>               vq->avail_idx = vhost16_to_cpu(vq, avail_idx);
>>>
>>>               if (unlikely((u16)(vq->avail_idx - last_avail_idx) > vq->num)) {
>>> -                     vq_err(vq, "Guest moved used index from %u to %u",
>>> -                             last_avail_idx, vq->avail_idx);
>>> +                     vq_err(vq, "Guest moved vq %p used index from %u to %u",
>>> +                             vq, last_avail_idx, vq->avail_idx);
>>>                       return -EFAULT;
>>>               }
>>>
>>> @@ -2316,6 +2331,9 @@ static int fetch_buf(struct vhost_virtqueue *vq)
>>>       BUG_ON(!(vq->used_flags & VRING_USED_F_NO_NOTIFY));
>>>
>>>       /* On success, increment avail index. */
>>> +     pr_debug(
>>> +             "[vq=%p][vq->last_avail_idx=%u][vq->avail_idx=%u][vq->ndescs=%d][vq->first_desc=%d]",
>>> +             vq, vq->last_avail_idx, vq->avail_idx, vq->ndescs, vq->first_desc);
>>>       vq->last_avail_idx++;
>>>
>>>       return 0;
>>> @@ -2432,6 +2450,9 @@ EXPORT_SYMBOL_GPL(vhost_get_vq_desc);
>>>  /* Reverse the effect of vhost_get_vq_desc. Useful for error handling. */
>>>  void vhost_discard_vq_desc(struct vhost_virtqueue *vq, int n)
>>>  {
>>> +     pr_debug(
>>> +             "DISCARD [vq=%p][vq->last_avail_idx=%u][vq->avail_idx=%u][n=%d]",
>>> +             vq, vq->last_avail_idx, vq->avail_idx, n);
>>>       vq->last_avail_idx -= n;
>>>  }
>>>  EXPORT_SYMBOL_GPL(vhost_discard_vq_desc);
>>> diff --git a/drivers/vhost/vhost.h b/drivers/vhost/vhost.h
>>> index 661088ae6dc7..08f6d2ccb697 100644
>>> --- a/drivers/vhost/vhost.h
>>> +++ b/drivers/vhost/vhost.h
>>> @@ -250,11 +250,11 @@ int vhost_init_device_iotlb(struct vhost_dev *d, bool enabled);
>>>       } while (0)
>>>
>>>  enum {
>>> -     VHOST_FEATURES = (1ULL << VIRTIO_F_NOTIFY_ON_EMPTY) |
>>> -                      (1ULL << VIRTIO_RING_F_INDIRECT_DESC) |
>>> -                      (1ULL << VIRTIO_RING_F_EVENT_IDX) |
>>> -                      (1ULL << VHOST_F_LOG_ALL) |
>>> -                      (1ULL << VIRTIO_F_ANY_LAYOUT) |
>>> +     VHOST_FEATURES = /* (1ULL << VIRTIO_F_NOTIFY_ON_EMPTY) | */
>>> +                      /* (1ULL << VIRTIO_RING_F_INDIRECT_DESC) | */
>>> +                      /* (1ULL << VIRTIO_RING_F_EVENT_IDX) | */
>>> +                      /* (1ULL << VHOST_F_LOG_ALL) | */
>>> +                      /* (1ULL << VIRTIO_F_ANY_LAYOUT) | */
>>>                        (1ULL << VIRTIO_F_VERSION_1)
>>>  };
>>>
>>>
>>
> 

Re: vhost changes (batched) in linux-next after 12/13 trigger random crashes in KVM guests after reboot

[Christian Borntraeger]

[-- Attachment #1: Type: text/plain, Size: 12556 bytes --]

repro


On 14.02.20 08:43, Christian Borntraeger wrote:
> 
> 
> On 14.02.20 08:40, Eugenio Perez Martin wrote:
>> Hi.
>>
>> Were the vhost and vhost_net modules loaded with dyndbg='+plt'? I miss
>> all the others regular debug traces on that one.
> 
> I did 
> 
>  echo -n 'file drivers/vhost/vhost.c +plt' > control
> and
> echo -n 'file drivers/vhost/net.c +plt'  > control
> 
> but apparently it did not work...me hates dynamic debug.
> 
>>
>> Thanks!
>>
>> On Fri, Feb 14, 2020 at 8:34 AM Christian Borntraeger
>> <borntraeger@de.ibm.com> wrote:
>>>
>>> I did
>>> ping -c 20 -f ... ; reboot
>>> twice
>>>
>>> The ping after the first reboot showed .......E
>>>
>>> this was on the host console
>>>
>>> [   55.951885] CPU: 34 PID: 1908 Comm: CPU 0/KVM Not tainted 5.5.0+ #21
>>> [   55.951891] Hardware name: IBM 3906 M04 704 (LPAR)
>>> [   55.951892] Call Trace:
>>> [   55.951902]  [<0000001ede114132>] show_stack+0x8a/0xd0
>>> [   55.951906]  [<0000001edeb0672a>] dump_stack+0x8a/0xb8
>>> [   55.951915]  [<000003ff803736a6>] vhost_vring_ioctl+0x6fe/0x858 [vhost]
>>> [   55.951919]  [<000003ff8042a608>] vhost_net_ioctl+0x510/0x570 [vhost_net]
>>> [   55.951924]  [<0000001ede3c4dd8>] do_vfs_ioctl+0x430/0x6f8
>>> [   55.951926]  [<0000001ede3c5124>] ksys_ioctl+0x84/0xb0
>>> [   55.951927]  [<0000001ede3c51ba>] __s390x_sys_ioctl+0x2a/0x38
>>> [   55.951931]  [<0000001edeb27f72>] system_call+0x2a6/0x2c8
>>> [   55.951949] CPU: 34 PID: 1908 Comm: CPU 0/KVM Not tainted 5.5.0+ #21
>>> [   55.951950] Hardware name: IBM 3906 M04 704 (LPAR)
>>> [   55.951951] Call Trace:
>>> [   55.951952]  [<0000001ede114132>] show_stack+0x8a/0xd0
>>> [   55.951954]  [<0000001edeb0672a>] dump_stack+0x8a/0xb8
>>> [   55.951956]  [<000003ff803736a6>] vhost_vring_ioctl+0x6fe/0x858 [vhost]
>>> [   55.951958]  [<000003ff8042a608>] vhost_net_ioctl+0x510/0x570 [vhost_net]
>>> [   55.951959]  [<0000001ede3c4dd8>] do_vfs_ioctl+0x430/0x6f8
>>> [   55.951961]  [<0000001ede3c5124>] ksys_ioctl+0x84/0xb0
>>> [   55.951962]  [<0000001ede3c51ba>] __s390x_sys_ioctl+0x2a/0x38
>>> [   55.951964]  [<0000001edeb27f72>] system_call+0x2a6/0x2c8
>>> [   55.951997] Guest moved vq 0000000063d896c6 used index from 44 to 0
>>> [   56.609831] unexpected descriptor format for RX: out 0, in 0
>>> [   86.540460] CPU: 6 PID: 1908 Comm: CPU 0/KVM Not tainted 5.5.0+ #21
>>> [   86.540464] Hardware name: IBM 3906 M04 704 (LPAR)
>>> [   86.540466] Call Trace:
>>> [   86.540473]  [<0000001ede114132>] show_stack+0x8a/0xd0
>>> [   86.540477]  [<0000001edeb0672a>] dump_stack+0x8a/0xb8
>>> [   86.540486]  [<000003ff803736a6>] vhost_vring_ioctl+0x6fe/0x858 [vhost]
>>> [   86.540490]  [<000003ff8042a608>] vhost_net_ioctl+0x510/0x570 [vhost_net]
>>> [   86.540494]  [<0000001ede3c4dd8>] do_vfs_ioctl+0x430/0x6f8
>>> [   86.540496]  [<0000001ede3c5124>] ksys_ioctl+0x84/0xb0
>>> [   86.540498]  [<0000001ede3c51ba>] __s390x_sys_ioctl+0x2a/0x38
>>> [   86.540501]  [<0000001edeb27f72>] system_call+0x2a6/0x2c8
>>> [   86.540524] CPU: 6 PID: 1908 Comm: CPU 0/KVM Not tainted 5.5.0+ #21
>>> [   86.540525] Hardware name: IBM 3906 M04 704 (LPAR)
>>> [   86.540526] Call Trace:
>>> [   86.540527]  [<0000001ede114132>] show_stack+0x8a/0xd0
>>> [   86.540528]  [<0000001edeb0672a>] dump_stack+0x8a/0xb8
>>> [   86.540531]  [<000003ff803736a6>] vhost_vring_ioctl+0x6fe/0x858 [vhost]
>>> [   86.540532]  [<000003ff8042a608>] vhost_net_ioctl+0x510/0x570 [vhost_net]
>>> [   86.540534]  [<0000001ede3c4dd8>] do_vfs_ioctl+0x430/0x6f8
>>> [   86.540536]  [<0000001ede3c5124>] ksys_ioctl+0x84/0xb0
>>> [   86.540537]  [<0000001ede3c51ba>] __s390x_sys_ioctl+0x2a/0x38
>>> [   86.540538]  [<0000001edeb27f72>] system_call+0x2a6/0x2c8
>>> [   86.540570] unexpected descriptor format for RX: out 0, in 0
>>> [   86.540577] Unexpected header len for TX: 0 expected 0
>>>
>>>
>>> On 14.02.20 08:06, Eugenio Pérez wrote:
>>>> Hi Christian.
>>>>
>>>> Sorry, that was meant to be applied over previous debug patch.
>>>>
>>>> Here I inline the one meant to be applied over eccb852f1fe6bede630e2e4f1a121a81e34354ab.
>>>>
>>>> Thanks!
>>>>
>>>> From d978ace99e4844b49b794d768385db3d128a4cc0 Mon Sep 17 00:00:00 2001
>>>> From: =?UTF-8?q?Eugenio=20P=C3=A9rez?= <eperezma@redhat.com>
>>>> Date: Fri, 14 Feb 2020 08:02:26 +0100
>>>> Subject: [PATCH] vhost: disable all features and trace last_avail_idx and
>>>>  ioctl calls
>>>>
>>>> ---
>>>>  drivers/vhost/net.c   | 20 +++++++++++++++++---
>>>>  drivers/vhost/vhost.c | 25 +++++++++++++++++++++++--
>>>>  drivers/vhost/vhost.h | 10 +++++-----
>>>>  3 files changed, 45 insertions(+), 10 deletions(-)
>>>>
>>>> diff --git a/drivers/vhost/net.c b/drivers/vhost/net.c
>>>> index e158159671fa..e4d5f843f9c0 100644
>>>> --- a/drivers/vhost/net.c
>>>> +++ b/drivers/vhost/net.c
>>>> @@ -1505,10 +1505,13 @@ static long vhost_net_set_backend(struct vhost_net *n, unsigned index, int fd)
>>>>
>>>>       mutex_lock(&n->dev.mutex);
>>>>       r = vhost_dev_check_owner(&n->dev);
>>>> -     if (r)
>>>> +     if (r) {
>>>> +             pr_debug("vhost_dev_check_owner index=%u fd=%d rc r=%d", index, fd, r);
>>>>               goto err;
>>>> +     }
>>>>
>>>>       if (index >= VHOST_NET_VQ_MAX) {
>>>> +             pr_debug("vhost_dev_check_owner index=%u fd=%d MAX=%d", index, fd, VHOST_NET_VQ_MAX);
>>>>               r = -ENOBUFS;
>>>>               goto err;
>>>>       }
>>>> @@ -1518,22 +1521,26 @@ static long vhost_net_set_backend(struct vhost_net *n, unsigned index, int fd)
>>>>
>>>>       /* Verify that ring has been setup correctly. */
>>>>       if (!vhost_vq_access_ok(vq)) {
>>>> +             pr_debug("vhost_net_set_backend index=%u fd=%d !vhost_vq_access_ok", index, fd);
>>>>               r = -EFAULT;
>>>>               goto err_vq;
>>>>       }
>>>>       sock = get_socket(fd);
>>>>       if (IS_ERR(sock)) {
>>>>               r = PTR_ERR(sock);
>>>> +             pr_debug("vhost_net_set_backend index=%u fd=%d get_socket err r=%d", index, fd, r);
>>>>               goto err_vq;
>>>>       }
>>>>
>>>>       /* start polling new socket */
>>>>       oldsock = vq->private_data;
>>>>       if (sock != oldsock) {
>>>> +             pr_debug("sock=%p != oldsock=%p index=%u fd=%d vq=%p", sock, oldsock, index, fd, vq);
>>>>               ubufs = vhost_net_ubuf_alloc(vq,
>>>>                                            sock && vhost_sock_zcopy(sock));
>>>>               if (IS_ERR(ubufs)) {
>>>>                       r = PTR_ERR(ubufs);
>>>> +                     pr_debug("ubufs index=%u fd=%d err r=%d vq=%p", index, fd, r, vq);
>>>>                       goto err_ubufs;
>>>>               }
>>>>
>>>> @@ -1541,11 +1548,15 @@ static long vhost_net_set_backend(struct vhost_net *n, unsigned index, int fd)
>>>>               vq->private_data = sock;
>>>>               vhost_net_buf_unproduce(nvq);
>>>>               r = vhost_vq_init_access(vq);
>>>> -             if (r)
>>>> +             if (r) {
>>>> +                     pr_debug("init_access index=%u fd=%d r=%d vq=%p", index, fd, r, vq);
>>>>                       goto err_used;
>>>> +             }
>>>>               r = vhost_net_enable_vq(n, vq);
>>>> -             if (r)
>>>> +             if (r) {
>>>> +                     pr_debug("enable_vq index=%u fd=%d r=%d vq=%p", index, fd, r, vq);
>>>>                       goto err_used;
>>>> +             }
>>>>               if (index == VHOST_NET_VQ_RX)
>>>>                       nvq->rx_ring = get_tap_ptr_ring(fd);
>>>>
>>>> @@ -1559,6 +1570,8 @@ static long vhost_net_set_backend(struct vhost_net *n, unsigned index, int fd)
>>>>
>>>>       mutex_unlock(&vq->mutex);
>>>>
>>>> +     pr_debug("sock=%p", sock);
>>>> +
>>>>       if (oldubufs) {
>>>>               vhost_net_ubuf_put_wait_and_free(oldubufs);
>>>>               mutex_lock(&vq->mutex);
>>>> @@ -1710,6 +1723,7 @@ static long vhost_net_ioctl(struct file *f, unsigned int ioctl,
>>>>
>>>>       switch (ioctl) {
>>>>       case VHOST_NET_SET_BACKEND:
>>>> +             pr_debug("VHOST_NET_SET_BACKEND");
>>>>               if (copy_from_user(&backend, argp, sizeof backend))
>>>>                       return -EFAULT;
>>>>               return vhost_net_set_backend(n, backend.index, backend.fd);
>>>> diff --git a/drivers/vhost/vhost.c b/drivers/vhost/vhost.c
>>>> index b5a51b1f2e79..ec25ba32fe81 100644
>>>> --- a/drivers/vhost/vhost.c
>>>> +++ b/drivers/vhost/vhost.c
>>>> @@ -1642,15 +1642,30 @@ long vhost_vring_ioctl(struct vhost_dev *d, unsigned int ioctl, void __user *arg
>>>>                       r = -EINVAL;
>>>>                       break;
>>>>               }
>>>> +
>>>> +             if (vq->last_avail_idx || vq->avail_idx) {
>>>> +                     pr_debug(
>>>> +                             "strange VHOST_SET_VRING_BASE [vq=%p][s.index=%u][s.num=%u]",
>>>> +                             vq, s.index, s.num);
>>>> +                     dump_stack();
>>>> +                     r = 0;
>>>> +                     break;
>>>> +             }
>>>>               vq->last_avail_idx = s.num;
>>>>               /* Forget the cached index value. */
>>>>               vq->avail_idx = vq->last_avail_idx;
>>>> +             pr_debug(
>>>> +                     "VHOST_SET_VRING_BASE [vq=%p][vq->last_avail_idx=%u][vq->avail_idx=%u][s.index=%u][s.num=%u]",
>>>> +                     vq, vq->last_avail_idx, vq->avail_idx, s.index, s.num);
>>>>               break;
>>>>       case VHOST_GET_VRING_BASE:
>>>>               s.index = idx;
>>>>               s.num = vq->last_avail_idx;
>>>>               if (copy_to_user(argp, &s, sizeof s))
>>>>                       r = -EFAULT;
>>>> +             pr_debug(
>>>> +                     "VHOST_GET_VRING_BASE [vq=%p][vq->last_avail_idx=%u][vq->avail_idx=%u][s.index=%u][s.num=%u]",
>>>> +                     vq, vq->last_avail_idx, vq->avail_idx, s.index, s.num);
>>>>               break;
>>>>       case VHOST_SET_VRING_KICK:
>>>>               if (copy_from_user(&f, argp, sizeof f)) {
>>>> @@ -2239,8 +2254,8 @@ static int fetch_buf(struct vhost_virtqueue *vq)
>>>>               vq->avail_idx = vhost16_to_cpu(vq, avail_idx);
>>>>
>>>>               if (unlikely((u16)(vq->avail_idx - last_avail_idx) > vq->num)) {
>>>> -                     vq_err(vq, "Guest moved used index from %u to %u",
>>>> -                             last_avail_idx, vq->avail_idx);
>>>> +                     vq_err(vq, "Guest moved vq %p used index from %u to %u",
>>>> +                             vq, last_avail_idx, vq->avail_idx);
>>>>                       return -EFAULT;
>>>>               }
>>>>
>>>> @@ -2316,6 +2331,9 @@ static int fetch_buf(struct vhost_virtqueue *vq)
>>>>       BUG_ON(!(vq->used_flags & VRING_USED_F_NO_NOTIFY));
>>>>
>>>>       /* On success, increment avail index. */
>>>> +     pr_debug(
>>>> +             "[vq=%p][vq->last_avail_idx=%u][vq->avail_idx=%u][vq->ndescs=%d][vq->first_desc=%d]",
>>>> +             vq, vq->last_avail_idx, vq->avail_idx, vq->ndescs, vq->first_desc);
>>>>       vq->last_avail_idx++;
>>>>
>>>>       return 0;
>>>> @@ -2432,6 +2450,9 @@ EXPORT_SYMBOL_GPL(vhost_get_vq_desc);
>>>>  /* Reverse the effect of vhost_get_vq_desc. Useful for error handling. */
>>>>  void vhost_discard_vq_desc(struct vhost_virtqueue *vq, int n)
>>>>  {
>>>> +     pr_debug(
>>>> +             "DISCARD [vq=%p][vq->last_avail_idx=%u][vq->avail_idx=%u][n=%d]",
>>>> +             vq, vq->last_avail_idx, vq->avail_idx, n);
>>>>       vq->last_avail_idx -= n;
>>>>  }
>>>>  EXPORT_SYMBOL_GPL(vhost_discard_vq_desc);
>>>> diff --git a/drivers/vhost/vhost.h b/drivers/vhost/vhost.h
>>>> index 661088ae6dc7..08f6d2ccb697 100644
>>>> --- a/drivers/vhost/vhost.h
>>>> +++ b/drivers/vhost/vhost.h
>>>> @@ -250,11 +250,11 @@ int vhost_init_device_iotlb(struct vhost_dev *d, bool enabled);
>>>>       } while (0)
>>>>
>>>>  enum {
>>>> -     VHOST_FEATURES = (1ULL << VIRTIO_F_NOTIFY_ON_EMPTY) |
>>>> -                      (1ULL << VIRTIO_RING_F_INDIRECT_DESC) |
>>>> -                      (1ULL << VIRTIO_RING_F_EVENT_IDX) |
>>>> -                      (1ULL << VHOST_F_LOG_ALL) |
>>>> -                      (1ULL << VIRTIO_F_ANY_LAYOUT) |
>>>> +     VHOST_FEATURES = /* (1ULL << VIRTIO_F_NOTIFY_ON_EMPTY) | */
>>>> +                      /* (1ULL << VIRTIO_RING_F_INDIRECT_DESC) | */
>>>> +                      /* (1ULL << VIRTIO_RING_F_EVENT_IDX) | */
>>>> +                      /* (1ULL << VHOST_F_LOG_ALL) | */
>>>> +                      /* (1ULL << VIRTIO_F_ANY_LAYOUT) | */
>>>>                        (1ULL << VIRTIO_F_VERSION_1)
>>>>  };
>>>>
>>>>
>>>
>>

[-- Attachment #2: log --]
[-- Type: text/plain, Size: 33477 bytes --]

[  924.337135] [1901] 1666: VHOST_GET_VRING_BASE [vq=0000000063d896c6][vq->last_avail_idx=109][vq->avail_idx=0][s.index=1][s.num=109]
[  937.802750] [2237] 1657: VHOST_SET_VRING_BASE [vq=00000000742665e9][vq->last_avail_idx=0][vq->avail_idx=0][s.index=0][s.num=0]
[  937.802757] [2237] 1657: VHOST_SET_VRING_BASE [vq=000000007d36f807][vq->last_avail_idx=0][vq->avail_idx=0][s.index=1][s.num=0]
[  937.802763] [2237] 1726: VHOST_NET_SET_BACKEND
[  937.802766] [2237] 1538: sock=00000000de4d87a3 != oldsock=0000000012e3bc88 index=0 fd=39 vq=00000000742665e9
[  937.802770] [2237] 1573: sock=00000000de4d87a3
[  937.802771] [2237] 1726: VHOST_NET_SET_BACKEND
[  937.802772] [2237] 1538: sock=00000000de4d87a3 != oldsock=0000000012e3bc88 index=1 fd=39 vq=000000007d36f807
[  937.802773] [2237] 1573: sock=00000000de4d87a3
[  937.802775] [2235] 2453: DISCARD [vq=00000000742665e9][vq->last_avail_idx=0][vq->avail_idx=0][n=0]
[  938.475541] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=0][vq->avail_idx=256][vq->ndescs=1][vq->first_desc=0]
[  938.475543] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=1][vq->avail_idx=256][vq->ndescs=2][vq->first_desc=0]
[  938.475544] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=2][vq->avail_idx=256][vq->ndescs=3][vq->first_desc=0]
[  938.475545] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=3][vq->avail_idx=256][vq->ndescs=4][vq->first_desc=0]
[  938.475546] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=4][vq->avail_idx=256][vq->ndescs=5][vq->first_desc=0]
[  938.475547] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=5][vq->avail_idx=256][vq->ndescs=6][vq->first_desc=0]
[  938.475548] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=6][vq->avail_idx=256][vq->ndescs=7][vq->first_desc=0]
[  938.475549] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=7][vq->avail_idx=256][vq->ndescs=8][vq->first_desc=0]
[  938.475550] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=8][vq->avail_idx=256][vq->ndescs=9][vq->first_desc=0]
[  938.475551] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=9][vq->avail_idx=256][vq->ndescs=10][vq->first_desc=0]
[  938.475552] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=10][vq->avail_idx=256][vq->ndescs=11][vq->first_desc=0]
[  938.475553] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=11][vq->avail_idx=256][vq->ndescs=12][vq->first_desc=0]
[  938.475554] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=12][vq->avail_idx=256][vq->ndescs=13][vq->first_desc=0]
[  938.475555] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=13][vq->avail_idx=256][vq->ndescs=14][vq->first_desc=0]
[  938.475556] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=14][vq->avail_idx=256][vq->ndescs=15][vq->first_desc=0]
[  938.475557] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=15][vq->avail_idx=256][vq->ndescs=16][vq->first_desc=0]
[  938.475558] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=16][vq->avail_idx=256][vq->ndescs=17][vq->first_desc=0]
[  938.475559] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=17][vq->avail_idx=256][vq->ndescs=18][vq->first_desc=0]
[  938.475560] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=18][vq->avail_idx=256][vq->ndescs=19][vq->first_desc=0]
[  938.475561] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=19][vq->avail_idx=256][vq->ndescs=20][vq->first_desc=0]
[  938.475562] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=20][vq->avail_idx=256][vq->ndescs=21][vq->first_desc=0]
[  938.475563] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=21][vq->avail_idx=256][vq->ndescs=22][vq->first_desc=0]
[  938.475564] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=22][vq->avail_idx=256][vq->ndescs=23][vq->first_desc=0]
[  938.475564] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=23][vq->avail_idx=256][vq->ndescs=24][vq->first_desc=0]
[  938.475565] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=24][vq->avail_idx=256][vq->ndescs=25][vq->first_desc=0]
[  938.475566] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=25][vq->avail_idx=256][vq->ndescs=26][vq->first_desc=0]
[  938.475567] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=26][vq->avail_idx=256][vq->ndescs=27][vq->first_desc=0]
[  938.475568] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=27][vq->avail_idx=256][vq->ndescs=28][vq->first_desc=0]
[  938.475569] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=28][vq->avail_idx=256][vq->ndescs=29][vq->first_desc=0]
[  938.475570] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=29][vq->avail_idx=256][vq->ndescs=30][vq->first_desc=0]
[  938.475571] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=30][vq->avail_idx=256][vq->ndescs=31][vq->first_desc=0]
[  938.475572] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=31][vq->avail_idx=256][vq->ndescs=32][vq->first_desc=0]
[  938.475573] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=32][vq->avail_idx=256][vq->ndescs=33][vq->first_desc=0]
[  938.475574] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=33][vq->avail_idx=256][vq->ndescs=34][vq->first_desc=0]
[  938.475575] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=34][vq->avail_idx=256][vq->ndescs=35][vq->first_desc=0]
[  938.475576] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=35][vq->avail_idx=256][vq->ndescs=36][vq->first_desc=0]
[  938.475577] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=36][vq->avail_idx=256][vq->ndescs=37][vq->first_desc=0]
[  938.475577] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=37][vq->avail_idx=256][vq->ndescs=38][vq->first_desc=0]
[  938.475578] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=38][vq->avail_idx=256][vq->ndescs=39][vq->first_desc=0]
[  938.475579] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=39][vq->avail_idx=256][vq->ndescs=40][vq->first_desc=0]
[  938.475580] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=40][vq->avail_idx=256][vq->ndescs=41][vq->first_desc=0]
[  938.475581] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=41][vq->avail_idx=256][vq->ndescs=42][vq->first_desc=0]
[  938.475582] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=42][vq->avail_idx=256][vq->ndescs=43][vq->first_desc=0]
[  938.475583] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=43][vq->avail_idx=256][vq->ndescs=44][vq->first_desc=0]
[  938.475584] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=44][vq->avail_idx=256][vq->ndescs=45][vq->first_desc=0]
[  938.475585] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=45][vq->avail_idx=256][vq->ndescs=46][vq->first_desc=0]
[  938.475586] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=46][vq->avail_idx=256][vq->ndescs=47][vq->first_desc=0]
[  938.475587] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=47][vq->avail_idx=256][vq->ndescs=48][vq->first_desc=0]
[  938.475588] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=48][vq->avail_idx=256][vq->ndescs=49][vq->first_desc=0]
[  938.475589] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=49][vq->avail_idx=256][vq->ndescs=50][vq->first_desc=0]
[  938.475589] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=50][vq->avail_idx=256][vq->ndescs=51][vq->first_desc=0]
[  938.475590] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=51][vq->avail_idx=256][vq->ndescs=52][vq->first_desc=0]
[  938.475591] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=52][vq->avail_idx=256][vq->ndescs=53][vq->first_desc=0]
[  938.475592] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=53][vq->avail_idx=256][vq->ndescs=54][vq->first_desc=0]
[  938.475593] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=54][vq->avail_idx=256][vq->ndescs=55][vq->first_desc=0]
[  938.475594] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=55][vq->avail_idx=256][vq->ndescs=56][vq->first_desc=0]
[  938.475595] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=56][vq->avail_idx=256][vq->ndescs=57][vq->first_desc=0]
[  938.475596] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=57][vq->avail_idx=256][vq->ndescs=58][vq->first_desc=0]
[  938.475597] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=58][vq->avail_idx=256][vq->ndescs=59][vq->first_desc=0]
[  938.475598] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=59][vq->avail_idx=256][vq->ndescs=60][vq->first_desc=0]
[  938.475599] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=60][vq->avail_idx=256][vq->ndescs=61][vq->first_desc=0]
[  938.475600] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=61][vq->avail_idx=256][vq->ndescs=62][vq->first_desc=0]
[  938.475601] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=62][vq->avail_idx=256][vq->ndescs=63][vq->first_desc=0]
[  938.475601] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=63][vq->avail_idx=256][vq->ndescs=64][vq->first_desc=0]
[  938.475603] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=64][vq->avail_idx=256][vq->ndescs=65][vq->first_desc=0]
[  938.591320] [2235] 2334: [vq=000000007d36f807][vq->last_avail_idx=0][vq->avail_idx=1][vq->ndescs=1][vq->first_desc=0]
[  938.607139] [2235] 2334: [vq=000000007d36f807][vq->last_avail_idx=1][vq->avail_idx=2][vq->ndescs=1][vq->first_desc=0]
[  938.607538] [2235] 2334: [vq=000000007d36f807][vq->last_avail_idx=2][vq->avail_idx=4][vq->ndescs=1][vq->first_desc=0]
[  938.607540] [2235] 2334: [vq=000000007d36f807][vq->last_avail_idx=3][vq->avail_idx=4][vq->ndescs=2][vq->first_desc=0]
[  938.616010] [2235] 2334: [vq=000000007d36f807][vq->last_avail_idx=4][vq->avail_idx=5][vq->ndescs=1][vq->first_desc=0]
[  938.796034] [2235] 2334: [vq=000000007d36f807][vq->last_avail_idx=5][vq->avail_idx=6][vq->ndescs=1][vq->first_desc=0]
[  938.811243] [2235] 2334: [vq=000000007d36f807][vq->last_avail_idx=6][vq->avail_idx=7][vq->ndescs=2][vq->first_desc=0]
[  938.905985] [2235] 2334: [vq=000000007d36f807][vq->last_avail_idx=7][vq->avail_idx=8][vq->ndescs=1][vq->first_desc=0]
[  939.011405] [2235] 2334: [vq=000000007d36f807][vq->last_avail_idx=8][vq->avail_idx=9][vq->ndescs=2][vq->first_desc=0]
[  939.212298] [2235] 2334: [vq=000000007d36f807][vq->last_avail_idx=9][vq->avail_idx=10][vq->ndescs=2][vq->first_desc=0]
[  939.413649] [2235] 2334: [vq=000000007d36f807][vq->last_avail_idx=10][vq->avail_idx=11][vq->ndescs=2][vq->first_desc=0]
[  939.596036] [2235] 2334: [vq=000000007d36f807][vq->last_avail_idx=11][vq->avail_idx=12][vq->ndescs=1][vq->first_desc=0]
[  939.996031] [2235] 2334: [vq=000000007d36f807][vq->last_avail_idx=12][vq->avail_idx=13][vq->ndescs=1][vq->first_desc=0]
[  940.591371] [2235] 2334: [vq=000000007d36f807][vq->last_avail_idx=13][vq->avail_idx=14][vq->ndescs=1][vq->first_desc=0]
[  940.636040] [2235] 2334: [vq=000000007d36f807][vq->last_avail_idx=14][vq->avail_idx=15][vq->ndescs=1][vq->first_desc=0]
[  940.665984] [2235] 2334: [vq=000000007d36f807][vq->last_avail_idx=15][vq->avail_idx=16][vq->ndescs=1][vq->first_desc=0]
[  940.865048] [2235] 2334: [vq=000000007d36f807][vq->last_avail_idx=16][vq->avail_idx=17][vq->ndescs=2][vq->first_desc=0]
[  941.067594] [2235] 2334: [vq=000000007d36f807][vq->last_avail_idx=17][vq->avail_idx=18][vq->ndescs=2][vq->first_desc=0]
[  941.126013] [2235] 2334: [vq=000000007d36f807][vq->last_avail_idx=18][vq->avail_idx=19][vq->ndescs=1][vq->first_desc=0]
[  941.273297] [2235] 2334: [vq=000000007d36f807][vq->last_avail_idx=19][vq->avail_idx=20][vq->ndescs=2][vq->first_desc=0]
[  941.294416] [2235] 2334: [vq=000000007d36f807][vq->last_avail_idx=20][vq->avail_idx=21][vq->ndescs=1][vq->first_desc=0]
[  941.476486] [2235] 2334: [vq=000000007d36f807][vq->last_avail_idx=21][vq->avail_idx=22][vq->ndescs=2][vq->first_desc=0]
[  941.516034] [2235] 2334: [vq=000000007d36f807][vq->last_avail_idx=22][vq->avail_idx=23][vq->ndescs=1][vq->first_desc=0]
[  941.651361] [2235] 2334: [vq=000000007d36f807][vq->last_avail_idx=23][vq->avail_idx=24][vq->ndescs=1][vq->first_desc=0]
[  941.651942] [2235] 2334: [vq=000000007d36f807][vq->last_avail_idx=24][vq->avail_idx=25][vq->ndescs=1][vq->first_desc=0]
[  941.652190] [2235] 2334: [vq=000000007d36f807][vq->last_avail_idx=25][vq->avail_idx=26][vq->ndescs=1][vq->first_desc=0]
[  941.652363] [2235] 2334: [vq=000000007d36f807][vq->last_avail_idx=26][vq->avail_idx=27][vq->ndescs=1][vq->first_desc=0]
[  941.652542] [2235] 2334: [vq=000000007d36f807][vq->last_avail_idx=27][vq->avail_idx=28][vq->ndescs=1][vq->first_desc=0]
[  941.652715] [2235] 2334: [vq=000000007d36f807][vq->last_avail_idx=28][vq->avail_idx=29][vq->ndescs=1][vq->first_desc=0]
[  941.652882] [2235] 2334: [vq=000000007d36f807][vq->last_avail_idx=29][vq->avail_idx=30][vq->ndescs=1][vq->first_desc=0]
[  941.653050] [2235] 2334: [vq=000000007d36f807][vq->last_avail_idx=30][vq->avail_idx=31][vq->ndescs=1][vq->first_desc=0]
[  941.653218] [2235] 2334: [vq=000000007d36f807][vq->last_avail_idx=31][vq->avail_idx=32][vq->ndescs=1][vq->first_desc=0]
[  941.653388] [2235] 2334: [vq=000000007d36f807][vq->last_avail_idx=32][vq->avail_idx=33][vq->ndescs=1][vq->first_desc=0]
[  941.653558] [2235] 2334: [vq=000000007d36f807][vq->last_avail_idx=33][vq->avail_idx=34][vq->ndescs=1][vq->first_desc=0]
[  941.653725] [2235] 2334: [vq=000000007d36f807][vq->last_avail_idx=34][vq->avail_idx=35][vq->ndescs=1][vq->first_desc=0]
[  941.653896] [2235] 2334: [vq=000000007d36f807][vq->last_avail_idx=35][vq->avail_idx=36][vq->ndescs=1][vq->first_desc=0]
[  941.654063] [2235] 2334: [vq=000000007d36f807][vq->last_avail_idx=36][vq->avail_idx=37][vq->ndescs=1][vq->first_desc=0]
[  941.654229] [2235] 2334: [vq=000000007d36f807][vq->last_avail_idx=37][vq->avail_idx=38][vq->ndescs=1][vq->first_desc=0]
[  941.654398] [2235] 2334: [vq=000000007d36f807][vq->last_avail_idx=38][vq->avail_idx=39][vq->ndescs=1][vq->first_desc=0]
[  941.654567] [2235] 2334: [vq=000000007d36f807][vq->last_avail_idx=39][vq->avail_idx=40][vq->ndescs=1][vq->first_desc=0]
[  941.654738] [2235] 2334: [vq=000000007d36f807][vq->last_avail_idx=40][vq->avail_idx=41][vq->ndescs=1][vq->first_desc=0]
[  941.654906] [2235] 2334: [vq=000000007d36f807][vq->last_avail_idx=41][vq->avail_idx=42][vq->ndescs=1][vq->first_desc=0]
[  941.655070] [2235] 2334: [vq=000000007d36f807][vq->last_avail_idx=42][vq->avail_idx=43][vq->ndescs=1][vq->first_desc=0]
[  941.655240] [2235] 2334: [vq=000000007d36f807][vq->last_avail_idx=43][vq->avail_idx=44][vq->ndescs=1][vq->first_desc=0]
[  943.627121] [2235] 2334: [vq=000000007d36f807][vq->last_avail_idx=44][vq->avail_idx=45][vq->ndescs=1][vq->first_desc=0]
[  945.158600] [2228] 1726: VHOST_NET_SET_BACKEND
[  945.158603] [2228] 1538: sock=0000000012e3bc88 != oldsock=00000000de4d87a3 index=0 fd=-1 vq=00000000742665e9
[  945.158604] [2228] 1573: sock=0000000012e3bc88
[  945.158622] [2228] 1726: VHOST_NET_SET_BACKEND
[  945.158623] [2228] 1538: sock=0000000012e3bc88 != oldsock=00000000de4d87a3 index=1 fd=-1 vq=000000007d36f807
[  945.158624] [2228] 1573: sock=0000000012e3bc88
[  945.158646] [2228] 1666: VHOST_GET_VRING_BASE [vq=00000000742665e9][vq->last_avail_idx=65][vq->avail_idx=256][s.index=0][s.num=65]
[  945.158648] [2228] 1666: VHOST_GET_VRING_BASE [vq=000000007d36f807][vq->last_avail_idx=45][vq->avail_idx=45][s.index=1][s.num=45]
[  947.678597] [2237] 1647: strange VHOST_SET_VRING_BASE [vq=00000000742665e9][s.index=0][s.num=0]
[  947.678600] CPU: 22 PID: 2237 Comm: CPU 0/KVM Not tainted 5.5.0+ #21
[  947.678604] Hardware name: IBM 3906 M04 704 (LPAR)
[  947.678606] Call Trace:
[  947.678612]  [<0000001ede114132>] show_stack+0x8a/0xd0 
[  947.678615]  [<0000001edeb0672a>] dump_stack+0x8a/0xb8 
[  947.678625]  [<000003ff803736a6>] vhost_vring_ioctl+0x6fe/0x858 [vhost] 
[  947.678628]  [<000003ff8042a608>] vhost_net_ioctl+0x510/0x570 [vhost_net] 
[  947.678632]  [<0000001ede3c4dd8>] do_vfs_ioctl+0x430/0x6f8 
[  947.678634]  [<0000001ede3c5124>] ksys_ioctl+0x84/0xb0 
[  947.678635]  [<0000001ede3c51ba>] __s390x_sys_ioctl+0x2a/0x38 
[  947.678638]  [<0000001edeb27f72>] system_call+0x2a6/0x2c8 
[  947.678660] [2237] 1647: strange VHOST_SET_VRING_BASE [vq=000000007d36f807][s.index=1][s.num=0]
[  947.678661] CPU: 22 PID: 2237 Comm: CPU 0/KVM Not tainted 5.5.0+ #21
[  947.678662] Hardware name: IBM 3906 M04 704 (LPAR)
[  947.678662] Call Trace:
[  947.678664]  [<0000001ede114132>] show_stack+0x8a/0xd0 
[  947.678665]  [<0000001edeb0672a>] dump_stack+0x8a/0xb8 
[  947.678667]  [<000003ff803736a6>] vhost_vring_ioctl+0x6fe/0x858 [vhost] 
[  947.678669]  [<000003ff8042a608>] vhost_net_ioctl+0x510/0x570 [vhost_net] 
[  947.678671]  [<0000001ede3c4dd8>] do_vfs_ioctl+0x430/0x6f8 
[  947.678672]  [<0000001ede3c5124>] ksys_ioctl+0x84/0xb0 
[  947.678673]  [<0000001ede3c51ba>] __s390x_sys_ioctl+0x2a/0x38 
[  947.678675]  [<0000001edeb27f72>] system_call+0x2a6/0x2c8 
[  947.678694] [2237] 1726: VHOST_NET_SET_BACKEND
[  947.678697] [2237] 1538: sock=00000000de4d87a3 != oldsock=0000000012e3bc88 index=0 fd=39 vq=00000000742665e9
[  947.678702] [2237] 1573: sock=00000000de4d87a3
[  947.678703] [2237] 1726: VHOST_NET_SET_BACKEND
[  947.678704] [2237] 1538: sock=00000000de4d87a3 != oldsock=0000000012e3bc88 index=1 fd=39 vq=000000007d36f807
[  947.678705] [2237] 1573: sock=00000000de4d87a3
[  947.678717] Guest moved vq 000000007d36f807 used index from 45 to 0
[  950.562666] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=65][vq->avail_idx=256][vq->ndescs=1][vq->first_desc=0]
[  950.562671] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=66][vq->avail_idx=256][vq->ndescs=2][vq->first_desc=0]
[  950.562672] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=67][vq->avail_idx=256][vq->ndescs=3][vq->first_desc=0]
[  950.562673] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=68][vq->avail_idx=256][vq->ndescs=4][vq->first_desc=0]
[  950.562674] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=69][vq->avail_idx=256][vq->ndescs=5][vq->first_desc=0]
[  950.562675] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=70][vq->avail_idx=256][vq->ndescs=6][vq->first_desc=0]
[  950.562676] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=71][vq->avail_idx=256][vq->ndescs=7][vq->first_desc=0]
[  950.562677] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=72][vq->avail_idx=256][vq->ndescs=8][vq->first_desc=0]
[  950.562678] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=73][vq->avail_idx=256][vq->ndescs=9][vq->first_desc=0]
[  950.562679] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=74][vq->avail_idx=256][vq->ndescs=10][vq->first_desc=0]
[  950.562679] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=75][vq->avail_idx=256][vq->ndescs=11][vq->first_desc=0]
[  950.562680] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=76][vq->avail_idx=256][vq->ndescs=12][vq->first_desc=0]
[  950.562681] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=77][vq->avail_idx=256][vq->ndescs=13][vq->first_desc=0]
[  950.562682] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=78][vq->avail_idx=256][vq->ndescs=14][vq->first_desc=0]
[  950.562683] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=79][vq->avail_idx=256][vq->ndescs=15][vq->first_desc=0]
[  950.562684] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=80][vq->avail_idx=256][vq->ndescs=16][vq->first_desc=0]
[  950.562685] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=81][vq->avail_idx=256][vq->ndescs=17][vq->first_desc=0]
[  950.562686] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=82][vq->avail_idx=256][vq->ndescs=18][vq->first_desc=0]
[  950.562687] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=83][vq->avail_idx=256][vq->ndescs=19][vq->first_desc=0]
[  950.562688] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=84][vq->avail_idx=256][vq->ndescs=20][vq->first_desc=0]
[  950.562689] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=85][vq->avail_idx=256][vq->ndescs=21][vq->first_desc=0]
[  950.562690] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=86][vq->avail_idx=256][vq->ndescs=22][vq->first_desc=0]
[  950.562690] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=87][vq->avail_idx=256][vq->ndescs=23][vq->first_desc=0]
[  950.562691] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=88][vq->avail_idx=256][vq->ndescs=24][vq->first_desc=0]
[  950.562692] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=89][vq->avail_idx=256][vq->ndescs=25][vq->first_desc=0]
[  950.562693] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=90][vq->avail_idx=256][vq->ndescs=26][vq->first_desc=0]
[  950.562694] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=91][vq->avail_idx=256][vq->ndescs=27][vq->first_desc=0]
[  950.562695] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=92][vq->avail_idx=256][vq->ndescs=28][vq->first_desc=0]
[  950.562696] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=93][vq->avail_idx=256][vq->ndescs=29][vq->first_desc=0]
[  950.562697] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=94][vq->avail_idx=256][vq->ndescs=30][vq->first_desc=0]
[  950.562698] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=95][vq->avail_idx=256][vq->ndescs=31][vq->first_desc=0]
[  950.562699] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=96][vq->avail_idx=256][vq->ndescs=32][vq->first_desc=0]
[  950.562700] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=97][vq->avail_idx=256][vq->ndescs=33][vq->first_desc=0]
[  950.562700] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=98][vq->avail_idx=256][vq->ndescs=34][vq->first_desc=0]
[  950.562701] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=99][vq->avail_idx=256][vq->ndescs=35][vq->first_desc=0]
[  950.562702] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=100][vq->avail_idx=256][vq->ndescs=36][vq->first_desc=0]
[  950.562703] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=101][vq->avail_idx=256][vq->ndescs=37][vq->first_desc=0]
[  950.562704] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=102][vq->avail_idx=256][vq->ndescs=38][vq->first_desc=0]
[  950.562705] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=103][vq->avail_idx=256][vq->ndescs=39][vq->first_desc=0]
[  950.562706] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=104][vq->avail_idx=256][vq->ndescs=40][vq->first_desc=0]
[  950.562707] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=105][vq->avail_idx=256][vq->ndescs=41][vq->first_desc=0]
[  950.562707] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=106][vq->avail_idx=256][vq->ndescs=42][vq->first_desc=0]
[  950.562708] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=107][vq->avail_idx=256][vq->ndescs=43][vq->first_desc=0]
[  950.562709] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=108][vq->avail_idx=256][vq->ndescs=44][vq->first_desc=0]
[  950.562710] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=109][vq->avail_idx=256][vq->ndescs=45][vq->first_desc=0]
[  950.562711] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=110][vq->avail_idx=256][vq->ndescs=46][vq->first_desc=0]
[  950.562712] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=111][vq->avail_idx=256][vq->ndescs=47][vq->first_desc=0]
[  950.562713] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=112][vq->avail_idx=256][vq->ndescs=48][vq->first_desc=0]
[  950.562714] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=113][vq->avail_idx=256][vq->ndescs=49][vq->first_desc=0]
[  950.562715] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=114][vq->avail_idx=256][vq->ndescs=50][vq->first_desc=0]
[  950.562715] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=115][vq->avail_idx=256][vq->ndescs=51][vq->first_desc=0]
[  950.562716] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=116][vq->avail_idx=256][vq->ndescs=52][vq->first_desc=0]
[  950.562717] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=117][vq->avail_idx=256][vq->ndescs=53][vq->first_desc=0]
[  950.562718] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=118][vq->avail_idx=256][vq->ndescs=54][vq->first_desc=0]
[  950.562719] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=119][vq->avail_idx=256][vq->ndescs=55][vq->first_desc=0]
[  950.562720] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=120][vq->avail_idx=256][vq->ndescs=56][vq->first_desc=0]
[  950.562721] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=121][vq->avail_idx=256][vq->ndescs=57][vq->first_desc=0]
[  950.562722] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=122][vq->avail_idx=256][vq->ndescs=58][vq->first_desc=0]
[  950.562723] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=123][vq->avail_idx=256][vq->ndescs=59][vq->first_desc=0]
[  950.562724] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=124][vq->avail_idx=256][vq->ndescs=60][vq->first_desc=0]
[  950.562725] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=125][vq->avail_idx=256][vq->ndescs=61][vq->first_desc=0]
[  950.562726] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=126][vq->avail_idx=256][vq->ndescs=62][vq->first_desc=0]
[  950.562727] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=127][vq->avail_idx=256][vq->ndescs=63][vq->first_desc=0]
[  950.562728] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=128][vq->avail_idx=256][vq->ndescs=64][vq->first_desc=0]
[  950.562730] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=129][vq->avail_idx=256][vq->ndescs=65][vq->first_desc=0]
[  968.267262] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=130][vq->avail_idx=256][vq->ndescs=1][vq->first_desc=0]
[  968.267264] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=131][vq->avail_idx=256][vq->ndescs=2][vq->first_desc=0]
[  968.267265] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=132][vq->avail_idx=256][vq->ndescs=3][vq->first_desc=0]
[  968.267266] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=133][vq->avail_idx=256][vq->ndescs=4][vq->first_desc=0]
[  968.267267] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=134][vq->avail_idx=256][vq->ndescs=5][vq->first_desc=0]
[  968.267268] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=135][vq->avail_idx=256][vq->ndescs=6][vq->first_desc=0]
[  968.267269] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=136][vq->avail_idx=256][vq->ndescs=7][vq->first_desc=0]
[  968.267270] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=137][vq->avail_idx=256][vq->ndescs=8][vq->first_desc=0]
[  968.267271] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=138][vq->avail_idx=256][vq->ndescs=9][vq->first_desc=0]
[  968.267271] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=139][vq->avail_idx=256][vq->ndescs=10][vq->first_desc=0]
[  968.267272] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=140][vq->avail_idx=256][vq->ndescs=11][vq->first_desc=0]
[  968.267273] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=141][vq->avail_idx=256][vq->ndescs=12][vq->first_desc=0]
[  968.267274] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=142][vq->avail_idx=256][vq->ndescs=13][vq->first_desc=0]
[  968.267275] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=143][vq->avail_idx=256][vq->ndescs=14][vq->first_desc=0]
[  968.267276] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=144][vq->avail_idx=256][vq->ndescs=15][vq->first_desc=0]
[  968.267277] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=145][vq->avail_idx=256][vq->ndescs=16][vq->first_desc=0]
[  968.267278] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=146][vq->avail_idx=256][vq->ndescs=17][vq->first_desc=0]
[  968.267278] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=147][vq->avail_idx=256][vq->ndescs=18][vq->first_desc=0]
[  968.267279] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=148][vq->avail_idx=256][vq->ndescs=19][vq->first_desc=0]
[  968.267280] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=149][vq->avail_idx=256][vq->ndescs=20][vq->first_desc=0]
[  968.267281] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=150][vq->avail_idx=256][vq->ndescs=21][vq->first_desc=0]
[  968.267282] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=151][vq->avail_idx=256][vq->ndescs=22][vq->first_desc=0]
[  968.267283] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=152][vq->avail_idx=256][vq->ndescs=23][vq->first_desc=0]
[  968.267284] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=153][vq->avail_idx=256][vq->ndescs=24][vq->first_desc=0]
[  968.267284] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=154][vq->avail_idx=256][vq->ndescs=25][vq->first_desc=0]
[  968.267285] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=155][vq->avail_idx=256][vq->ndescs=26][vq->first_desc=0]
[  968.267286] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=156][vq->avail_idx=256][vq->ndescs=27][vq->first_desc=0]
[  968.267287] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=157][vq->avail_idx=256][vq->ndescs=28][vq->first_desc=0]
[  968.267288] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=158][vq->avail_idx=256][vq->ndescs=29][vq->first_desc=0]
[  968.267289] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=159][vq->avail_idx=256][vq->ndescs=30][vq->first_desc=0]
[  968.267290] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=160][vq->avail_idx=256][vq->ndescs=31][vq->first_desc=0]
[  968.267291] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=161][vq->avail_idx=256][vq->ndescs=32][vq->first_desc=0]
[  968.267292] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=162][vq->avail_idx=256][vq->ndescs=33][vq->first_desc=0]
[  968.267292] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=163][vq->avail_idx=256][vq->ndescs=34][vq->first_desc=0]
[  968.267293] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=164][vq->avail_idx=256][vq->ndescs=35][vq->first_desc=0]
[  968.267294] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=165][vq->avail_idx=256][vq->ndescs=36][vq->first_desc=0]
[  968.267295] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=166][vq->avail_idx=256][vq->ndescs=37][vq->first_desc=0]
[  968.267296] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=167][vq->avail_idx=256][vq->ndescs=38][vq->first_desc=0]
[  968.267297] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=168][vq->avail_idx=256][vq->ndescs=39][vq->first_desc=0]
[  968.267298] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=169][vq->avail_idx=256][vq->ndescs=40][vq->first_desc=0]
[  968.267298] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=170][vq->avail_idx=256][vq->ndescs=41][vq->first_desc=0]
[  968.267299] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=171][vq->avail_idx=256][vq->ndescs=42][vq->first_desc=0]
[  968.267300] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=172][vq->avail_idx=256][vq->ndescs=43][vq->first_desc=0]
[  968.267301] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=173][vq->avail_idx=256][vq->ndescs=44][vq->first_desc=0]
[  968.267302] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=174][vq->avail_idx=256][vq->ndescs=45][vq->first_desc=0]
[  968.267303] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=175][vq->avail_idx=256][vq->ndescs=46][vq->first_desc=0]
[  968.267304] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=176][vq->avail_idx=256][vq->ndescs=47][vq->first_desc=0]
[  968.267304] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=177][vq->avail_idx=256][vq->ndescs=48][vq->first_desc=0]
[  968.267305] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=178][vq->avail_idx=256][vq->ndescs=49][vq->first_desc=0]
[  968.267306] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=179][vq->avail_idx=256][vq->ndescs=50][vq->first_desc=0]
[  968.267307] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=180][vq->avail_idx=256][vq->ndescs=51][vq->first_desc=0]
[  968.267308] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=181][vq->avail_idx=256][vq->ndescs=52][vq->first_desc=0]
[  968.267309] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=182][vq->avail_idx=256][vq->ndescs=53][vq->first_desc=0]
[  968.267310] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=183][vq->avail_idx=256][vq->ndescs=54][vq->first_desc=0]
[  968.267310] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=184][vq->avail_idx=256][vq->ndescs=55][vq->first_desc=0]
[  968.267311] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=185][vq->avail_idx=256][vq->ndescs=56][vq->first_desc=0]
[  968.267312] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=186][vq->avail_idx=256][vq->ndescs=57][vq->first_desc=0]
[  968.267313] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=187][vq->avail_idx=256][vq->ndescs=58][vq->first_desc=0]
[  968.267314] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=188][vq->avail_idx=256][vq->ndescs=59][vq->first_desc=0]
[  968.267315] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=189][vq->avail_idx=256][vq->ndescs=60][vq->first_desc=0]
[  968.267316] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=190][vq->avail_idx=256][vq->ndescs=61][vq->first_desc=0]
[  968.267317] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=191][vq->avail_idx=256][vq->ndescs=62][vq->first_desc=0]
[  968.267318] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=192][vq->avail_idx=256][vq->ndescs=63][vq->first_desc=0]
[  968.267319] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=193][vq->avail_idx=256][vq->ndescs=64][vq->first_desc=0]

Re: vhost changes (batched) in linux-next after 12/13 trigger random crashes in KVM guests after reboot

[Eugenio Pérez]

On Fri, 2020-02-14 at 08:47 +0100, Christian Borntraeger wrote:
> repro
> 
> 
> On 14.02.20 08:43, Christian Borntraeger wrote:
> > 
> > On 14.02.20 08:40, Eugenio Perez Martin wrote:
> > > Hi.
> > > 
> > > Were the vhost and vhost_net modules loaded with dyndbg='+plt'? I miss
> > > all the others regular debug traces on that one.
> > 
> > I did 
> > 
> >  echo -n 'file drivers/vhost/vhost.c +plt' > control
> > and
> > echo -n 'file drivers/vhost/net.c +plt'  > control
> > 
> > but apparently it did not work...me hates dynamic debug.

Sorry about use dyndbg, but it is the easiest way to obtain accurate line numbers, threads id...

I usually load the module with that option (modprobe vhost dyndbg=+pfmlt && modprobe vhost_net dyndbg=+pfmlt), so it is
available from the very beginning.

> > 
> > > Thanks!
> > > 
> > > On Fri, Feb 14, 2020 at 8:34 AM Christian Borntraeger
> > > <borntraeger@de.ibm.com> wrote:
> > > > I did
> > > > ping -c 20 -f ... ; reboot
> > > > twice
> > > > 
> > > > The ping after the first reboot showed .......E
> > > > 
> > > > this was on the host console
> > > > 
> > > > [   55.951885] CPU: 34 PID: 1908 Comm: CPU 0/KVM Not tainted 5.5.0+ #21
> > > > [   55.951891] Hardware name: IBM 3906 M04 704 (LPAR)
> > > > [   55.951892] Call Trace:
> > > > [   55.951902]  [<0000001ede114132>] show_stack+0x8a/0xd0
> > > > [   55.951906]  [<0000001edeb0672a>] dump_stack+0x8a/0xb8
> > > > [   55.951915]  [<000003ff803736a6>] vhost_vring_ioctl+0x6fe/0x858 [vhost]
> > > > [   55.951919]  [<000003ff8042a608>] vhost_net_ioctl+0x510/0x570 [vhost_net]
> > > > [   55.951924]  [<0000001ede3c4dd8>] do_vfs_ioctl+0x430/0x6f8
> > > > [   55.951926]  [<0000001ede3c5124>] ksys_ioctl+0x84/0xb0
> > > > [   55.951927]  [<0000001ede3c51ba>] __s390x_sys_ioctl+0x2a/0x38
> > > > [   55.951931]  [<0000001edeb27f72>] system_call+0x2a6/0x2c8
> > > > [   55.951949] CPU: 34 PID: 1908 Comm: CPU 0/KVM Not tainted 5.5.0+ #21
> > > > [   55.951950] Hardware name: IBM 3906 M04 704 (LPAR)
> > > > [   55.951951] Call Trace:
> > > > [   55.951952]  [<0000001ede114132>] show_stack+0x8a/0xd0
> > > > [   55.951954]  [<0000001edeb0672a>] dump_stack+0x8a/0xb8
> > > > [   55.951956]  [<000003ff803736a6>] vhost_vring_ioctl+0x6fe/0x858 [vhost]
> > > > [   55.951958]  [<000003ff8042a608>] vhost_net_ioctl+0x510/0x570 [vhost_net]
> > > > [   55.951959]  [<0000001ede3c4dd8>] do_vfs_ioctl+0x430/0x6f8
> > > > [   55.951961]  [<0000001ede3c5124>] ksys_ioctl+0x84/0xb0
> > > > [   55.951962]  [<0000001ede3c51ba>] __s390x_sys_ioctl+0x2a/0x38
> > > > [   55.951964]  [<0000001edeb27f72>] system_call+0x2a6/0x2c8
> > > > [   55.951997] Guest moved vq 0000000063d896c6 used index from 44 to 0
> > > > [   56.609831] unexpected descriptor format for RX: out 0, in 0
> > > > [   86.540460] CPU: 6 PID: 1908 Comm: CPU 0/KVM Not tainted 5.5.0+ #21
> > > > [   86.540464] Hardware name: IBM 3906 M04 704 (LPAR)
> > > > [   86.540466] Call Trace:
> > > > [   86.540473]  [<0000001ede114132>] show_stack+0x8a/0xd0
> > > > [   86.540477]  [<0000001edeb0672a>] dump_stack+0x8a/0xb8
> > > > [   86.540486]  [<000003ff803736a6>] vhost_vring_ioctl+0x6fe/0x858 [vhost]
> > > > [   86.540490]  [<000003ff8042a608>] vhost_net_ioctl+0x510/0x570 [vhost_net]
> > > > [   86.540494]  [<0000001ede3c4dd8>] do_vfs_ioctl+0x430/0x6f8
> > > > [   86.540496]  [<0000001ede3c5124>] ksys_ioctl+0x84/0xb0
> > > > [   86.540498]  [<0000001ede3c51ba>] __s390x_sys_ioctl+0x2a/0x38
> > > > [   86.540501]  [<0000001edeb27f72>] system_call+0x2a6/0x2c8
> > > > [   86.540524] CPU: 6 PID: 1908 Comm: CPU 0/KVM Not tainted 5.5.0+ #21
> > > > [   86.540525] Hardware name: IBM 3906 M04 704 (LPAR)
> > > > [   86.540526] Call Trace:
> > > > [   86.540527]  [<0000001ede114132>] show_stack+0x8a/0xd0
> > > > [   86.540528]  [<0000001edeb0672a>] dump_stack+0x8a/0xb8
> > > > [   86.540531]  [<000003ff803736a6>] vhost_vring_ioctl+0x6fe/0x858 [vhost]
> > > > [   86.540532]  [<000003ff8042a608>] vhost_net_ioctl+0x510/0x570 [vhost_net]
> > > > [   86.540534]  [<0000001ede3c4dd8>] do_vfs_ioctl+0x430/0x6f8
> > > > [   86.540536]  [<0000001ede3c5124>] ksys_ioctl+0x84/0xb0
> > > > [   86.540537]  [<0000001ede3c51ba>] __s390x_sys_ioctl+0x2a/0x38
> > > > [   86.540538]  [<0000001edeb27f72>] system_call+0x2a6/0x2c8
> > > > [   86.540570] unexpected descriptor format for RX: out 0, in 0
> > > > [   86.540577] Unexpected header len for TX: 0 expected 0
> > > > 
> > > > 
> > > > On 14.02.20 08:06, Eugenio Pérez wrote:
> > > > > Hi Christian.
> > > > > 
> > > > > Sorry, that was meant to be applied over previous debug patch.
> > > > > 
> > > > > Here I inline the one meant to be applied over eccb852f1fe6bede630e2e4f1a121a81e34354ab.
> > > > > 
> > > > > Thanks!
> > > > > 
> > > > > From d978ace99e4844b49b794d768385db3d128a4cc0 Mon Sep 17 00:00:00 2001
> > > > > From: =?UTF-8?q?Eugenio=20P=C3=A9rez?= <eperezma@redhat.com>
> > > > > Date: Fri, 14 Feb 2020 08:02:26 +0100
> > > > > Subject: [PATCH] vhost: disable all features and trace last_avail_idx and
> > > > >  ioctl calls
> > > > > 
> > > > > ---
> > > > >  drivers/vhost/net.c   | 20 +++++++++++++++++---
> > > > >  drivers/vhost/vhost.c | 25 +++++++++++++++++++++++--
> > > > >  drivers/vhost/vhost.h | 10 +++++-----
> > > > >  3 files changed, 45 insertions(+), 10 deletions(-)
> > > > > 
> > > > > diff --git a/drivers/vhost/net.c b/drivers/vhost/net.c
> > > > > index e158159671fa..e4d5f843f9c0 100644
> > > > > --- a/drivers/vhost/net.c
> > > > > +++ b/drivers/vhost/net.c
> > > > > @@ -1505,10 +1505,13 @@ static long vhost_net_set_backend(struct vhost_net *n, unsigned index, int fd)
> > > > > 
> > > > >       mutex_lock(&n->dev.mutex);
> > > > >       r = vhost_dev_check_owner(&n->dev);
> > > > > -     if (r)
> > > > > +     if (r) {
> > > > > +             pr_debug("vhost_dev_check_owner index=%u fd=%d rc r=%d", index, fd, r);
> > > > >               goto err;
> > > > > +     }
> > > > > 
> > > > >       if (index >= VHOST_NET_VQ_MAX) {
> > > > > +             pr_debug("vhost_dev_check_owner index=%u fd=%d MAX=%d", index, fd, VHOST_NET_VQ_MAX);
> > > > >               r = -ENOBUFS;
> > > > >               goto err;
> > > > >       }
> > > > > @@ -1518,22 +1521,26 @@ static long vhost_net_set_backend(struct vhost_net *n, unsigned index, int fd)
> > > > > 
> > > > >       /* Verify that ring has been setup correctly. */
> > > > >       if (!vhost_vq_access_ok(vq)) {
> > > > > +             pr_debug("vhost_net_set_backend index=%u fd=%d !vhost_vq_access_ok", index, fd);
> > > > >               r = -EFAULT;
> > > > >               goto err_vq;
> > > > >       }
> > > > >       sock = get_socket(fd);
> > > > >       if (IS_ERR(sock)) {
> > > > >               r = PTR_ERR(sock);
> > > > > +             pr_debug("vhost_net_set_backend index=%u fd=%d get_socket err r=%d", index, fd, r);
> > > > >               goto err_vq;
> > > > >       }
> > > > > 
> > > > >       /* start polling new socket */
> > > > >       oldsock = vq->private_data;
> > > > >       if (sock != oldsock) {
> > > > > +             pr_debug("sock=%p != oldsock=%p index=%u fd=%d vq=%p", sock, oldsock, index, fd, vq);
> > > > >               ubufs = vhost_net_ubuf_alloc(vq,
> > > > >                                            sock && vhost_sock_zcopy(sock));
> > > > >               if (IS_ERR(ubufs)) {
> > > > >                       r = PTR_ERR(ubufs);
> > > > > +                     pr_debug("ubufs index=%u fd=%d err r=%d vq=%p", index, fd, r, vq);
> > > > >                       goto err_ubufs;
> > > > >               }
> > > > > 
> > > > > @@ -1541,11 +1548,15 @@ static long vhost_net_set_backend(struct vhost_net *n, unsigned index, int fd)
> > > > >               vq->private_data = sock;
> > > > >               vhost_net_buf_unproduce(nvq);
> > > > >               r = vhost_vq_init_access(vq);
> > > > > -             if (r)
> > > > > +             if (r) {
> > > > > +                     pr_debug("init_access index=%u fd=%d r=%d vq=%p", index, fd, r, vq);
> > > > >                       goto err_used;
> > > > > +             }
> > > > >               r = vhost_net_enable_vq(n, vq);
> > > > > -             if (r)
> > > > > +             if (r) {
> > > > > +                     pr_debug("enable_vq index=%u fd=%d r=%d vq=%p", index, fd, r, vq);
> > > > >                       goto err_used;
> > > > > +             }
> > > > >               if (index == VHOST_NET_VQ_RX)
> > > > >                       nvq->rx_ring = get_tap_ptr_ring(fd);
> > > > > 
> > > > > @@ -1559,6 +1570,8 @@ static long vhost_net_set_backend(struct vhost_net *n, unsigned index, int fd)
> > > > > 
> > > > >       mutex_unlock(&vq->mutex);
> > > > > 
> > > > > +     pr_debug("sock=%p", sock);
> > > > > +
> > > > >       if (oldubufs) {
> > > > >               vhost_net_ubuf_put_wait_and_free(oldubufs);
> > > > >               mutex_lock(&vq->mutex);
> > > > > @@ -1710,6 +1723,7 @@ static long vhost_net_ioctl(struct file *f, unsigned int ioctl,
> > > > > 
> > > > >       switch (ioctl) {
> > > > >       case VHOST_NET_SET_BACKEND:
> > > > > +             pr_debug("VHOST_NET_SET_BACKEND");
> > > > >               if (copy_from_user(&backend, argp, sizeof backend))
> > > > >                       return -EFAULT;
> > > > >               return vhost_net_set_backend(n, backend.index, backend.fd);
> > > > > diff --git a/drivers/vhost/vhost.c b/drivers/vhost/vhost.c
> > > > > index b5a51b1f2e79..ec25ba32fe81 100644
> > > > > --- a/drivers/vhost/vhost.c
> > > > > +++ b/drivers/vhost/vhost.c
> > > > > @@ -1642,15 +1642,30 @@ long vhost_vring_ioctl(struct vhost_dev *d, unsigned int ioctl, void __user *arg
> > > > >                       r = -EINVAL;
> > > > >                       break;
> > > > >               }
> > > > > +
> > > > > +             if (vq->last_avail_idx || vq->avail_idx) {
> > > > > +                     pr_debug(
> > > > > +                             "strange VHOST_SET_VRING_BASE [vq=%p][s.index=%u][s.num=%u]",
> > > > > +                             vq, s.index, s.num);
> > > > > +                     dump_stack();
> > > > > +                     r = 0;
> > > > > +                     break;
> > > > > +             }
> > > > >               vq->last_avail_idx = s.num;
> > > > >               /* Forget the cached index value. */
> > > > >               vq->avail_idx = vq->last_avail_idx;
> > > > > +             pr_debug(
> > > > > +                     "VHOST_SET_VRING_BASE [vq=%p][vq->last_avail_idx=%u][vq-
> > > > > >avail_idx=%u][s.index=%u][s.num=%u]",
> > > > > +                     vq, vq->last_avail_idx, vq->avail_idx, s.index, s.num);
> > > > >               break;
> > > > >       case VHOST_GET_VRING_BASE:
> > > > >               s.index = idx;
> > > > >               s.num = vq->last_avail_idx;
> > > > >               if (copy_to_user(argp, &s, sizeof s))
> > > > >                       r = -EFAULT;
> > > > > +             pr_debug(
> > > > > +                     "VHOST_GET_VRING_BASE [vq=%p][vq->last_avail_idx=%u][vq-
> > > > > >avail_idx=%u][s.index=%u][s.num=%u]",
> > > > > +                     vq, vq->last_avail_idx, vq->avail_idx, s.index, s.num);
> > > > >               break;
> > > > >       case VHOST_SET_VRING_KICK:
> > > > >               if (copy_from_user(&f, argp, sizeof f)) {
> > > > > @@ -2239,8 +2254,8 @@ static int fetch_buf(struct vhost_virtqueue *vq)
> > > > >               vq->avail_idx = vhost16_to_cpu(vq, avail_idx);
> > > > > 
> > > > >               if (unlikely((u16)(vq->avail_idx - last_avail_idx) > vq->num)) {
> > > > > -                     vq_err(vq, "Guest moved used index from %u to %u",
> > > > > -                             last_avail_idx, vq->avail_idx);
> > > > > +                     vq_err(vq, "Guest moved vq %p used index from %u to %u",
> > > > > +                             vq, last_avail_idx, vq->avail_idx);
> > > > >                       return -EFAULT;
> > > > >               }
> > > > > 
> > > > > @@ -2316,6 +2331,9 @@ static int fetch_buf(struct vhost_virtqueue *vq)
> > > > >       BUG_ON(!(vq->used_flags & VRING_USED_F_NO_NOTIFY));
> > > > > 
> > > > >       /* On success, increment avail index. */
> > > > > +     pr_debug(
> > > > > +             "[vq=%p][vq->last_avail_idx=%u][vq->avail_idx=%u][vq->ndescs=%d][vq->first_desc=%d]",
> > > > > +             vq, vq->last_avail_idx, vq->avail_idx, vq->ndescs, vq->first_desc);
> > > > >       vq->last_avail_idx++;
> > > > > 
> > > > >       return 0;
> > > > > @@ -2432,6 +2450,9 @@ EXPORT_SYMBOL_GPL(vhost_get_vq_desc);
> > > > >  /* Reverse the effect of vhost_get_vq_desc. Useful for error handling. */
> > > > >  void vhost_discard_vq_desc(struct vhost_virtqueue *vq, int n)
> > > > >  {
> > > > > +     pr_debug(
> > > > > +             "DISCARD [vq=%p][vq->last_avail_idx=%u][vq->avail_idx=%u][n=%d]",
> > > > > +             vq, vq->last_avail_idx, vq->avail_idx, n);
> > > > >       vq->last_avail_idx -= n;
> > > > >  }
> > > > >  EXPORT_SYMBOL_GPL(vhost_discard_vq_desc);
> > > > > diff --git a/drivers/vhost/vhost.h b/drivers/vhost/vhost.h
> > > > > index 661088ae6dc7..08f6d2ccb697 100644
> > > > > --- a/drivers/vhost/vhost.h
> > > > > +++ b/drivers/vhost/vhost.h
> > > > > @@ -250,11 +250,11 @@ int vhost_init_device_iotlb(struct vhost_dev *d, bool enabled);
> > > > >       } while (0)
> > > > > 
> > > > >  enum {
> > > > > -     VHOST_FEATURES = (1ULL << VIRTIO_F_NOTIFY_ON_EMPTY) |
> > > > > -                      (1ULL << VIRTIO_RING_F_INDIRECT_DESC) |
> > > > > -                      (1ULL << VIRTIO_RING_F_EVENT_IDX) |
> > > > > -                      (1ULL << VHOST_F_LOG_ALL) |
> > > > > -                      (1ULL << VIRTIO_F_ANY_LAYOUT) |
> > > > > +     VHOST_FEATURES = /* (1ULL << VIRTIO_F_NOTIFY_ON_EMPTY) | */
> > > > > +                      /* (1ULL << VIRTIO_RING_F_INDIRECT_DESC) | */
> > > > > +                      /* (1ULL << VIRTIO_RING_F_EVENT_IDX) | */
> > > > > +                      /* (1ULL << VHOST_F_LOG_ALL) | */
> > > > > +                      /* (1ULL << VIRTIO_F_ANY_LAYOUT) | */
> > > > >                        (1ULL << VIRTIO_F_VERSION_1)
> > > > >  };
> > > > > 
> > > > > 

Can you try the inlined patch over 52c36ce7f334 ("vhost: use batched version by default")? My intention is to check if
"strange VHOST_SET_VRING_BASE" line appears. In previous tests, it appears very fast, but maybe it takes some time for
it to appear, or it does not appear anymore.

Thanks!

From 756a96e489688b2c04230580770aac17c8a46265 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Eugenio=20P=C3=A9rez?= <eperezma@redhat.com>
Date: Fri, 14 Feb 2020 08:02:26 +0100
Subject: [PATCH] vhost: disable all features and trace last_avail_idx and
 ioctl calls

---
 drivers/vhost/net.c   | 20 +++++++++++++++++---
 drivers/vhost/vhost.c | 25 +++++++++++++++++++++++--
 drivers/vhost/vhost.h | 10 +++++-----
 3 files changed, 45 insertions(+), 10 deletions(-)

diff --git a/drivers/vhost/net.c b/drivers/vhost/net.c
index e158159671fa..e4d5f843f9c0 100644
--- a/drivers/vhost/net.c
+++ b/drivers/vhost/net.c
@@ -1505,10 +1505,13 @@ static long vhost_net_set_backend(struct vhost_net *n, unsigned index, int fd)
 
 	mutex_lock(&n->dev.mutex);
 	r = vhost_dev_check_owner(&n->dev);
-	if (r)
+	if (r) {
+		pr_debug("vhost_dev_check_owner index=%u fd=%d rc r=%d", index, fd, r);
 		goto err;
+	}
 
 	if (index >= VHOST_NET_VQ_MAX) {
+		pr_debug("vhost_dev_check_owner index=%u fd=%d MAX=%d", index, fd, VHOST_NET_VQ_MAX);
 		r = -ENOBUFS;
 		goto err;
 	}
@@ -1518,22 +1521,26 @@ static long vhost_net_set_backend(struct vhost_net *n, unsigned index, int fd)
 
 	/* Verify that ring has been setup correctly. */
 	if (!vhost_vq_access_ok(vq)) {
+		pr_debug("vhost_net_set_backend index=%u fd=%d !vhost_vq_access_ok", index, fd);
 		r = -EFAULT;
 		goto err_vq;
 	}
 	sock = get_socket(fd);
 	if (IS_ERR(sock)) {
 		r = PTR_ERR(sock);
+		pr_debug("vhost_net_set_backend index=%u fd=%d get_socket err r=%d", index, fd, r);
 		goto err_vq;
 	}
 
 	/* start polling new socket */
 	oldsock = vq->private_data;
 	if (sock != oldsock) {
+		pr_debug("sock=%p != oldsock=%p index=%u fd=%d vq=%p", sock, oldsock, index, fd, vq);
 		ubufs = vhost_net_ubuf_alloc(vq,
 					     sock && vhost_sock_zcopy(sock));
 		if (IS_ERR(ubufs)) {
 			r = PTR_ERR(ubufs);
+			pr_debug("ubufs index=%u fd=%d err r=%d vq=%p", index, fd, r, vq);
 			goto err_ubufs;
 		}
 
@@ -1541,11 +1548,15 @@ static long vhost_net_set_backend(struct vhost_net *n, unsigned index, int fd)
 		vq->private_data = sock;
 		vhost_net_buf_unproduce(nvq);
 		r = vhost_vq_init_access(vq);
-		if (r)
+		if (r) {
+			pr_debug("init_access index=%u fd=%d r=%d vq=%p", index, fd, r, vq);
 			goto err_used;
+		}
 		r = vhost_net_enable_vq(n, vq);
-		if (r)
+		if (r) {
+			pr_debug("enable_vq index=%u fd=%d r=%d vq=%p", index, fd, r, vq);
 			goto err_used;
+		}
 		if (index == VHOST_NET_VQ_RX)
 			nvq->rx_ring = get_tap_ptr_ring(fd);
 
@@ -1559,6 +1570,8 @@ static long vhost_net_set_backend(struct vhost_net *n, unsigned index, int fd)
 
 	mutex_unlock(&vq->mutex);
 
+	pr_debug("sock=%p", sock);
+
 	if (oldubufs) {
 		vhost_net_ubuf_put_wait_and_free(oldubufs);
 		mutex_lock(&vq->mutex);
@@ -1710,6 +1723,7 @@ static long vhost_net_ioctl(struct file *f, unsigned int ioctl,
 
 	switch (ioctl) {
 	case VHOST_NET_SET_BACKEND:
+		pr_debug("VHOST_NET_SET_BACKEND");
 		if (copy_from_user(&backend, argp, sizeof backend))
 			return -EFAULT;
 		return vhost_net_set_backend(n, backend.index, backend.fd);
diff --git a/drivers/vhost/vhost.c b/drivers/vhost/vhost.c
index 56c5253056ee..babc48c8a8c4 100644
--- a/drivers/vhost/vhost.c
+++ b/drivers/vhost/vhost.c
@@ -1640,15 +1640,30 @@ long vhost_vring_ioctl(struct vhost_dev *d, unsigned int ioctl, void __user *arg
 			r = -EINVAL;
 			break;
 		}
+
+		if (vq->last_avail_idx || vq->avail_idx) {
+			pr_debug(
+				"strange VHOST_SET_VRING_BASE [vq=%p][s.index=%u][s.num=%u]",
+				vq, s.index, s.num);
+			dump_stack();
+			r = 0;
+			break;
+		}
 		vq->last_avail_idx = s.num;
 		/* Forget the cached index value. */
 		vq->avail_idx = vq->last_avail_idx;
+		pr_debug(
+			"VHOST_SET_VRING_BASE [vq=%p][vq->last_avail_idx=%u][vq->avail_idx=%u][s.index=%u][s.num=%u]",
+			vq, vq->last_avail_idx, vq->avail_idx, s.index, s.num);
 		break;
 	case VHOST_GET_VRING_BASE:
 		s.index = idx;
 		s.num = vq->last_avail_idx;
 		if (copy_to_user(argp, &s, sizeof s))
 			r = -EFAULT;
+		pr_debug(
+			"VHOST_GET_VRING_BASE [vq=%p][vq->last_avail_idx=%u][vq->avail_idx=%u][s.index=%u][s.num=%u]",
+			vq, vq->last_avail_idx, vq->avail_idx, s.index, s.num);
 		break;
 	case VHOST_SET_VRING_KICK:
 		if (copy_from_user(&f, argp, sizeof f)) {
@@ -2233,8 +2248,8 @@ static int fetch_descs(struct vhost_virtqueue *vq)
 		vq->avail_idx = vhost16_to_cpu(vq, avail_idx);
 
 		if (unlikely((u16)(vq->avail_idx - last_avail_idx) > vq->num)) {
-			vq_err(vq, "Guest moved used index from %u to %u",
-				last_avail_idx, vq->avail_idx);
+			vq_err(vq, "Guest moved vq %p used index from %u to %u",
+				vq, last_avail_idx, vq->avail_idx);
 			return -EFAULT;
 		}
 
@@ -2310,6 +2325,9 @@ static int fetch_descs(struct vhost_virtqueue *vq)
 	BUG_ON(!(vq->used_flags & VRING_USED_F_NO_NOTIFY));
 
 	/* On success, increment avail index. */
+	pr_debug(
+		"[vq=%p][vq->last_avail_idx=%u][vq->avail_idx=%u][vq->ndescs=%d][vq->first_desc=%d]",
+		vq, vq->last_avail_idx, vq->avail_idx, vq->ndescs, vq->first_desc);
 	vq->last_avail_idx++;
 
 	return 0;
@@ -2403,6 +2421,9 @@ EXPORT_SYMBOL_GPL(vhost_get_vq_desc);
 /* Reverse the effect of vhost_get_vq_desc. Useful for error handling. */
 void vhost_discard_vq_desc(struct vhost_virtqueue *vq, int n)
 {
+	pr_debug(
+		"DISCARD [vq=%p][vq->last_avail_idx=%u][vq->avail_idx=%u][n=%d]",
+		vq, vq->last_avail_idx, vq->avail_idx, n);
 	vq->last_avail_idx -= n;
 }
 EXPORT_SYMBOL_GPL(vhost_discard_vq_desc);
diff --git a/drivers/vhost/vhost.h b/drivers/vhost/vhost.h
index a0bcf8bffa43..2ce2d3a97c31 100644
--- a/drivers/vhost/vhost.h
+++ b/drivers/vhost/vhost.h
@@ -248,11 +248,11 @@ int vhost_init_device_iotlb(struct vhost_dev *d, bool enabled);
 	} while (0)
 
 enum {
-	VHOST_FEATURES = (1ULL << VIRTIO_F_NOTIFY_ON_EMPTY) |
-			 (1ULL << VIRTIO_RING_F_INDIRECT_DESC) |
-			 (1ULL << VIRTIO_RING_F_EVENT_IDX) |
-			 (1ULL << VHOST_F_LOG_ALL) |
-			 (1ULL << VIRTIO_F_ANY_LAYOUT) |
+	VHOST_FEATURES = /* (1ULL << VIRTIO_F_NOTIFY_ON_EMPTY) | */
+			 /* (1ULL << VIRTIO_RING_F_INDIRECT_DESC) | */
+			 /* (1ULL << VIRTIO_RING_F_EVENT_IDX) | */
+			 /* (1ULL << VHOST_F_LOG_ALL) | */
+			 /* (1ULL << VIRTIO_F_ANY_LAYOUT) | */
 			 (1ULL << VIRTIO_F_VERSION_1)
 };
 
-- 
2.18.1

Re: vhost changes (batched) in linux-next after 12/13 trigger random crashes in KVM guests after reboot

[Christian Borntraeger]

On 14.02.20 13:17, Eugenio Pérez wrote:
> Can you try the inlined patch over 52c36ce7f334 ("vhost: use batched version by default")? My intention is to check if
> "strange VHOST_SET_VRING_BASE" line appears. In previous tests, it appears very fast, but maybe it takes some time for
> it to appear, or it does not appear anymore.

  LD [M]  drivers/vhost/vhost_vsock.o
  CC [M]  drivers/vhost/vhost.o
In file included from ./include/linux/printk.h:331,
                 from ./include/linux/kernel.h:15,
                 from ./include/linux/list.h:9,
                 from ./include/linux/wait.h:7,
                 from ./include/linux/eventfd.h:13,
                 from drivers/vhost/vhost.c:13:
drivers/vhost/vhost.c: In function ‘fetch_descs’:
drivers/vhost/vhost.c:2330:56: error: ‘struct vhost_virtqueue’ has no member named ‘first_desc’
 2330 |   vq, vq->last_avail_idx, vq->avail_idx, vq->ndescs, vq->first_desc);
      |                                                        ^~
./include/linux/dynamic_debug.h:125:15: note: in definition of macro ‘__dynamic_func_call’
  125 |   func(&id, ##__VA_ARGS__);  \
      |               ^~~~~~~~~~~
./include/linux/dynamic_debug.h:153:2: note: in expansion of macro ‘_dynamic_func_call’
  153 |  _dynamic_func_call(fmt, __dynamic_pr_debug,  \
      |  ^~~~~~~~~~~~~~~~~~
./include/linux/printk.h:335:2: note: in expansion of macro ‘dynamic_pr_debug’
  335 |  dynamic_pr_debug(fmt, ##__VA_ARGS__)
      |  ^~~~~~~~~~~~~~~~
drivers/vhost/vhost.c:2328:2: note: in expansion of macro ‘pr_debug’
 2328 |  pr_debug(
      |  ^~~~~~~~
make[2]: *** [scripts/Makefile.build:266: drivers/vhost/vhost.o] Error 1
make[1]: *** [scripts/Makefile.build:503: drivers/vhost] Error 2

Re: vhost changes (batched) in linux-next after 12/13 trigger random crashes in KVM guests after reboot

[Eugenio Pérez]

On Fri, 2020-02-14 at 13:22 +0100, Christian Borntraeger wrote:
> 
> On 14.02.20 13:17, Eugenio Pérez wrote:
> > Can you try the inlined patch over 52c36ce7f334 ("vhost: use batched version by default")? My intention is to check
> > if
> > "strange VHOST_SET_VRING_BASE" line appears. In previous tests, it appears very fast, but maybe it takes some time
> > for
> > it to appear, or it does not appear anymore.
> 
>   LD [M]  drivers/vhost/vhost_vsock.o
>   CC [M]  drivers/vhost/vhost.o
> In file included from ./include/linux/printk.h:331,
>                  from ./include/linux/kernel.h:15,
>                  from ./include/linux/list.h:9,
>                  from ./include/linux/wait.h:7,
>                  from ./include/linux/eventfd.h:13,
>                  from drivers/vhost/vhost.c:13:
> drivers/vhost/vhost.c: In function ‘fetch_descs’:
> drivers/vhost/vhost.c:2330:56: error: ‘struct vhost_virtqueue’ has no member named ‘first_desc’
>  2330 |   vq, vq->last_avail_idx, vq->avail_idx, vq->ndescs, vq->first_desc);
>       |                                                        ^~
> ./include/linux/dynamic_debug.h:125:15: note: in definition of macro ‘__dynamic_func_call’
>   125 |   func(&id, ##__VA_ARGS__);  \
>       |               ^~~~~~~~~~~
> ./include/linux/dynamic_debug.h:153:2: note: in expansion of macro ‘_dynamic_func_call’
>   153 |  _dynamic_func_call(fmt, __dynamic_pr_debug,  \
>       |  ^~~~~~~~~~~~~~~~~~
> ./include/linux/printk.h:335:2: note: in expansion of macro ‘dynamic_pr_debug’
>   335 |  dynamic_pr_debug(fmt, ##__VA_ARGS__)
>       |  ^~~~~~~~~~~~~~~~
> drivers/vhost/vhost.c:2328:2: note: in expansion of macro ‘pr_debug’
>  2328 |  pr_debug(
>       |  ^~~~~~~~
> make[2]: *** [scripts/Makefile.build:266: drivers/vhost/vhost.o] Error 1
> make[1]: *** [scripts/Makefile.build:503: drivers/vhost] Error 2
> 

Sorry about that. Here is the right patch.

From 5d7b5304c163910936382d46561fc43eb770aad2 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Eugenio=20P=C3=A9rez?= <eperezma@redhat.com>
Date: Fri, 14 Feb 2020 08:02:26 +0100
Subject: [PATCH] vhost: disable all features and trace last_avail_idx and
 ioctl calls

---
 drivers/vhost/net.c   | 20 +++++++++++++++++---
 drivers/vhost/vhost.c | 25 +++++++++++++++++++++++--
 drivers/vhost/vhost.h | 10 +++++-----
 3 files changed, 45 insertions(+), 10 deletions(-)

diff --git a/drivers/vhost/net.c b/drivers/vhost/net.c
index e158159671fa..e4d5f843f9c0 100644
--- a/drivers/vhost/net.c
+++ b/drivers/vhost/net.c
@@ -1505,10 +1505,13 @@ static long vhost_net_set_backend(struct vhost_net *n, unsigned index, int fd)
 
 	mutex_lock(&n->dev.mutex);
 	r = vhost_dev_check_owner(&n->dev);
-	if (r)
+	if (r) {
+		pr_debug("vhost_dev_check_owner index=%u fd=%d rc r=%d", index, fd, r);
 		goto err;
+	}
 
 	if (index >= VHOST_NET_VQ_MAX) {
+		pr_debug("vhost_dev_check_owner index=%u fd=%d MAX=%d", index, fd, VHOST_NET_VQ_MAX);
 		r = -ENOBUFS;
 		goto err;
 	}
@@ -1518,22 +1521,26 @@ static long vhost_net_set_backend(struct vhost_net *n, unsigned index, int fd)
 
 	/* Verify that ring has been setup correctly. */
 	if (!vhost_vq_access_ok(vq)) {
+		pr_debug("vhost_net_set_backend index=%u fd=%d !vhost_vq_access_ok", index, fd);
 		r = -EFAULT;
 		goto err_vq;
 	}
 	sock = get_socket(fd);
 	if (IS_ERR(sock)) {
 		r = PTR_ERR(sock);
+		pr_debug("vhost_net_set_backend index=%u fd=%d get_socket err r=%d", index, fd, r);
 		goto err_vq;
 	}
 
 	/* start polling new socket */
 	oldsock = vq->private_data;
 	if (sock != oldsock) {
+		pr_debug("sock=%p != oldsock=%p index=%u fd=%d vq=%p", sock, oldsock, index, fd, vq);
 		ubufs = vhost_net_ubuf_alloc(vq,
 					     sock && vhost_sock_zcopy(sock));
 		if (IS_ERR(ubufs)) {
 			r = PTR_ERR(ubufs);
+			pr_debug("ubufs index=%u fd=%d err r=%d vq=%p", index, fd, r, vq);
 			goto err_ubufs;
 		}
 
@@ -1541,11 +1548,15 @@ static long vhost_net_set_backend(struct vhost_net *n, unsigned index, int fd)
 		vq->private_data = sock;
 		vhost_net_buf_unproduce(nvq);
 		r = vhost_vq_init_access(vq);
-		if (r)
+		if (r) {
+			pr_debug("init_access index=%u fd=%d r=%d vq=%p", index, fd, r, vq);
 			goto err_used;
+		}
 		r = vhost_net_enable_vq(n, vq);
-		if (r)
+		if (r) {
+			pr_debug("enable_vq index=%u fd=%d r=%d vq=%p", index, fd, r, vq);
 			goto err_used;
+		}
 		if (index == VHOST_NET_VQ_RX)
 			nvq->rx_ring = get_tap_ptr_ring(fd);
 
@@ -1559,6 +1570,8 @@ static long vhost_net_set_backend(struct vhost_net *n, unsigned index, int fd)
 
 	mutex_unlock(&vq->mutex);
 
+	pr_debug("sock=%p", sock);
+
 	if (oldubufs) {
 		vhost_net_ubuf_put_wait_and_free(oldubufs);
 		mutex_lock(&vq->mutex);
@@ -1710,6 +1723,7 @@ static long vhost_net_ioctl(struct file *f, unsigned int ioctl,
 
 	switch (ioctl) {
 	case VHOST_NET_SET_BACKEND:
+		pr_debug("VHOST_NET_SET_BACKEND");
 		if (copy_from_user(&backend, argp, sizeof backend))
 			return -EFAULT;
 		return vhost_net_set_backend(n, backend.index, backend.fd);
diff --git a/drivers/vhost/vhost.c b/drivers/vhost/vhost.c
index 56c5253056ee..2e72bbeffac7 100644
--- a/drivers/vhost/vhost.c
+++ b/drivers/vhost/vhost.c
@@ -1640,15 +1640,30 @@ long vhost_vring_ioctl(struct vhost_dev *d, unsigned int ioctl, void __user *arg
 			r = -EINVAL;
 			break;
 		}
+
+		if (vq->last_avail_idx || vq->avail_idx) {
+			pr_debug(
+				"strange VHOST_SET_VRING_BASE [vq=%p][s.index=%u][s.num=%u]",
+				vq, s.index, s.num);
+			dump_stack();
+			r = 0;
+			break;
+		}
 		vq->last_avail_idx = s.num;
 		/* Forget the cached index value. */
 		vq->avail_idx = vq->last_avail_idx;
+		pr_debug(
+			"VHOST_SET_VRING_BASE [vq=%p][vq->last_avail_idx=%u][vq->avail_idx=%u][s.index=%u][s.num=%u]",
+			vq, vq->last_avail_idx, vq->avail_idx, s.index, s.num);
 		break;
 	case VHOST_GET_VRING_BASE:
 		s.index = idx;
 		s.num = vq->last_avail_idx;
 		if (copy_to_user(argp, &s, sizeof s))
 			r = -EFAULT;
+		pr_debug(
+			"VHOST_GET_VRING_BASE [vq=%p][vq->last_avail_idx=%u][vq->avail_idx=%u][s.index=%u][s.num=%u]",
+			vq, vq->last_avail_idx, vq->avail_idx, s.index, s.num);
 		break;
 	case VHOST_SET_VRING_KICK:
 		if (copy_from_user(&f, argp, sizeof f)) {
@@ -2233,8 +2248,8 @@ static int fetch_descs(struct vhost_virtqueue *vq)
 		vq->avail_idx = vhost16_to_cpu(vq, avail_idx);
 
 		if (unlikely((u16)(vq->avail_idx - last_avail_idx) > vq->num)) {
-			vq_err(vq, "Guest moved used index from %u to %u",
-				last_avail_idx, vq->avail_idx);
+			vq_err(vq, "Guest moved vq %p used index from %u to %u",
+				vq, last_avail_idx, vq->avail_idx);
 			return -EFAULT;
 		}
 
@@ -2310,6 +2325,9 @@ static int fetch_descs(struct vhost_virtqueue *vq)
 	BUG_ON(!(vq->used_flags & VRING_USED_F_NO_NOTIFY));
 
 	/* On success, increment avail index. */
+	pr_debug(
+		"[vq=%p][vq->last_avail_idx=%u][vq->avail_idx=%u][vq->ndescs=%d]",
+		vq, vq->last_avail_idx, vq->avail_idx, vq->ndescs);
 	vq->last_avail_idx++;
 
 	return 0;
@@ -2403,6 +2421,9 @@ EXPORT_SYMBOL_GPL(vhost_get_vq_desc);
 /* Reverse the effect of vhost_get_vq_desc. Useful for error handling. */
 void vhost_discard_vq_desc(struct vhost_virtqueue *vq, int n)
 {
+	pr_debug(
+		"DISCARD [vq=%p][vq->last_avail_idx=%u][vq->avail_idx=%u][n=%d]",
+		vq, vq->last_avail_idx, vq->avail_idx, n);
 	vq->last_avail_idx -= n;
 }
 EXPORT_SYMBOL_GPL(vhost_discard_vq_desc);
diff --git a/drivers/vhost/vhost.h b/drivers/vhost/vhost.h
index a0bcf8bffa43..2ce2d3a97c31 100644
--- a/drivers/vhost/vhost.h
+++ b/drivers/vhost/vhost.h
@@ -248,11 +248,11 @@ int vhost_init_device_iotlb(struct vhost_dev *d, bool enabled);
 	} while (0)
 
 enum {
-	VHOST_FEATURES = (1ULL << VIRTIO_F_NOTIFY_ON_EMPTY) |
-			 (1ULL << VIRTIO_RING_F_INDIRECT_DESC) |
-			 (1ULL << VIRTIO_RING_F_EVENT_IDX) |
-			 (1ULL << VHOST_F_LOG_ALL) |
-			 (1ULL << VIRTIO_F_ANY_LAYOUT) |
+	VHOST_FEATURES = /* (1ULL << VIRTIO_F_NOTIFY_ON_EMPTY) | */
+			 /* (1ULL << VIRTIO_RING_F_INDIRECT_DESC) | */
+			 /* (1ULL << VIRTIO_RING_F_EVENT_IDX) | */
+			 /* (1ULL << VHOST_F_LOG_ALL) | */
+			 /* (1ULL << VIRTIO_F_ANY_LAYOUT) | */
 			 (1ULL << VIRTIO_F_VERSION_1)
 };
 
-- 
2.18.1

Re: vhost changes (batched) in linux-next after 12/13 trigger random crashes in KVM guests after reboot

[Christian Borntraeger]

On 14.02.20 13:26, Eugenio Pérez wrote:
> On Fri, 2020-02-14 at 13:22 +0100, Christian Borntraeger wrote:
>>
>> On 14.02.20 13:17, Eugenio Pérez wrote:
>>> Can you try the inlined patch over 52c36ce7f334 ("vhost: use batched version by default")? My intention is to check
>>> if
>>> "strange VHOST_SET_VRING_BASE" line appears. In previous tests, it appears very fast, but maybe it takes some time
>>> for
>>> it to appear, or it does not appear anymore.


yep it does:

[   67.801012] [1917] vhost:vhost_vring_ioctl:1655: VHOST_SET_VRING_BASE [vq=0000000088199421][vq->last_avail_idx=0][vq->avail_idx=0][s.index=0][s.num=0]
[   67.801018] [1917] vhost:vhost_vring_ioctl:1655: VHOST_SET_VRING_BASE [vq=00000000175f11ec][vq->last_avail_idx=0][vq->avail_idx=0][s.index=1][s.num=0]
[   67.801026] [1917] vhost_net:vhost_net_ioctl:1726: VHOST_NET_SET_BACKEND
[   67.801028] [1917] vhost_net:vhost_net_set_backend:1538: sock=0000000082d8d291 != oldsock=000000001ae027fd index=0 fd=39 vq=0000000088199421
[   67.801032] [1917] vhost_net:vhost_net_set_backend:1573: sock=0000000082d8d291
[   67.801033] [1917] vhost_net:vhost_net_ioctl:1726: VHOST_NET_SET_BACKEND
[   67.801034] [1917] vhost_net:vhost_net_set_backend:1538: sock=0000000082d8d291 != oldsock=000000001ae027fd index=1 fd=39 vq=00000000175f11ec
[   67.801035] [1917] vhost_net:vhost_net_set_backend:1573: sock=0000000082d8d291
[   67.801037] [1915] vhost:vhost_discard_vq_desc:2424: DISCARD [vq=0000000088199421][vq->last_avail_idx=0][vq->avail_idx=0][n=0]
[   68.648803] [1915] vhost:fetch_descs:2328: [vq=0000000088199421][vq->last_avail_idx=0][vq->avail_idx=256][vq->ndescs=1]
[   68.648810] [1915] vhost:fetch_descs:2328: [vq=0000000088199421][vq->last_avail_idx=1][vq->avail_idx=256][vq->ndescs=1]
[   68.648812] [1915] vhost:fetch_descs:2328: [vq=0000000088199421][vq->last_avail_idx=2][vq->avail_idx=256][vq->ndescs=1]
[   68.648815] [1915] vhost:fetch_descs:2328: [vq=0000000088199421][vq->last_avail_idx=3][vq->avail_idx=256][vq->ndescs=1]
[   68.648817] [1915] vhost:fetch_descs:2328: [vq=0000000088199421][vq->last_avail_idx=4][vq->avail_idx=256][vq->ndescs=1]
[   68.648818] [1915] vhost:fetch_descs:2328: [vq=0000000088199421][vq->last_avail_idx=5][vq->avail_idx=256][vq->ndescs=1]
[   68.648820] [1915] vhost:fetch_descs:2328: [vq=0000000088199421][vq->last_avail_idx=6][vq->avail_idx=256][vq->ndescs=1]
[   68.648822] [1915] vhost:fetch_descs:2328: [vq=0000000088199421][vq->last_avail_idx=7][vq->avail_idx=256][vq->ndescs=1]
[   68.648824] [1915] vhost:fetch_descs:2328: [vq=0000000088199421][vq->last_avail_idx=8][vq->avail_idx=256][vq->ndescs=1]
[   68.648826] [1915] vhost:fetch_descs:2328: [vq=0000000088199421][vq->last_avail_idx=9][vq->avail_idx=256][vq->ndescs=1]
[   68.648828] [1915] vhost:fetch_descs:2328: [vq=0000000088199421][vq->last_avail_idx=10][vq->avail_idx=256][vq->ndescs=1]
[   68.648829] [1915] vhost:fetch_descs:2328: [vq=0000000088199421][vq->last_avail_idx=11][vq->avail_idx=256][vq->ndescs=1]
[   68.648831] [1915] vhost:fetch_descs:2328: [vq=0000000088199421][vq->last_avail_idx=12][vq->avail_idx=256][vq->ndescs=1]
[   68.648832] [1915] vhost:fetch_descs:2328: [vq=0000000088199421][vq->last_avail_idx=13][vq->avail_idx=256][vq->ndescs=1]
[   68.648833] [1915] vhost:fetch_descs:2328: [vq=0000000088199421][vq->last_avail_idx=14][vq->avail_idx=256][vq->ndescs=1]
[   68.670292] [1915] vhost:fetch_descs:2328: [vq=00000000175f11ec][vq->last_avail_idx=0][vq->avail_idx=1][vq->ndescs=1]
[   68.687187] [1915] vhost:fetch_descs:2328: [vq=00000000175f11ec][vq->last_avail_idx=1][vq->avail_idx=2][vq->ndescs=1]
[   68.687623] [1915] vhost:fetch_descs:2328: [vq=0000000088199421][vq->last_avail_idx=15][vq->avail_idx=256][vq->ndescs=1]
[   68.687641] [1915] vhost:fetch_descs:2328: [vq=00000000175f11ec][vq->last_avail_idx=2][vq->avail_idx=4][vq->ndescs=1]
[   68.687642] [1915] vhost:fetch_descs:2328: [vq=00000000175f11ec][vq->last_avail_idx=3][vq->avail_idx=4][vq->ndescs=1]
[   68.690274] [1915] vhost:fetch_descs:2328: [vq=0000000088199421][vq->last_avail_idx=16][vq->avail_idx=256][vq->ndescs=1]
[   68.690539] [1915] vhost:fetch_descs:2328: [vq=00000000175f11ec][vq->last_avail_idx=4][vq->avail_idx=5][vq->ndescs=1]
[   68.715379] [1915] vhost:fetch_descs:2328: [vq=0000000088199421][vq->last_avail_idx=17][vq->avail_idx=256][vq->ndescs=1]
[   68.800525] [1915] vhost:fetch_descs:2328: [vq=00000000175f11ec][vq->last_avail_idx=5][vq->avail_idx=6][vq->ndescs=1]
[   68.890537] [1915] vhost:fetch_descs:2328: [vq=00000000175f11ec][vq->last_avail_idx=6][vq->avail_idx=7][vq->ndescs=1]
[   68.900587] [1915] vhost:fetch_descs:2328: [vq=00000000175f11ec][vq->last_avail_idx=7][vq->avail_idx=8][vq->ndescs=1]
[   68.916837] [1915] vhost:fetch_descs:2328: [vq=00000000175f11ec][vq->last_avail_idx=8][vq->avail_idx=9][vq->ndescs=2]
[   68.928828] [1915] vhost:fetch_descs:2328: [vq=0000000088199421][vq->last_avail_idx=18][vq->avail_idx=256][vq->ndescs=1]
[   69.090540] [1915] vhost:fetch_descs:2328: [vq=00000000175f11ec][vq->last_avail_idx=9][vq->avail_idx=10][vq->ndescs=1]
[   69.119651] [1915] vhost:fetch_descs:2328: [vq=00000000175f11ec][vq->last_avail_idx=10][vq->avail_idx=11][vq->ndescs=2]
[   69.132325] [1915] vhost:fetch_descs:2328: [vq=0000000088199421][vq->last_avail_idx=19][vq->avail_idx=256][vq->ndescs=1]
[   69.323473] [1915] vhost:fetch_descs:2328: [vq=00000000175f11ec][vq->last_avail_idx=11][vq->avail_idx=12][vq->ndescs=2]
[   69.354557] [1915] vhost:fetch_descs:2328: [vq=0000000088199421][vq->last_avail_idx=20][vq->avail_idx=256][vq->ndescs=1]
[   69.442550] [1915] vhost:fetch_descs:2328: [vq=0000000088199421][vq->last_avail_idx=21][vq->avail_idx=256][vq->ndescs=1]
[   69.523593] [1915] vhost:fetch_descs:2328: [vq=00000000175f11ec][vq->last_avail_idx=12][vq->avail_idx=13][vq->ndescs=2]
[   69.557360] [1915] vhost:fetch_descs:2328: [vq=0000000088199421][vq->last_avail_idx=22][vq->avail_idx=256][vq->ndescs=1]
[   69.980634] [1915] vhost:fetch_descs:2328: [vq=00000000175f11ec][vq->last_avail_idx=13][vq->avail_idx=14][vq->ndescs=1]
[   69.981364] [1915] vhost:fetch_descs:2328: [vq=00000000175f11ec][vq->last_avail_idx=14][vq->avail_idx=15][vq->ndescs=1]
[   70.010545] [1915] vhost:fetch_descs:2328: [vq=00000000175f11ec][vq->last_avail_idx=15][vq->avail_idx=16][vq->ndescs=1]
[   70.161316] [1915] vhost:fetch_descs:2328: [vq=0000000088199421][vq->last_avail_idx=23][vq->avail_idx=256][vq->ndescs=1]
[   70.177640] [1915] vhost:fetch_descs:2328: [vq=0000000088199421][vq->last_avail_idx=24][vq->avail_idx=256][vq->ndescs=1]
[   70.280564] [1915] vhost:fetch_descs:2328: [vq=00000000175f11ec][vq->last_avail_idx=16][vq->avail_idx=17][vq->ndescs=1]
[   70.670327] [1915] vhost:fetch_descs:2328: [vq=00000000175f11ec][vq->last_avail_idx=17][vq->avail_idx=18][vq->ndescs=1]
[   70.932887] [1915] vhost:fetch_descs:2328: [vq=00000000175f11ec][vq->last_avail_idx=18][vq->avail_idx=19][vq->ndescs=2]
[   70.940587] [1915] vhost:fetch_descs:2328: [vq=00000000175f11ec][vq->last_avail_idx=19][vq->avail_idx=20][vq->ndescs=1]
[   70.947598] [1915] vhost:fetch_descs:2328: [vq=0000000088199421][vq->last_avail_idx=25][vq->avail_idx=256][vq->ndescs=1]
[   71.070388] [1915] vhost:fetch_descs:2328: [vq=00000000175f11ec][vq->last_avail_idx=20][vq->avail_idx=21][vq->ndescs=1]
[   71.070770] [1915] vhost:fetch_descs:2328: [vq=0000000088199421][vq->last_avail_idx=26][vq->avail_idx=256][vq->ndescs=1]
[   71.070805] [1915] vhost:fetch_descs:2328: [vq=00000000175f11ec][vq->last_avail_idx=21][vq->avail_idx=22][vq->ndescs=1]
[   71.070977] [1915] vhost:fetch_descs:2328: [vq=0000000088199421][vq->last_avail_idx=27][vq->avail_idx=256][vq->ndescs=1]
[   71.071049] [1915] vhost:fetch_descs:2328: [vq=00000000175f11ec][vq->last_avail_idx=22][vq->avail_idx=23][vq->ndescs=1]
[   71.071195] [1915] vhost:fetch_descs:2328: [vq=0000000088199421][vq->last_avail_idx=28][vq->avail_idx=256][vq->ndescs=1]
[   71.071243] [1915] vhost:fetch_descs:2328: [vq=00000000175f11ec][vq->last_avail_idx=23][vq->avail_idx=24][vq->ndescs=1]
[   71.071386] [1915] vhost:fetch_descs:2328: [vq=0000000088199421][vq->last_avail_idx=29][vq->avail_idx=256][vq->ndescs=1]
[   71.071433] [1915] vhost:fetch_descs:2328: [vq=00000000175f11ec][vq->last_avail_idx=24][vq->avail_idx=25][vq->ndescs=1]
[   71.071575] [1915] vhost:fetch_descs:2328: [vq=0000000088199421][vq->last_avail_idx=30][vq->avail_idx=256][vq->ndescs=1]
[   71.071611] [1915] vhost:fetch_descs:2328: [vq=00000000175f11ec][vq->last_avail_idx=25][vq->avail_idx=26][vq->ndescs=1]
[   71.071747] [1915] vhost:fetch_descs:2328: [vq=0000000088199421][vq->last_avail_idx=31][vq->avail_idx=256][vq->ndescs=1]
[   71.071789] [1915] vhost:fetch_descs:2328: [vq=00000000175f11ec][vq->last_avail_idx=26][vq->avail_idx=27][vq->ndescs=1]
[   71.071923] [1915] vhost:fetch_descs:2328: [vq=0000000088199421][vq->last_avail_idx=32][vq->avail_idx=256][vq->ndescs=1]
[   71.071960] [1915] vhost:fetch_descs:2328: [vq=00000000175f11ec][vq->last_avail_idx=27][vq->avail_idx=28][vq->ndescs=1]
[   71.072096] [1915] vhost:fetch_descs:2328: [vq=0000000088199421][vq->last_avail_idx=33][vq->avail_idx=256][vq->ndescs=1]
[   71.072128] [1915] vhost:fetch_descs:2328: [vq=00000000175f11ec][vq->last_avail_idx=28][vq->avail_idx=29][vq->ndescs=1]
[   71.072267] [1915] vhost:fetch_descs:2328: [vq=0000000088199421][vq->last_avail_idx=34][vq->avail_idx=256][vq->ndescs=1]
[   71.072300] [1915] vhost:fetch_descs:2328: [vq=00000000175f11ec][vq->last_avail_idx=29][vq->avail_idx=30][vq->ndescs=1]
[   71.072432] [1915] vhost:fetch_descs:2328: [vq=0000000088199421][vq->last_avail_idx=35][vq->avail_idx=256][vq->ndescs=1]
[   71.072463] [1915] vhost:fetch_descs:2328: [vq=00000000175f11ec][vq->last_avail_idx=30][vq->avail_idx=31][vq->ndescs=1]
[   71.072596] [1915] vhost:fetch_descs:2328: [vq=0000000088199421][vq->last_avail_idx=36][vq->avail_idx=256][vq->ndescs=1]
[   71.072630] [1915] vhost:fetch_descs:2328: [vq=00000000175f11ec][vq->last_avail_idx=31][vq->avail_idx=32][vq->ndescs=1]
[   71.072759] [1915] vhost:fetch_descs:2328: [vq=0000000088199421][vq->last_avail_idx=37][vq->avail_idx=256][vq->ndescs=1]
[   71.072791] [1915] vhost:fetch_descs:2328: [vq=00000000175f11ec][vq->last_avail_idx=32][vq->avail_idx=33][vq->ndescs=1]
[   71.072933] [1915] vhost:fetch_descs:2328: [vq=0000000088199421][vq->last_avail_idx=38][vq->avail_idx=256][vq->ndescs=1]
[   71.073054] [1915] vhost:fetch_descs:2328: [vq=00000000175f11ec][vq->last_avail_idx=33][vq->avail_idx=34][vq->ndescs=1]
[   71.073193] [1915] vhost:fetch_descs:2328: [vq=0000000088199421][vq->last_avail_idx=39][vq->avail_idx=256][vq->ndescs=1]
[   71.073247] [1915] vhost:fetch_descs:2328: [vq=00000000175f11ec][vq->last_avail_idx=34][vq->avail_idx=35][vq->ndescs=1]
[   71.073383] [1915] vhost:fetch_descs:2328: [vq=0000000088199421][vq->last_avail_idx=40][vq->avail_idx=256][vq->ndescs=1]
[   71.073434] [1915] vhost:fetch_descs:2328: [vq=00000000175f11ec][vq->last_avail_idx=35][vq->avail_idx=36][vq->ndescs=1]
[   71.073571] [1915] vhost:fetch_descs:2328: [vq=0000000088199421][vq->last_avail_idx=41][vq->avail_idx=256][vq->ndescs=1]
[   71.073627] [1915] vhost:fetch_descs:2328: [vq=00000000175f11ec][vq->last_avail_idx=36][vq->avail_idx=37][vq->ndescs=1]
[   71.073762] [1915] vhost:fetch_descs:2328: [vq=0000000088199421][vq->last_avail_idx=42][vq->avail_idx=256][vq->ndescs=1]
[   71.073813] [1915] vhost:fetch_descs:2328: [vq=00000000175f11ec][vq->last_avail_idx=37][vq->avail_idx=38][vq->ndescs=1]
[   71.073948] [1915] vhost:fetch_descs:2328: [vq=0000000088199421][vq->last_avail_idx=43][vq->avail_idx=256][vq->ndescs=1]
[   71.073998] [1915] vhost:fetch_descs:2328: [vq=00000000175f11ec][vq->last_avail_idx=38][vq->avail_idx=39][vq->ndescs=1]
[   71.074136] [1915] vhost:fetch_descs:2328: [vq=0000000088199421][vq->last_avail_idx=44][vq->avail_idx=256][vq->ndescs=1]
[   71.074186] [1915] vhost:fetch_descs:2328: [vq=00000000175f11ec][vq->last_avail_idx=39][vq->avail_idx=40][vq->ndescs=1]
[   71.074320] [1915] vhost:fetch_descs:2328: [vq=0000000088199421][vq->last_avail_idx=45][vq->avail_idx=256][vq->ndescs=1]
[   71.074370] [1915] vhost:fetch_descs:2328: [vq=00000000175f11ec][vq->last_avail_idx=40][vq->avail_idx=41][vq->ndescs=1]
[   71.074503] [1915] vhost:fetch_descs:2328: [vq=0000000088199421][vq->last_avail_idx=46][vq->avail_idx=256][vq->ndescs=1]
[   72.344493] [1915] vhost:fetch_descs:2328: [vq=0000000088199421][vq->last_avail_idx=47][vq->avail_idx=256][vq->ndescs=1]
[   72.553413] [1915] vhost:fetch_descs:2328: [vq=0000000088199421][vq->last_avail_idx=48][vq->avail_idx=256][vq->ndescs=1]
[   73.522174] [1915] vhost:fetch_descs:2328: [vq=0000000088199421][vq->last_avail_idx=49][vq->avail_idx=256][vq->ndescs=1]
[   73.705202] [1915] vhost:fetch_descs:2328: [vq=0000000088199421][vq->last_avail_idx=50][vq->avail_idx=256][vq->ndescs=1]
[   73.705239] [1915] vhost:fetch_descs:2328: [vq=00000000175f11ec][vq->last_avail_idx=41][vq->avail_idx=42][vq->ndescs=1]
[   73.994388] [1915] vhost:fetch_descs:2328: [vq=0000000088199421][vq->last_avail_idx=51][vq->avail_idx=256][vq->ndescs=1]
[   74.208443] [1915] vhost:fetch_descs:2328: [vq=0000000088199421][vq->last_avail_idx=52][vq->avail_idx=256][vq->ndescs=1]
[   74.433345] [1915] vhost:fetch_descs:2328: [vq=0000000088199421][vq->last_avail_idx=53][vq->avail_idx=256][vq->ndescs=1]
[   74.594756] [1910] vhost_net:vhost_net_ioctl:1726: VHOST_NET_SET_BACKEND
[   74.594761] [1910] vhost_net:vhost_net_set_backend:1538: sock=000000001ae027fd != oldsock=0000000082d8d291 index=0 fd=-1 vq=0000000088199421
[   74.594762] [1910] vhost_net:vhost_net_set_backend:1573: sock=000000001ae027fd
[   74.594803] [1910] vhost_net:vhost_net_ioctl:1726: VHOST_NET_SET_BACKEND
[   74.594804] [1910] vhost_net:vhost_net_set_backend:1538: sock=000000001ae027fd != oldsock=0000000082d8d291 index=1 fd=-1 vq=00000000175f11ec
[   74.594805] [1910] vhost_net:vhost_net_set_backend:1573: sock=000000001ae027fd
[   74.594847] [1910] vhost:vhost_vring_ioctl:1664: VHOST_GET_VRING_BASE [vq=0000000088199421][vq->last_avail_idx=54][vq->avail_idx=256][s.index=0][s.num=54]
[   74.594850] [1910] vhost:vhost_vring_ioctl:1664: VHOST_GET_VRING_BASE [vq=00000000175f11ec][vq->last_avail_idx=42][vq->avail_idx=42][s.index=1][s.num=42]
[   77.003191] [1918] vhost:vhost_vring_ioctl:1645: strange VHOST_SET_VRING_BASE [vq=0000000088199421][s.index=0][s.num=0]
[   77.003194] CPU: 62 PID: 1918 Comm: CPU 1/KVM Not tainted 5.5.0+ #22
[   77.003197] Hardware name: IBM 3906 M04 704 (LPAR)
[   77.003198] Call Trace:
[   77.003207]  [<0000000b8d93c132>] show_stack+0x8a/0xd0 
[   77.003211]  [<0000000b8e32e72a>] dump_stack+0x8a/0xb8 
[   77.003224]  [<000003ff803567ae>] vhost_vring_ioctl+0x6fe/0x858 [vhost] 
[   77.003228]  [<000003ff8036c670>] vhost_net_ioctl+0x510/0x570 [vhost_net] 
[   77.003234]  [<0000000b8dbecdd8>] do_vfs_ioctl+0x430/0x6f8 
[   77.003235]  [<0000000b8dbed124>] ksys_ioctl+0x84/0xb0 
[   77.003237]  [<0000000b8dbed1ba>] __s390x_sys_ioctl+0x2a/0x38 
[   77.003240]  [<0000000b8e34ff72>] system_call+0x2a6/0x2c8 
[   77.003261] [1918] vhost:vhost_vring_ioctl:1645: strange VHOST_SET_VRING_BASE [vq=00000000175f11ec][s.index=1][s.num=0]
[   77.003262] CPU: 62 PID: 1918 Comm: CPU 1/KVM Not tainted 5.5.0+ #22
[   77.003263] Hardware name: IBM 3906 M04 704 (LPAR)
[   77.003264] Call Trace:
[   77.003266]  [<0000000b8d93c132>] show_stack+0x8a/0xd0 
[   77.003267]  [<0000000b8e32e72a>] dump_stack+0x8a/0xb8 
[   77.003270]  [<000003ff803567ae>] vhost_vring_ioctl+0x6fe/0x858 [vhost] 
[   77.003271]  [<000003ff8036c670>] vhost_net_ioctl+0x510/0x570 [vhost_net] 
[   77.003273]  [<0000000b8dbecdd8>] do_vfs_ioctl+0x430/0x6f8 
[   77.003274]  [<0000000b8dbed124>] ksys_ioctl+0x84/0xb0 
[   77.003276]  [<0000000b8dbed1ba>] __s390x_sys_ioctl+0x2a/0x38 
[   77.003277]  [<0000000b8e34ff72>] system_call+0x2a6/0x2c8 
[   77.003297] [1918] vhost_net:vhost_net_ioctl:1726: VHOST_NET_SET_BACKEND
[   77.003300] [1918] vhost_net:vhost_net_set_backend:1538: sock=0000000082d8d291 != oldsock=000000001ae027fd index=0 fd=39 vq=0000000088199421
[   77.003304] [1918] vhost_net:vhost_net_set_backend:1573: sock=0000000082d8d291
[   77.003305] [1918] vhost_net:vhost_net_ioctl:1726: VHOST_NET_SET_BACKEND
[   77.003306] [1918] vhost_net:vhost_net_set_backend:1538: sock=0000000082d8d291 != oldsock=000000001ae027fd index=1 fd=39 vq=00000000175f11ec
[   77.003308] [1915] vhost:fetch_descs:2328: [vq=0000000088199421][vq->last_avail_idx=54][vq->avail_idx=256][vq->ndescs=1]
[   77.003308] [1918] vhost_net:vhost_net_set_backend:1573: sock=0000000082d8d291
[   77.003310] [1915] vhost_net:get_rx_bufs:1061: unexpected descriptor format for RX: out 0, in 0
[   77.003312] [1915] vhost:vhost_discard_vq_desc:2424: DISCARD [vq=0000000088199421][vq->last_avail_idx=55][vq->avail_idx=256][n=0]

Re: vhost changes (batched) in linux-next after 12/13 trigger random crashes in KVM guests after reboot

[Eugenio Perez Martin]

On Tue, Feb 11, 2020 at 2:58 PM Michael S. Tsirkin <mst@redhat.com> wrote:
>
> On Tue, Feb 11, 2020 at 02:04:54PM +0100, Eugenio Pérez wrote:
> > On Mon, 2020-02-10 at 12:01 +0100, Christian Borntraeger wrote:
> > >
> > > On 10.02.20 10:47, Eugenio Perez Martin wrote:
> > > > Hi Christian.
> > > >
> > > > I'm not able to reproduce the failure with eccb852f1fe6bede630e2e4f1a121a81e34354ab commit. Could you add more data?
> > > > Your configuration (libvirt or qemu line), and host's dmesg output if any?
> > > >
> > > > Thanks!
> > >
> > > If it was not obvious, this is on s390x, a big endian system.
> > >
> >
> > Hi Christian. Thank you very much for your fast responses.
> >
> > Could you try this patch on top of eccb852f1fe6bede630e2e4f1a121a81e34354ab?
> >
> > Thanks!
> >
> > >From 71d0f9108a18aa894cc0c0c1c7efbad39f465a27 Mon Sep 17 00:00:00 2001
> > From: =?UTF-8?q?Eugenio=20P=C3=A9rez?= <
> > eperezma@redhat.com>
> > Date: Tue, 11 Feb 2020 13:19:10 +0100
> > Subject: [PATCH] vhost: fix return value of vhost_get_vq_desc
> >
> > Before of the batch change, it was the chain's head. Need to keep that
> > way or we will not be able to free a chain of descriptors.
>
> I think it's cleaner to have all descriptors in the chain
> have the same id.
>

Hi Michael.

Not sure if I understood you correctly. All the descriptors in the
chain still have the same id, this just returns the first one instead
of update it in every loop iteration. Can left unapplied in the final
version though, is up to you.

> > Fixes: eccb852f1fe6 ("vhost: batching fetches")
> > ---
> >  drivers/vhost/vhost.c | 3 +--
> >  1 file changed, 1 insertion(+), 2 deletions(-)
> >
> > diff --git a/drivers/vhost/vhost.c b/drivers/vhost/vhost.c
> > index b5a51b1f2e79..fc422c3e5c08 100644
> > --- a/drivers/vhost/vhost.c
> > +++ b/drivers/vhost/vhost.c
> > @@ -2409,12 +2409,11 @@ int vhost_get_vq_desc(struct vhost_virtqueue *vq,
> >                       *out_num += ret;
> >               }
> >
> > -             ret = desc->id;
> > -
> >               if (!(desc->flags & VRING_DESC_F_NEXT))
> >                       break;
> >       }
> >
> > +     ret = vq->descs[vq->first_desc].id;
> >       vq->first_desc = i + 1;
> >
> >       return ret;
> > --
> > 2.18.1
>

Re: vhost changes (batched) in linux-next after 12/13 trigger random crashes in KVM guests after reboot

[Eugenio Pérez]

Hi Christian.

Sorry for the late response. Could we try this one over eccb852f1fe6bede630e2e4f1a121a81e34354ab, and see if you still
can reproduce the bug?

Apart from that, could you print me the backtrace when qemu calls vhost_kernel_set_vring_base and
vhost_kernel_get_vring_base functions?

Thank you very much!

diff --git a/drivers/vhost/net.c b/drivers/vhost/net.c
index e158159671fa..a1a4239512bb 100644
--- a/drivers/vhost/net.c
+++ b/drivers/vhost/net.c
@@ -1505,10 +1505,13 @@ static long vhost_net_set_backend(struct vhost_net *n, unsigned index, int fd)
 
 	mutex_lock(&n->dev.mutex);
 	r = vhost_dev_check_owner(&n->dev);
-	if (r)
+	if (r) {
+		pr_debug("vhost_dev_check_owner index=%u fd=%d rc r=%d", index, fd, r);
 		goto err;
+	}
 
 	if (index >= VHOST_NET_VQ_MAX) {
+		pr_debug("vhost_dev_check_owner index=%u fd=%d MAX=%d", index, fd, VHOST_NET_VQ_MAX);
 		r = -ENOBUFS;
 		goto err;
 	}
@@ -1518,22 +1521,26 @@ static long vhost_net_set_backend(struct vhost_net *n, unsigned index, int fd)
 
 	/* Verify that ring has been setup correctly. */
 	if (!vhost_vq_access_ok(vq)) {
+		pr_debug("vhost_net_set_backend index=%u fd=%d !vhost_vq_access_ok", index, fd);
 		r = -EFAULT;
 		goto err_vq;
 	}
 	sock = get_socket(fd);
 	if (IS_ERR(sock)) {
 		r = PTR_ERR(sock);
+		pr_debug("vhost_net_set_backend index=%u fd=%d get_socket err r=%d", index, fd, r);
 		goto err_vq;
 	}
 
 	/* start polling new socket */
 	oldsock = vq->private_data;
 	if (sock != oldsock) {
+		pr_debug("sock=%p != oldsock=%p index=%u fd=%d vq=%p", sock, oldsock, index, fd, vq);
 		ubufs = vhost_net_ubuf_alloc(vq,
 					     sock && vhost_sock_zcopy(sock));
 		if (IS_ERR(ubufs)) {
 			r = PTR_ERR(ubufs);
+			pr_debug("ubufs index=%u fd=%d err r=%d vq=%p", index, fd, r, vq);
 			goto err_ubufs;
 		}
 
@@ -1541,11 +1548,15 @@ static long vhost_net_set_backend(struct vhost_net *n, unsigned index, int fd)
 		vq->private_data = sock;
 		vhost_net_buf_unproduce(nvq);
 		r = vhost_vq_init_access(vq);
-		if (r)
+		if (r) {
+			pr_debug("init_access index=%u fd=%d r=%d vq=%p", index, fd, r, vq);
 			goto err_used;
+		}
 		r = vhost_net_enable_vq(n, vq);
-		if (r)
+		if (r) {
+			pr_debug("enable_vq index=%u fd=%d r=%d vq=%p", index, fd, r, vq);
 			goto err_used;
+		}
 		if (index == VHOST_NET_VQ_RX)
 			nvq->rx_ring = get_tap_ptr_ring(fd);
 
@@ -1559,6 +1570,8 @@ static long vhost_net_set_backend(struct vhost_net *n, unsigned index, int fd)
 
 	mutex_unlock(&vq->mutex);
 
+	pr_debug("sock=%p", sock);
+
 	if (oldubufs) {
 		vhost_net_ubuf_put_wait_and_free(oldubufs);
 		mutex_lock(&vq->mutex);
@@ -1712,6 +1725,9 @@ static long vhost_net_ioctl(struct file *f, unsigned int ioctl,
 	case VHOST_NET_SET_BACKEND:
 		if (copy_from_user(&backend, argp, sizeof backend))
 			return -EFAULT;
+		pr_debug("VHOST_NET_SET_BACKEND [b.index=%u][b.fd=%d]",
+			 backend.index, backend.fd);
+		dump_stack();
 		return vhost_net_set_backend(n, backend.index, backend.fd);
 	case VHOST_GET_FEATURES:
 		features = VHOST_NET_FEATURES;
diff --git a/drivers/vhost/vhost.c b/drivers/vhost/vhost.c
index b5a51b1f2e79..9dd0bcae0b22 100644
--- a/drivers/vhost/vhost.c
+++ b/drivers/vhost/vhost.c
@@ -372,6 +372,11 @@ static int vhost_worker(void *data)
 	return 0;
 }
 
+static int vhost_vq_num_batch_descs(struct vhost_virtqueue *vq)
+{
+	return vq->max_descs - UIO_MAXIOV;
+}
+
 static void vhost_vq_free_iovecs(struct vhost_virtqueue *vq)
 {
 	kfree(vq->descs);
@@ -394,7 +399,9 @@ static long vhost_dev_alloc_iovecs(struct vhost_dev *dev)
 	for (i = 0; i < dev->nvqs; ++i) {
 		vq = dev->vqs[i];
 		vq->max_descs = dev->iov_limit;
-		vq->batch_descs = dev->iov_limit - UIO_MAXIOV;
+		if (vhost_vq_num_batch_descs(vq) < 0) {
+			return -EINVAL;
+		}
 		vq->descs = kmalloc_array(vq->max_descs,
 					  sizeof(*vq->descs),
 					  GFP_KERNEL);
@@ -1642,15 +1649,27 @@ long vhost_vring_ioctl(struct vhost_dev *d, unsigned int ioctl, void __user *arg
 			r = -EINVAL;
 			break;
 		}
+
+		pr_debug(
+			"VHOST_SET_VRING_BASE [vq=%p][s.index=%u][s.num=%u][vq->avail_idx=%d][vq->last_avail_idx=%d][vq-
>ndescs=%d][vq->first_desc=%d]",
+			vq, s.index, s.num, vq->avail_idx, vq->last_avail_idx,
+			vq->ndescs, vq->first_desc);
+		dump_stack();
 		vq->last_avail_idx = s.num;
 		/* Forget the cached index value. */
 		vq->avail_idx = vq->last_avail_idx;
+		vq->ndescs = vq->first_desc = 0;
 		break;
 	case VHOST_GET_VRING_BASE:
 		s.index = idx;
 		s.num = vq->last_avail_idx;
 		if (copy_to_user(argp, &s, sizeof s))
 			r = -EFAULT;
+		pr_debug(
+			"VHOST_GET_VRING_BASE [vq=%p][s.index=%u][s.num=%u][vq->avail_idx=%d][vq->last_avail_idx=%d][vq-
>ndescs=%d][vq->first_desc=%d]",
+			vq, s.index, s.num, vq->avail_idx, vq->last_avail_idx,
+			vq->ndescs, vq->first_desc);
+		dump_stack();
 		break;
 	case VHOST_SET_VRING_KICK:
 		if (copy_from_user(&f, argp, sizeof f)) {
@@ -2239,8 +2258,8 @@ static int fetch_buf(struct vhost_virtqueue *vq)
 		vq->avail_idx = vhost16_to_cpu(vq, avail_idx);
 
 		if (unlikely((u16)(vq->avail_idx - last_avail_idx) > vq->num)) {
-			vq_err(vq, "Guest moved used index from %u to %u",
-				last_avail_idx, vq->avail_idx);
+			vq_err(vq, "Guest moved vq %p used index from %u to %u",
+				vq, last_avail_idx, vq->avail_idx);
 			return -EFAULT;
 		}
 
@@ -2316,6 +2335,9 @@ static int fetch_buf(struct vhost_virtqueue *vq)
 	BUG_ON(!(vq->used_flags & VRING_USED_F_NO_NOTIFY));
 
 	/* On success, increment avail index. */
+	pr_debug(
+		"[vq=%p][vq->last_avail_idx=%u][vq->avail_idx=%u][vq->ndescs=%d][vq->first_desc=%d]",
+		vq, vq->last_avail_idx, vq->avail_idx, vq->ndescs, vq->first_desc);
 	vq->last_avail_idx++;
 
 	return 0;
@@ -2333,7 +2355,7 @@ static int fetch_descs(struct vhost_virtqueue *vq)
 	if (vq->ndescs)
 		return 0;
 
-	while (!ret && vq->ndescs <= vq->batch_descs)
+	while (!ret && vq->ndescs <= vhost_vq_num_batch_descs(vq))
 		ret = fetch_buf(vq);
 
 	return vq->ndescs ? 0 : ret;
@@ -2432,6 +2454,9 @@ EXPORT_SYMBOL_GPL(vhost_get_vq_desc);
 /* Reverse the effect of vhost_get_vq_desc. Useful for error handling. */
 void vhost_discard_vq_desc(struct vhost_virtqueue *vq, int n)
 {
+	pr_debug(
+		"DISCARD [vq=%p][vq->last_avail_idx=%u][vq->avail_idx=%u][n=%d]",
+		vq, vq->last_avail_idx, vq->avail_idx, n);
 	vq->last_avail_idx -= n;
 }
 EXPORT_SYMBOL_GPL(vhost_discard_vq_desc);
diff --git a/drivers/vhost/vhost.h b/drivers/vhost/vhost.h
index 661088ae6dc7..e648b9b997d4 100644
--- a/drivers/vhost/vhost.h
+++ b/drivers/vhost/vhost.h
@@ -102,7 +102,6 @@ struct vhost_virtqueue {
 	int ndescs;
 	int first_desc;
 	int max_descs;
-	int batch_descs;
 
 	const struct vhost_umem_node *meta_iotlb[VHOST_NUM_ADDRS];
 	struct file *kick;

Re: vhost changes (batched) in linux-next after 12/13 trigger random crashes in KVM guests after reboot

[Christian Borntraeger]

On 27.03.20 12:08, Eugenio Pérez wrote:
> Hi Christian.
> 
> Sorry for the late response. Could we try this one over eccb852f1fe6bede630e2e4f1a121a81e34354ab, and see if you still
> can reproduce the bug?

To much time has passed and too many things have changed on that system.
I have trouble reproducing this with either
eccb852f1fe6bede630e2e4f1a121a81e34354ab or 52c36ce7f334.
I will try to reproduce this again :-/

> 
> Apart from that, could you print me the backtrace when qemu calls vhost_kernel_set_vring_base and
> vhost_kernel_get_vring_base functions?
> 
> Thank you very much!
> 
> diff --git a/drivers/vhost/net.c b/drivers/vhost/net.c
> index e158159671fa..a1a4239512bb 100644
> --- a/drivers/vhost/net.c
> +++ b/drivers/vhost/net.c
> @@ -1505,10 +1505,13 @@ static long vhost_net_set_backend(struct vhost_net *n, unsigned index, int fd)
>  
>  	mutex_lock(&n->dev.mutex);
>  	r = vhost_dev_check_owner(&n->dev);
> -	if (r)
> +	if (r) {
> +		pr_debug("vhost_dev_check_owner index=%u fd=%d rc r=%d", index, fd, r);
>  		goto err;
> +	}
>  
>  	if (index >= VHOST_NET_VQ_MAX) {
> +		pr_debug("vhost_dev_check_owner index=%u fd=%d MAX=%d", index, fd, VHOST_NET_VQ_MAX);
>  		r = -ENOBUFS;
>  		goto err;
>  	}
> @@ -1518,22 +1521,26 @@ static long vhost_net_set_backend(struct vhost_net *n, unsigned index, int fd)
>  
>  	/* Verify that ring has been setup correctly. */
>  	if (!vhost_vq_access_ok(vq)) {
> +		pr_debug("vhost_net_set_backend index=%u fd=%d !vhost_vq_access_ok", index, fd);
>  		r = -EFAULT;
>  		goto err_vq;
>  	}
>  	sock = get_socket(fd);
>  	if (IS_ERR(sock)) {
>  		r = PTR_ERR(sock);
> +		pr_debug("vhost_net_set_backend index=%u fd=%d get_socket err r=%d", index, fd, r);
>  		goto err_vq;
>  	}
>  
>  	/* start polling new socket */
>  	oldsock = vq->private_data;
>  	if (sock != oldsock) {
> +		pr_debug("sock=%p != oldsock=%p index=%u fd=%d vq=%p", sock, oldsock, index, fd, vq);
>  		ubufs = vhost_net_ubuf_alloc(vq,
>  					     sock && vhost_sock_zcopy(sock));
>  		if (IS_ERR(ubufs)) {
>  			r = PTR_ERR(ubufs);
> +			pr_debug("ubufs index=%u fd=%d err r=%d vq=%p", index, fd, r, vq);
>  			goto err_ubufs;
>  		}
>  
> @@ -1541,11 +1548,15 @@ static long vhost_net_set_backend(struct vhost_net *n, unsigned index, int fd)
>  		vq->private_data = sock;
>  		vhost_net_buf_unproduce(nvq);
>  		r = vhost_vq_init_access(vq);
> -		if (r)
> +		if (r) {
> +			pr_debug("init_access index=%u fd=%d r=%d vq=%p", index, fd, r, vq);
>  			goto err_used;
> +		}
>  		r = vhost_net_enable_vq(n, vq);
> -		if (r)
> +		if (r) {
> +			pr_debug("enable_vq index=%u fd=%d r=%d vq=%p", index, fd, r, vq);
>  			goto err_used;
> +		}
>  		if (index == VHOST_NET_VQ_RX)
>  			nvq->rx_ring = get_tap_ptr_ring(fd);
>  
> @@ -1559,6 +1570,8 @@ static long vhost_net_set_backend(struct vhost_net *n, unsigned index, int fd)
>  
>  	mutex_unlock(&vq->mutex);
>  
> +	pr_debug("sock=%p", sock);
> +
>  	if (oldubufs) {
>  		vhost_net_ubuf_put_wait_and_free(oldubufs);
>  		mutex_lock(&vq->mutex);
> @@ -1712,6 +1725,9 @@ static long vhost_net_ioctl(struct file *f, unsigned int ioctl,
>  	case VHOST_NET_SET_BACKEND:
>  		if (copy_from_user(&backend, argp, sizeof backend))
>  			return -EFAULT;
> +		pr_debug("VHOST_NET_SET_BACKEND [b.index=%u][b.fd=%d]",
> +			 backend.index, backend.fd);
> +		dump_stack();
>  		return vhost_net_set_backend(n, backend.index, backend.fd);
>  	case VHOST_GET_FEATURES:
>  		features = VHOST_NET_FEATURES;
> diff --git a/drivers/vhost/vhost.c b/drivers/vhost/vhost.c
> index b5a51b1f2e79..9dd0bcae0b22 100644
> --- a/drivers/vhost/vhost.c
> +++ b/drivers/vhost/vhost.c
> @@ -372,6 +372,11 @@ static int vhost_worker(void *data)
>  	return 0;
>  }
>  
> +static int vhost_vq_num_batch_descs(struct vhost_virtqueue *vq)
> +{
> +	return vq->max_descs - UIO_MAXIOV;
> +}
> +
>  static void vhost_vq_free_iovecs(struct vhost_virtqueue *vq)
>  {
>  	kfree(vq->descs);
> @@ -394,7 +399,9 @@ static long vhost_dev_alloc_iovecs(struct vhost_dev *dev)
>  	for (i = 0; i < dev->nvqs; ++i) {
>  		vq = dev->vqs[i];
>  		vq->max_descs = dev->iov_limit;
> -		vq->batch_descs = dev->iov_limit - UIO_MAXIOV;
> +		if (vhost_vq_num_batch_descs(vq) < 0) {
> +			return -EINVAL;
> +		}
>  		vq->descs = kmalloc_array(vq->max_descs,
>  					  sizeof(*vq->descs),
>  					  GFP_KERNEL);
> @@ -1642,15 +1649,27 @@ long vhost_vring_ioctl(struct vhost_dev *d, unsigned int ioctl, void __user *arg
>  			r = -EINVAL;
>  			break;
>  		}
> +
> +		pr_debug(
> +			"VHOST_SET_VRING_BASE [vq=%p][s.index=%u][s.num=%u][vq->avail_idx=%d][vq->last_avail_idx=%d][vq-
>> ndescs=%d][vq->first_desc=%d]",
> +			vq, s.index, s.num, vq->avail_idx, vq->last_avail_idx,
> +			vq->ndescs, vq->first_desc);
> +		dump_stack();
>  		vq->last_avail_idx = s.num;
>  		/* Forget the cached index value. */
>  		vq->avail_idx = vq->last_avail_idx;
> +		vq->ndescs = vq->first_desc = 0;
>  		break;
>  	case VHOST_GET_VRING_BASE:
>  		s.index = idx;
>  		s.num = vq->last_avail_idx;
>  		if (copy_to_user(argp, &s, sizeof s))
>  			r = -EFAULT;
> +		pr_debug(
> +			"VHOST_GET_VRING_BASE [vq=%p][s.index=%u][s.num=%u][vq->avail_idx=%d][vq->last_avail_idx=%d][vq-
>> ndescs=%d][vq->first_desc=%d]",
> +			vq, s.index, s.num, vq->avail_idx, vq->last_avail_idx,
> +			vq->ndescs, vq->first_desc);
> +		dump_stack();
>  		break;
>  	case VHOST_SET_VRING_KICK:
>  		if (copy_from_user(&f, argp, sizeof f)) {
> @@ -2239,8 +2258,8 @@ static int fetch_buf(struct vhost_virtqueue *vq)
>  		vq->avail_idx = vhost16_to_cpu(vq, avail_idx);
>  
>  		if (unlikely((u16)(vq->avail_idx - last_avail_idx) > vq->num)) {
> -			vq_err(vq, "Guest moved used index from %u to %u",
> -				last_avail_idx, vq->avail_idx);
> +			vq_err(vq, "Guest moved vq %p used index from %u to %u",
> +				vq, last_avail_idx, vq->avail_idx);
>  			return -EFAULT;
>  		}
>  
> @@ -2316,6 +2335,9 @@ static int fetch_buf(struct vhost_virtqueue *vq)
>  	BUG_ON(!(vq->used_flags & VRING_USED_F_NO_NOTIFY));
>  
>  	/* On success, increment avail index. */
> +	pr_debug(
> +		"[vq=%p][vq->last_avail_idx=%u][vq->avail_idx=%u][vq->ndescs=%d][vq->first_desc=%d]",
> +		vq, vq->last_avail_idx, vq->avail_idx, vq->ndescs, vq->first_desc);
>  	vq->last_avail_idx++;
>  
>  	return 0;
> @@ -2333,7 +2355,7 @@ static int fetch_descs(struct vhost_virtqueue *vq)
>  	if (vq->ndescs)
>  		return 0;
>  
> -	while (!ret && vq->ndescs <= vq->batch_descs)
> +	while (!ret && vq->ndescs <= vhost_vq_num_batch_descs(vq))
>  		ret = fetch_buf(vq);
>  
>  	return vq->ndescs ? 0 : ret;
> @@ -2432,6 +2454,9 @@ EXPORT_SYMBOL_GPL(vhost_get_vq_desc);
>  /* Reverse the effect of vhost_get_vq_desc. Useful for error handling. */
>  void vhost_discard_vq_desc(struct vhost_virtqueue *vq, int n)
>  {
> +	pr_debug(
> +		"DISCARD [vq=%p][vq->last_avail_idx=%u][vq->avail_idx=%u][n=%d]",
> +		vq, vq->last_avail_idx, vq->avail_idx, n);
>  	vq->last_avail_idx -= n;
>  }
>  EXPORT_SYMBOL_GPL(vhost_discard_vq_desc);
> diff --git a/drivers/vhost/vhost.h b/drivers/vhost/vhost.h
> index 661088ae6dc7..e648b9b997d4 100644
> --- a/drivers/vhost/vhost.h
> +++ b/drivers/vhost/vhost.h
> @@ -102,7 +102,6 @@ struct vhost_virtqueue {
>  	int ndescs;
>  	int first_desc;
>  	int max_descs;
> -	int batch_descs;
>  
>  	const struct vhost_umem_node *meta_iotlb[VHOST_NUM_ADDRS];
>  	struct file *kick;
>