vhost changes (batched) in linux-next after 12/13 trigger random crashes in KVM guests after reboot

vhost changes (batched) in linux-next after 12/13 trigger random crashes in KVM guests after reboot

[Christian Borntraeger]

Michael,

with 
commit db7286b100b503ef80612884453bed53d74c9a16 (refs/bisect/skip-db7286b100b503ef80612884453bed53d74c9a16)
    vhost: use batched version by default
plus
commit 6bd262d5eafcdf8cdfae491e2e748e4e434dcda6 (HEAD, refs/bisect/bad)
    Revert "vhost/net: add an option to test new code"
to make things compile (your next tree is not easily bisectable, can you fix that as well?).

I get random crashes in my s390 KVM guests after reboot.
Reverting both patches together with commit decd9b8 "vhost: use vhost_desc instead of vhost_log" to
make it compile again) on top of linux-next-1218 makes the problem go away.

Looks like the batched version is not yet ready for prime time. Can you drop these patches until
we have fixed the issues?

Christian

Re: vhost changes (batched) in linux-next after 12/13 trigger random crashes in KVM guests after reboot

[Michael S. Tsirkin]

On Wed, Dec 18, 2019 at 03:43:43PM +0100, Christian Borntraeger wrote:
> Michael,
> 
> with 
> commit db7286b100b503ef80612884453bed53d74c9a16 (refs/bisect/skip-db7286b100b503ef80612884453bed53d74c9a16)
>     vhost: use batched version by default
> plus
> commit 6bd262d5eafcdf8cdfae491e2e748e4e434dcda6 (HEAD, refs/bisect/bad)
>     Revert "vhost/net: add an option to test new code"
> to make things compile (your next tree is not easily bisectable, can you fix that as well?).

I'll try.

> 
> I get random crashes in my s390 KVM guests after reboot.
> Reverting both patches together with commit decd9b8 "vhost: use vhost_desc instead of vhost_log" to
> make it compile again) on top of linux-next-1218 makes the problem go away.
> 
> Looks like the batched version is not yet ready for prime time. Can you drop these patches until
> we have fixed the issues?
> 
> Christian
> 

Will do, thanks for letting me know.

-- 
MST

Re: vhost changes (batched) in linux-next after 12/13 trigger random crashes in KVM guests after reboot

[Christian Borntraeger]

On 18.12.19 16:10, Michael S. Tsirkin wrote:
> On Wed, Dec 18, 2019 at 03:43:43PM +0100, Christian Borntraeger wrote:
>> Michael,
>>
>> with 
>> commit db7286b100b503ef80612884453bed53d74c9a16 (refs/bisect/skip-db7286b100b503ef80612884453bed53d74c9a16)
>>     vhost: use batched version by default
>> plus
>> commit 6bd262d5eafcdf8cdfae491e2e748e4e434dcda6 (HEAD, refs/bisect/bad)
>>     Revert "vhost/net: add an option to test new code"
>> to make things compile (your next tree is not easily bisectable, can you fix that as well?).
> 
> I'll try.
> 
>>
>> I get random crashes in my s390 KVM guests after reboot.
>> Reverting both patches together with commit decd9b8 "vhost: use vhost_desc instead of vhost_log" to
>> make it compile again) on top of linux-next-1218 makes the problem go away.
>>
>> Looks like the batched version is not yet ready for prime time. Can you drop these patches until
>> we have fixed the issues?
>>
>> Christian
>>
> 
> Will do, thanks for letting me know.

I have confirmed with the initial reporter (internal test team) that <driver name='qemu'/> 
with a known to be broken linux next kernel also fixes the problem, so it is really the
vhost changes.

Re: vhost changes (batched) in linux-next after 12/13 trigger random crashes in KVM guests after reboot

[Michael S. Tsirkin]

On Wed, Dec 18, 2019 at 04:59:02PM +0100, Christian Borntraeger wrote:
> On 18.12.19 16:10, Michael S. Tsirkin wrote:
> > On Wed, Dec 18, 2019 at 03:43:43PM +0100, Christian Borntraeger wrote:
> >> Michael,
> >>
> >> with 
> >> commit db7286b100b503ef80612884453bed53d74c9a16 (refs/bisect/skip-db7286b100b503ef80612884453bed53d74c9a16)
> >>     vhost: use batched version by default
> >> plus
> >> commit 6bd262d5eafcdf8cdfae491e2e748e4e434dcda6 (HEAD, refs/bisect/bad)
> >>     Revert "vhost/net: add an option to test new code"
> >> to make things compile (your next tree is not easily bisectable, can you fix that as well?).
> > 
> > I'll try.
> > 
> >>
> >> I get random crashes in my s390 KVM guests after reboot.
> >> Reverting both patches together with commit decd9b8 "vhost: use vhost_desc instead of vhost_log" to
> >> make it compile again) on top of linux-next-1218 makes the problem go away.
> >>
> >> Looks like the batched version is not yet ready for prime time. Can you drop these patches until
> >> we have fixed the issues?
> >>
> >> Christian
> >>
> > 
> > Will do, thanks for letting me know.
> 
> I have confirmed with the initial reporter (internal test team) that <driver name='qemu'/> 
> with a known to be broken linux next kernel also fixes the problem, so it is really the
> vhost changes.

OK I'm back and trying to make it more bisectable.

I pushed a new tag "batch-v2".
It's same code but with this bisect should get more information.


I suspect one of the following:

commit 1414d7ee3d10d2ec2bc4ee652d1d90ec91da1c79
Author: Michael S. Tsirkin <mst@redhat.com>
Date:   Mon Oct 7 06:11:18 2019 -0400

    vhost: batching fetches
    
    With this patch applied, new and old code perform identically.
    
    Lots of extra optimizations are now possible, e.g.
    we can fetch multiple heads with copy_from/to_user now.
    We can get rid of maintaining the log array.  Etc etc.
    
    Signed-off-by: Michael S. Tsirkin <mst@redhat.com>

commit 50297a8480b439efc5f3f23088cb2d90b799acef
Author: Michael S. Tsirkin <mst@redhat.com>
Date:   Wed Dec 11 12:19:26 2019 -0500

    vhost: use batched version by default
    
    As testing shows no performance change, switch to that now.
    
    Signed-off-by: Michael S. Tsirkin <mst@redhat.com>


and would like to know which.

Thanks!

Re: vhost changes (batched) in linux-next after 12/13 trigger random crashes in KVM guests after reboot

[Christian Borntraeger]

On 06.01.20 11:50, Michael S. Tsirkin wrote:
> On Wed, Dec 18, 2019 at 04:59:02PM +0100, Christian Borntraeger wrote:
>> On 18.12.19 16:10, Michael S. Tsirkin wrote:
>>> On Wed, Dec 18, 2019 at 03:43:43PM +0100, Christian Borntraeger wrote:
>>>> Michael,
>>>>
>>>> with 
>>>> commit db7286b100b503ef80612884453bed53d74c9a16 (refs/bisect/skip-db7286b100b503ef80612884453bed53d74c9a16)
>>>>     vhost: use batched version by default
>>>> plus
>>>> commit 6bd262d5eafcdf8cdfae491e2e748e4e434dcda6 (HEAD, refs/bisect/bad)
>>>>     Revert "vhost/net: add an option to test new code"
>>>> to make things compile (your next tree is not easily bisectable, can you fix that as well?).
>>>
>>> I'll try.
>>>
>>>>
>>>> I get random crashes in my s390 KVM guests after reboot.
>>>> Reverting both patches together with commit decd9b8 "vhost: use vhost_desc instead of vhost_log" to
>>>> make it compile again) on top of linux-next-1218 makes the problem go away.
>>>>
>>>> Looks like the batched version is not yet ready for prime time. Can you drop these patches until
>>>> we have fixed the issues?
>>>>
>>>> Christian
>>>>
>>>
>>> Will do, thanks for letting me know.
>>
>> I have confirmed with the initial reporter (internal test team) that <driver name='qemu'/> 
>> with a known to be broken linux next kernel also fixes the problem, so it is really the
>> vhost changes.
> 
> OK I'm back and trying to make it more bisectable.
> 
> I pushed a new tag "batch-v2".
> It's same code but with this bisect should get more information.

I get the following with this tag

drivers/vhost/net.c: In function ‘vhost_net_tx_get_vq_desc’:
drivers/vhost/net.c:574:7: error: implicit declaration of function ‘vhost_get_vq_desc_batch’; did you mean ‘vhost_get_vq_desc’? [-Werror=implicit-function-declaration]
  574 |   r = vhost_get_vq_desc_batch(tvq, tvq->iov, ARRAY_SIZE(tvq->iov),
      |       ^~~~~~~~~~~~~~~~~~~~~~~
      |       vhost_get_vq_desc

Re: vhost changes (batched) in linux-next after 12/13 trigger random crashes in KVM guests after reboot

[Michael S. Tsirkin]

On Tue, Jan 07, 2020 at 09:59:16AM +0100, Christian Borntraeger wrote:
> 
> 
> On 06.01.20 11:50, Michael S. Tsirkin wrote:
> > On Wed, Dec 18, 2019 at 04:59:02PM +0100, Christian Borntraeger wrote:
> >> On 18.12.19 16:10, Michael S. Tsirkin wrote:
> >>> On Wed, Dec 18, 2019 at 03:43:43PM +0100, Christian Borntraeger wrote:
> >>>> Michael,
> >>>>
> >>>> with 
> >>>> commit db7286b100b503ef80612884453bed53d74c9a16 (refs/bisect/skip-db7286b100b503ef80612884453bed53d74c9a16)
> >>>>     vhost: use batched version by default
> >>>> plus
> >>>> commit 6bd262d5eafcdf8cdfae491e2e748e4e434dcda6 (HEAD, refs/bisect/bad)
> >>>>     Revert "vhost/net: add an option to test new code"
> >>>> to make things compile (your next tree is not easily bisectable, can you fix that as well?).
> >>>
> >>> I'll try.
> >>>
> >>>>
> >>>> I get random crashes in my s390 KVM guests after reboot.
> >>>> Reverting both patches together with commit decd9b8 "vhost: use vhost_desc instead of vhost_log" to
> >>>> make it compile again) on top of linux-next-1218 makes the problem go away.
> >>>>
> >>>> Looks like the batched version is not yet ready for prime time. Can you drop these patches until
> >>>> we have fixed the issues?
> >>>>
> >>>> Christian
> >>>>
> >>>
> >>> Will do, thanks for letting me know.
> >>
> >> I have confirmed with the initial reporter (internal test team) that <driver name='qemu'/> 
> >> with a known to be broken linux next kernel also fixes the problem, so it is really the
> >> vhost changes.
> > 
> > OK I'm back and trying to make it more bisectable.
> > 
> > I pushed a new tag "batch-v2".
> > It's same code but with this bisect should get more information.
> 
> I get the following with this tag
> 
> drivers/vhost/net.c: In function ‘vhost_net_tx_get_vq_desc’:
> drivers/vhost/net.c:574:7: error: implicit declaration of function ‘vhost_get_vq_desc_batch’; did you mean ‘vhost_get_vq_desc’? [-Werror=implicit-function-declaration]
>   574 |   r = vhost_get_vq_desc_batch(tvq, tvq->iov, ARRAY_SIZE(tvq->iov),
>       |       ^~~~~~~~~~~~~~~~~~~~~~~
>       |       vhost_get_vq_desc
> 

Not sure why but I pushed a wrong commit. Sorry. Should be good now.

-- 
MST

Re: vhost changes (batched) in linux-next after 12/13 trigger random crashes in KVM guests after reboot

[Christian Borntraeger]

On 07.01.20 10:39, Michael S. Tsirkin wrote:
> On Tue, Jan 07, 2020 at 09:59:16AM +0100, Christian Borntraeger wrote:
>>
>>
>> On 06.01.20 11:50, Michael S. Tsirkin wrote:
>>> On Wed, Dec 18, 2019 at 04:59:02PM +0100, Christian Borntraeger wrote:
>>>> On 18.12.19 16:10, Michael S. Tsirkin wrote:
>>>>> On Wed, Dec 18, 2019 at 03:43:43PM +0100, Christian Borntraeger wrote:
>>>>>> Michael,
>>>>>>
>>>>>> with 
>>>>>> commit db7286b100b503ef80612884453bed53d74c9a16 (refs/bisect/skip-db7286b100b503ef80612884453bed53d74c9a16)
>>>>>>     vhost: use batched version by default
>>>>>> plus
>>>>>> commit 6bd262d5eafcdf8cdfae491e2e748e4e434dcda6 (HEAD, refs/bisect/bad)
>>>>>>     Revert "vhost/net: add an option to test new code"
>>>>>> to make things compile (your next tree is not easily bisectable, can you fix that as well?).
>>>>>
>>>>> I'll try.
>>>>>
>>>>>>
>>>>>> I get random crashes in my s390 KVM guests after reboot.
>>>>>> Reverting both patches together with commit decd9b8 "vhost: use vhost_desc instead of vhost_log" to
>>>>>> make it compile again) on top of linux-next-1218 makes the problem go away.
>>>>>>
>>>>>> Looks like the batched version is not yet ready for prime time. Can you drop these patches until
>>>>>> we have fixed the issues?
>>>>>>
>>>>>> Christian
>>>>>>
>>>>>
>>>>> Will do, thanks for letting me know.
>>>>
>>>> I have confirmed with the initial reporter (internal test team) that <driver name='qemu'/> 
>>>> with a known to be broken linux next kernel also fixes the problem, so it is really the
>>>> vhost changes.
>>>
>>> OK I'm back and trying to make it more bisectable.
>>>
>>> I pushed a new tag "batch-v2".
>>> It's same code but with this bisect should get more information.
>>
>> I get the following with this tag
>>
>> drivers/vhost/net.c: In function ‘vhost_net_tx_get_vq_desc’:
>> drivers/vhost/net.c:574:7: error: implicit declaration of function ‘vhost_get_vq_desc_batch’; did you mean ‘vhost_get_vq_desc’? [-Werror=implicit-function-declaration]
>>   574 |   r = vhost_get_vq_desc_batch(tvq, tvq->iov, ARRAY_SIZE(tvq->iov),
>>       |       ^~~~~~~~~~~~~~~~~~~~~~~
>>       |       vhost_get_vq_desc
>>
> 
> Not sure why but I pushed a wrong commit. Sorry. Should be good now.
> 

during bisect:

drivers/vhost/vhost.c: In function ‘vhost_get_vq_desc_batch’:
drivers/vhost/vhost.c:2634:8: error: ‘id’ undeclared (first use in this function); did you mean ‘i’?
 2634 |  ret = id;
      |        ^~
      |        i

I changed that to i


The last step then gave me  (on commit 50297a8480b439efc5f3f23088cb2d90b799acef vhost: use batched version by default)
net enc1: Unexpected TXQ (0) queue failure: -5
in the guest.

bisect log so far:
[cborntra@m83lp52 linux]$ git bisect log
git bisect start
# bad: [3131e79bb9e9892a5a6bd33513de9bc90b20e867] vhost: use vhost_desc instead of vhost_log
git bisect bad 3131e79bb9e9892a5a6bd33513de9bc90b20e867
# good: [d1281e3a562ec6a08f944a876481dd043ba739b9] virtio-blk: remove VIRTIO_BLK_F_SCSI support
git bisect good d1281e3a562ec6a08f944a876481dd043ba739b9
# good: [5b00aab5b6332a67e32dace1dcd3a198ab94ed56] vhost: option to fetch descriptors through an independent struct
git bisect good 5b00aab5b6332a67e32dace1dcd3a198ab94ed56
# good: [5b00aab5b6332a67e32dace1dcd3a198ab94ed56] vhost: option to fetch descriptors through an independent struct
git bisect good 5b00aab5b6332a67e32dace1dcd3a198ab94ed56
# bad: [1414d7ee3d10d2ec2bc4ee652d1d90ec91da1c79] vhost: batching fetches
git bisect bad 1414d7ee3d10d2ec2bc4ee652d1d90ec91da1c79

Re: vhost changes (batched) in linux-next after 12/13 trigger random crashes in KVM guests after reboot

[Michael S. Tsirkin]

On Tue, Jan 07, 2020 at 12:34:50PM +0100, Christian Borntraeger wrote:
> 
> 
> On 07.01.20 10:39, Michael S. Tsirkin wrote:
> > On Tue, Jan 07, 2020 at 09:59:16AM +0100, Christian Borntraeger wrote:
> >>
> >>
> >> On 06.01.20 11:50, Michael S. Tsirkin wrote:
> >>> On Wed, Dec 18, 2019 at 04:59:02PM +0100, Christian Borntraeger wrote:
> >>>> On 18.12.19 16:10, Michael S. Tsirkin wrote:
> >>>>> On Wed, Dec 18, 2019 at 03:43:43PM +0100, Christian Borntraeger wrote:
> >>>>>> Michael,
> >>>>>>
> >>>>>> with 
> >>>>>> commit db7286b100b503ef80612884453bed53d74c9a16 (refs/bisect/skip-db7286b100b503ef80612884453bed53d74c9a16)
> >>>>>>     vhost: use batched version by default
> >>>>>> plus
> >>>>>> commit 6bd262d5eafcdf8cdfae491e2e748e4e434dcda6 (HEAD, refs/bisect/bad)
> >>>>>>     Revert "vhost/net: add an option to test new code"
> >>>>>> to make things compile (your next tree is not easily bisectable, can you fix that as well?).
> >>>>>
> >>>>> I'll try.
> >>>>>
> >>>>>>
> >>>>>> I get random crashes in my s390 KVM guests after reboot.
> >>>>>> Reverting both patches together with commit decd9b8 "vhost: use vhost_desc instead of vhost_log" to
> >>>>>> make it compile again) on top of linux-next-1218 makes the problem go away.
> >>>>>>
> >>>>>> Looks like the batched version is not yet ready for prime time. Can you drop these patches until
> >>>>>> we have fixed the issues?
> >>>>>>
> >>>>>> Christian
> >>>>>>
> >>>>>
> >>>>> Will do, thanks for letting me know.
> >>>>
> >>>> I have confirmed with the initial reporter (internal test team) that <driver name='qemu'/> 
> >>>> with a known to be broken linux next kernel also fixes the problem, so it is really the
> >>>> vhost changes.
> >>>
> >>> OK I'm back and trying to make it more bisectable.
> >>>
> >>> I pushed a new tag "batch-v2".
> >>> It's same code but with this bisect should get more information.
> >>
> >> I get the following with this tag
> >>
> >> drivers/vhost/net.c: In function ‘vhost_net_tx_get_vq_desc’:
> >> drivers/vhost/net.c:574:7: error: implicit declaration of function ‘vhost_get_vq_desc_batch’; did you mean ‘vhost_get_vq_desc’? [-Werror=implicit-function-declaration]
> >>   574 |   r = vhost_get_vq_desc_batch(tvq, tvq->iov, ARRAY_SIZE(tvq->iov),
> >>       |       ^~~~~~~~~~~~~~~~~~~~~~~
> >>       |       vhost_get_vq_desc
> >>
> > 
> > Not sure why but I pushed a wrong commit. Sorry. Should be good now.
> > 
> 
> during bisect:
> 
> drivers/vhost/vhost.c: In function ‘vhost_get_vq_desc_batch’:
> drivers/vhost/vhost.c:2634:8: error: ‘id’ undeclared (first use in this function); did you mean ‘i’?
>  2634 |  ret = id;
>       |        ^~
>       |        i
> 
> I changed that to i

Hmm no that's wrong I think. Sorry about all the errors. Let me push a
fixed v3.

> 
> The last step then gave me  (on commit 50297a8480b439efc5f3f23088cb2d90b799acef vhost: use batched version by default)
> net enc1: Unexpected TXQ (0) queue failure: -5
> in the guest.
> 
> bisect log so far:
> [cborntra@m83lp52 linux]$ git bisect log
> git bisect start
> # bad: [3131e79bb9e9892a5a6bd33513de9bc90b20e867] vhost: use vhost_desc instead of vhost_log
> git bisect bad 3131e79bb9e9892a5a6bd33513de9bc90b20e867
> # good: [d1281e3a562ec6a08f944a876481dd043ba739b9] virtio-blk: remove VIRTIO_BLK_F_SCSI support
> git bisect good d1281e3a562ec6a08f944a876481dd043ba739b9
> # good: [5b00aab5b6332a67e32dace1dcd3a198ab94ed56] vhost: option to fetch descriptors through an independent struct
> git bisect good 5b00aab5b6332a67e32dace1dcd3a198ab94ed56
> # good: [5b00aab5b6332a67e32dace1dcd3a198ab94ed56] vhost: option to fetch descriptors through an independent struct
> git bisect good 5b00aab5b6332a67e32dace1dcd3a198ab94ed56
> # bad: [1414d7ee3d10d2ec2bc4ee652d1d90ec91da1c79] vhost: batching fetches
> git bisect bad 1414d7ee3d10d2ec2bc4ee652d1d90ec91da1c79
> 
> 
> 

Re: vhost changes (batched) in linux-next after 12/13 trigger random crashes in KVM guests after reboot

[Michael S. Tsirkin]

On Tue, Jan 07, 2020 at 12:34:50PM +0100, Christian Borntraeger wrote:
> 
> 
> On 07.01.20 10:39, Michael S. Tsirkin wrote:
> > On Tue, Jan 07, 2020 at 09:59:16AM +0100, Christian Borntraeger wrote:
> >>
> >>
> >> On 06.01.20 11:50, Michael S. Tsirkin wrote:
> >>> On Wed, Dec 18, 2019 at 04:59:02PM +0100, Christian Borntraeger wrote:
> >>>> On 18.12.19 16:10, Michael S. Tsirkin wrote:
> >>>>> On Wed, Dec 18, 2019 at 03:43:43PM +0100, Christian Borntraeger wrote:
> >>>>>> Michael,
> >>>>>>
> >>>>>> with 
> >>>>>> commit db7286b100b503ef80612884453bed53d74c9a16 (refs/bisect/skip-db7286b100b503ef80612884453bed53d74c9a16)
> >>>>>>     vhost: use batched version by default
> >>>>>> plus
> >>>>>> commit 6bd262d5eafcdf8cdfae491e2e748e4e434dcda6 (HEAD, refs/bisect/bad)
> >>>>>>     Revert "vhost/net: add an option to test new code"
> >>>>>> to make things compile (your next tree is not easily bisectable, can you fix that as well?).
> >>>>>
> >>>>> I'll try.
> >>>>>
> >>>>>>
> >>>>>> I get random crashes in my s390 KVM guests after reboot.
> >>>>>> Reverting both patches together with commit decd9b8 "vhost: use vhost_desc instead of vhost_log" to
> >>>>>> make it compile again) on top of linux-next-1218 makes the problem go away.
> >>>>>>
> >>>>>> Looks like the batched version is not yet ready for prime time. Can you drop these patches until
> >>>>>> we have fixed the issues?
> >>>>>>
> >>>>>> Christian
> >>>>>>
> >>>>>
> >>>>> Will do, thanks for letting me know.
> >>>>
> >>>> I have confirmed with the initial reporter (internal test team) that <driver name='qemu'/> 
> >>>> with a known to be broken linux next kernel also fixes the problem, so it is really the
> >>>> vhost changes.
> >>>
> >>> OK I'm back and trying to make it more bisectable.
> >>>
> >>> I pushed a new tag "batch-v2".
> >>> It's same code but with this bisect should get more information.
> >>
> >> I get the following with this tag
> >>
> >> drivers/vhost/net.c: In function ‘vhost_net_tx_get_vq_desc’:
> >> drivers/vhost/net.c:574:7: error: implicit declaration of function ‘vhost_get_vq_desc_batch’; did you mean ‘vhost_get_vq_desc’? [-Werror=implicit-function-declaration]
> >>   574 |   r = vhost_get_vq_desc_batch(tvq, tvq->iov, ARRAY_SIZE(tvq->iov),
> >>       |       ^~~~~~~~~~~~~~~~~~~~~~~
> >>       |       vhost_get_vq_desc
> >>
> > 
> > Not sure why but I pushed a wrong commit. Sorry. Should be good now.
> > 
> 
> during bisect:
> 
> drivers/vhost/vhost.c: In function ‘vhost_get_vq_desc_batch’:
> drivers/vhost/vhost.c:2634:8: error: ‘id’ undeclared (first use in this function); did you mean ‘i’?
>  2634 |  ret = id;
>       |        ^~
>       |        i
> 
> I changed that to i
> 
> 
> The last step then gave me  (on commit 50297a8480b439efc5f3f23088cb2d90b799acef vhost: use batched version by default)
> net enc1: Unexpected TXQ (0) queue failure: -5
> in the guest.
> 
> bisect log so far:
> [cborntra@m83lp52 linux]$ git bisect log
> git bisect start
> # bad: [3131e79bb9e9892a5a6bd33513de9bc90b20e867] vhost: use vhost_desc instead of vhost_log
> git bisect bad 3131e79bb9e9892a5a6bd33513de9bc90b20e867
> # good: [d1281e3a562ec6a08f944a876481dd043ba739b9] virtio-blk: remove VIRTIO_BLK_F_SCSI support
> git bisect good d1281e3a562ec6a08f944a876481dd043ba739b9
> # good: [5b00aab5b6332a67e32dace1dcd3a198ab94ed56] vhost: option to fetch descriptors through an independent struct
> git bisect good 5b00aab5b6332a67e32dace1dcd3a198ab94ed56
> # good: [5b00aab5b6332a67e32dace1dcd3a198ab94ed56] vhost: option to fetch descriptors through an independent struct
> git bisect good 5b00aab5b6332a67e32dace1dcd3a198ab94ed56
> # bad: [1414d7ee3d10d2ec2bc4ee652d1d90ec91da1c79] vhost: batching fetches
> git bisect bad 1414d7ee3d10d2ec2bc4ee652d1d90ec91da1c79
> 
> 

I pushed batched-v3 - same head but bisect should work now.

-- 
MST

Re: vhost changes (batched) in linux-next after 12/13 trigger random crashes in KVM guests after reboot

[Christian Borntraeger]

On 07.01.20 12:55, Michael S. Tsirkin wrote:

> 
> I pushed batched-v3 - same head but bisect should work now.
> 

With 
commit 38ced0208491103b50f1056f0d1c8f28e2e13d08 (HEAD)
Author:     Michael S. Tsirkin <mst@redhat.com>
AuthorDate: Wed Dec 11 12:19:26 2019 -0500
Commit:     Michael S. Tsirkin <mst@redhat.com>
CommitDate: Tue Jan 7 06:52:42 2020 -0500

    vhost: use batched version by default


I have exactly one successful ping and then the network inside the guest is broken (no packet
anymore).

So you could consider this commit broken (but in a different way and also without any
guest reboot necessary).


bisect log:
git bisect start
# bad: [d2f6175f52062ee51ee69754a6925608213475d2] vhost: use vhost_desc instead of vhost_log
git bisect bad d2f6175f52062ee51ee69754a6925608213475d2
# good: [d1281e3a562ec6a08f944a876481dd043ba739b9] virtio-blk: remove VIRTIO_BLK_F_SCSI support
git bisect good d1281e3a562ec6a08f944a876481dd043ba739b9
# good: [fac7c0f46996e32d996f5c46121df24a6b95ec3b] vhost: option to fetch descriptors through an independent struct
git bisect good fac7c0f46996e32d996f5c46121df24a6b95ec3b
# bad: [539eb9d738f048cd7be61f404e8f9c7d9d2ff3cc] vhost: batching fetches
git bisect bad 539eb9d738f048cd7be61f404e8f9c7d9d2ff3cc

Re: vhost changes (batched) in linux-next after 12/13 trigger random crashes in KVM guests after reboot

[Michael S. Tsirkin]

On Tue, Jan 07, 2020 at 01:16:50PM +0100, Christian Borntraeger wrote:
> On 07.01.20 12:55, Michael S. Tsirkin wrote:
> 
> > 
> > I pushed batched-v3 - same head but bisect should work now.
> > 
> 
> With 
> commit 38ced0208491103b50f1056f0d1c8f28e2e13d08 (HEAD)
> Author:     Michael S. Tsirkin <mst@redhat.com>
> AuthorDate: Wed Dec 11 12:19:26 2019 -0500
> Commit:     Michael S. Tsirkin <mst@redhat.com>
> CommitDate: Tue Jan 7 06:52:42 2020 -0500
> 
>     vhost: use batched version by default
> 
> 
> I have exactly one successful ping and then the network inside the guest is broken (no packet
> anymore).

Does anything appear in host's dmesg when this happens?


> So you could consider this commit broken (but in a different way and also without any
> guest reboot necessary).
> 
> 
> bisect log:
> git bisect start
> # bad: [d2f6175f52062ee51ee69754a6925608213475d2] vhost: use vhost_desc instead of vhost_log
> git bisect bad d2f6175f52062ee51ee69754a6925608213475d2
> # good: [d1281e3a562ec6a08f944a876481dd043ba739b9] virtio-blk: remove VIRTIO_BLK_F_SCSI support
> git bisect good d1281e3a562ec6a08f944a876481dd043ba739b9
> # good: [fac7c0f46996e32d996f5c46121df24a6b95ec3b] vhost: option to fetch descriptors through an independent struct
> git bisect good fac7c0f46996e32d996f5c46121df24a6b95ec3b
> # bad: [539eb9d738f048cd7be61f404e8f9c7d9d2ff3cc] vhost: batching fetches
> git bisect bad 539eb9d738f048cd7be61f404e8f9c7d9d2ff3cc

Re: vhost changes (batched) in linux-next after 12/13 trigger random crashes in KVM guests after reboot

[Christian Borntraeger]

On 20.01.20 07:27, Michael S. Tsirkin wrote:
> On Tue, Jan 07, 2020 at 01:16:50PM +0100, Christian Borntraeger wrote:
>> On 07.01.20 12:55, Michael S. Tsirkin wrote:
>>
>>>
>>> I pushed batched-v3 - same head but bisect should work now.
>>>
>>
>> With 
>> commit 38ced0208491103b50f1056f0d1c8f28e2e13d08 (HEAD)
>> Author:     Michael S. Tsirkin <mst@redhat.com>
>> AuthorDate: Wed Dec 11 12:19:26 2019 -0500
>> Commit:     Michael S. Tsirkin <mst@redhat.com>
>> CommitDate: Tue Jan 7 06:52:42 2020 -0500
>>
>>     vhost: use batched version by default
>>
>>
>> I have exactly one successful ping and then the network inside the guest is broken (no packet
>> anymore).
> 
> Does anything appear in host's dmesg when this happens?

I think there was nothing, but I am not sure. I would need to redo the test if this is important to know.

> 
> 
>> So you could consider this commit broken (but in a different way and also without any
>> guest reboot necessary).
>>
>>
>> bisect log:
>> git bisect start
>> # bad: [d2f6175f52062ee51ee69754a6925608213475d2] vhost: use vhost_desc instead of vhost_log
>> git bisect bad d2f6175f52062ee51ee69754a6925608213475d2
>> # good: [d1281e3a562ec6a08f944a876481dd043ba739b9] virtio-blk: remove VIRTIO_BLK_F_SCSI support
>> git bisect good d1281e3a562ec6a08f944a876481dd043ba739b9
>> # good: [fac7c0f46996e32d996f5c46121df24a6b95ec3b] vhost: option to fetch descriptors through an independent struct
>> git bisect good fac7c0f46996e32d996f5c46121df24a6b95ec3b
>> # bad: [539eb9d738f048cd7be61f404e8f9c7d9d2ff3cc] vhost: batching fetches
>> git bisect bad 539eb9d738f048cd7be61f404e8f9c7d9d2ff3cc
> 

Re: vhost changes (batched) in linux-next after 12/13 trigger random crashes in KVM guests after reboot

[eperezma]

Hi Christian.

Could you try this patch on top of ("38ced0208491 vhost: use batched version by default")?

It will not solve your first random crash but it should help with the lost of network connectivity.

Please let me know how does it goes.

Thanks!

From 99f0f543f3939dbe803988c9153a95616ccccacd Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Eugenio=20P=C3=A9rez?= <eperezma@redhat.com>
Date: Thu, 6 Feb 2020 15:13:42 +0100
Subject: [PATCH] vhost: filter valid vhost descriptors flags
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

Previous commit copy _NEXT flag, and it complains if a copied descriptor
contains it.

Signed-off-by: Eugenio Pérez <eperezma@redhat.com>
---
 drivers/vhost/vhost.c | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)

diff --git a/drivers/vhost/vhost.c b/drivers/vhost/vhost.c
index 27ae5b4872a0..56c5253056ee 100644
--- a/drivers/vhost/vhost.c
+++ b/drivers/vhost/vhost.c
@@ -2125,6 +2125,8 @@ static void pop_split_desc(struct vhost_virtqueue *vq)
 	--vq->ndescs;
 }
 
+#define VHOST_DESC_FLAGS (VRING_DESC_F_INDIRECT | VRING_DESC_F_WRITE | \
+			  VRING_DESC_F_NEXT)
 static int push_split_desc(struct vhost_virtqueue *vq, struct vring_desc *desc, u16 id)
 {
 	struct vhost_desc *h;
@@ -2134,7 +2136,7 @@ static int push_split_desc(struct vhost_virtqueue *vq, struct vring_desc *desc,
 	h = &vq->descs[vq->ndescs++];
 	h->addr = vhost64_to_cpu(vq, desc->addr);
 	h->len = vhost32_to_cpu(vq, desc->len);
-	h->flags = vhost16_to_cpu(vq, desc->flags);
+	h->flags = vhost16_to_cpu(vq, desc->flags) & VHOST_DESC_FLAGS;
 	h->id = id;
 
 	return 0;
@@ -2343,7 +2345,7 @@ int vhost_get_vq_desc(struct vhost_virtqueue *vq,
 		struct vhost_desc *desc = &vq->descs[i];
 		int access;
 
-		if (desc->flags & ~(VRING_DESC_F_INDIRECT | VRING_DESC_F_WRITE)) {
+		if (desc->flags & ~VHOST_DESC_FLAGS) {
 			vq_err(vq, "Unexpected flags: 0x%x at descriptor id 0x%x\n",
 			       desc->flags, desc->id);
 			ret = -EINVAL;
-- 
2.18.1


On Wed, 2020-01-22 at 20:32 +0100, Christian Borntraeger wrote:
> 
> On 20.01.20 07:27, Michael S. Tsirkin wrote:
> > On Tue, Jan 07, 2020 at 01:16:50PM +0100, Christian Borntraeger wrote:
> > > On 07.01.20 12:55, Michael S. Tsirkin wrote:
> > > 
> > > > I pushed batched-v3 - same head but bisect should work now.
> > > > 
> > > 
> > > With 
> > > commit 38ced0208491103b50f1056f0d1c8f28e2e13d08 (HEAD)
> > > Author:     Michael S. Tsirkin <mst@redhat.com>
> > > AuthorDate: Wed Dec 11 12:19:26 2019 -0500
> > > Commit:     Michael S. Tsirkin <mst@redhat.com>
> > > CommitDate: Tue Jan 7 06:52:42 2020 -0500
> > > 
> > >     vhost: use batched version by default
> > > 
> > > 
> > > I have exactly one successful ping and then the network inside the guest is broken (no packet
> > > anymore).
> > 
> > Does anything appear in host's dmesg when this happens?
> 
> I think there was nothing, but I am not sure. I would need to redo the test if this is important to know.
> 
> > 
> > > So you could consider this commit broken (but in a different way and also without any
> > > guest reboot necessary).
> > > 
> > > 
> > > bisect log:
> > > git bisect start
> > > # bad: [d2f6175f52062ee51ee69754a6925608213475d2] vhost: use vhost_desc instead of vhost_log
> > > git bisect bad d2f6175f52062ee51ee69754a6925608213475d2
> > > # good: [d1281e3a562ec6a08f944a876481dd043ba739b9] virtio-blk: remove VIRTIO_BLK_F_SCSI support
> > > git bisect good d1281e3a562ec6a08f944a876481dd043ba739b9
> > > # good: [fac7c0f46996e32d996f5c46121df24a6b95ec3b] vhost: option to fetch descriptors through an independent
> > > struct
> > > git bisect good fac7c0f46996e32d996f5c46121df24a6b95ec3b
> > > # bad: [539eb9d738f048cd7be61f404e8f9c7d9d2ff3cc] vhost: batching fetches
> > > git bisect bad 539eb9d738f048cd7be61f404e8f9c7d9d2ff3cc

Re: vhost changes (batched) in linux-next after 12/13 trigger random crashes in KVM guests after reboot

[Christian Borntraeger]

On 06.02.20 15:22, eperezma@redhat.com wrote:
> Hi Christian.
> 
> Could you try this patch on top of ("38ced0208491 vhost: use batched version by default")?
> 
> It will not solve your first random crash but it should help with the lost of network connectivity.
> 
> Please let me know how does it goes.


38ced0208491 + this seem to be ok.

Not sure if you can make out anything of this (and the previous git bisect log)

Re: vhost changes (batched) in linux-next after 12/13 trigger random crashes in KVM guests after reboot

[Michael S. Tsirkin]

On Thu, Feb 06, 2020 at 04:12:21PM +0100, Christian Borntraeger wrote:
> 
> 
> On 06.02.20 15:22, eperezma@redhat.com wrote:
> > Hi Christian.
> > 
> > Could you try this patch on top of ("38ced0208491 vhost: use batched version by default")?
> > 
> > It will not solve your first random crash but it should help with the lost of network connectivity.
> > 
> > Please let me know how does it goes.
> 
> 
> 38ced0208491 + this seem to be ok.
> 
> Not sure if you can make out anything of this (and the previous git bisect log)

Yes it does - that this is just bad split-up of patches, and there's
still a real bug that caused worse crashes :)

So I just pushed batch-v4.
I expect that will fail, and bisect to give us
    vhost: batching fetches
Can you try that please?

Re: vhost changes (batched) in linux-next after 12/13 trigger random crashes in KVM guests after reboot

[Christian Borntraeger]

Also adding Cornelia.


On 06.02.20 23:17, Michael S. Tsirkin wrote:
> On Thu, Feb 06, 2020 at 04:12:21PM +0100, Christian Borntraeger wrote:
>>
>>
>> On 06.02.20 15:22, eperezma@redhat.com wrote:
>>> Hi Christian.
>>>
>>> Could you try this patch on top of ("38ced0208491 vhost: use batched version by default")?
>>>
>>> It will not solve your first random crash but it should help with the lost of network connectivity.
>>>
>>> Please let me know how does it goes.
>>
>>
>> 38ced0208491 + this seem to be ok.
>>
>> Not sure if you can make out anything of this (and the previous git bisect log)
> 
> Yes it does - that this is just bad split-up of patches, and there's
> still a real bug that caused worse crashes :)
> 
> So I just pushed batch-v4.
> I expect that will fail, and bisect to give us
>     vhost: batching fetches
> Can you try that please?
> 

yes.

eccb852f1fe6bede630e2e4f1a121a81e34354ab is the first bad commit
commit eccb852f1fe6bede630e2e4f1a121a81e34354ab
Author: Michael S. Tsirkin <mst@redhat.com>
Date:   Mon Oct 7 06:11:18 2019 -0400

    vhost: batching fetches
    
    With this patch applied, new and old code perform identically.
    
    Lots of extra optimizations are now possible, e.g.
    we can fetch multiple heads with copy_from/to_user now.
    We can get rid of maintaining the log array.  Etc etc.
    
    Signed-off-by: Michael S. Tsirkin <mst@redhat.com>

 drivers/vhost/test.c  |  2 +-
 drivers/vhost/vhost.c | 39 ++++++++++++++++++++++++++++++++++-----
 drivers/vhost/vhost.h |  4 +++-
 3 files changed, 38 insertions(+), 7 deletions(-)


> 
			

Re: vhost changes (batched) in linux-next after 12/13 trigger random crashes in KVM guests after reboot

[Michael S. Tsirkin]

On Fri, Feb 07, 2020 at 08:47:14AM +0100, Christian Borntraeger wrote:
> Also adding Cornelia.
> 
> 
> On 06.02.20 23:17, Michael S. Tsirkin wrote:
> > On Thu, Feb 06, 2020 at 04:12:21PM +0100, Christian Borntraeger wrote:
> >>
> >>
> >> On 06.02.20 15:22, eperezma@redhat.com wrote:
> >>> Hi Christian.
> >>>
> >>> Could you try this patch on top of ("38ced0208491 vhost: use batched version by default")?
> >>>
> >>> It will not solve your first random crash but it should help with the lost of network connectivity.
> >>>
> >>> Please let me know how does it goes.
> >>
> >>
> >> 38ced0208491 + this seem to be ok.
> >>
> >> Not sure if you can make out anything of this (and the previous git bisect log)
> > 
> > Yes it does - that this is just bad split-up of patches, and there's
> > still a real bug that caused worse crashes :)
> > 
> > So I just pushed batch-v4.
> > I expect that will fail, and bisect to give us
> >     vhost: batching fetches
> > Can you try that please?
> > 
> 
> yes.
> 
> eccb852f1fe6bede630e2e4f1a121a81e34354ab is the first bad commit
> commit eccb852f1fe6bede630e2e4f1a121a81e34354ab
> Author: Michael S. Tsirkin <mst@redhat.com>
> Date:   Mon Oct 7 06:11:18 2019 -0400
> 
>     vhost: batching fetches
>     
>     With this patch applied, new and old code perform identically.
>     
>     Lots of extra optimizations are now possible, e.g.
>     we can fetch multiple heads with copy_from/to_user now.
>     We can get rid of maintaining the log array.  Etc etc.
>     
>     Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
> 
>  drivers/vhost/test.c  |  2 +-
>  drivers/vhost/vhost.c | 39 ++++++++++++++++++++++++++++++++++-----
>  drivers/vhost/vhost.h |  4 +++-
>  3 files changed, 38 insertions(+), 7 deletions(-)
> 


And the symptom is still the same - random crashes
after a bit of traffic, right?

> > 
> 			

Re: vhost changes (batched) in linux-next after 12/13 trigger random crashes in KVM guests after reboot

[Christian Borntraeger]

On 07.02.20 08:58, Michael S. Tsirkin wrote:
> On Fri, Feb 07, 2020 at 08:47:14AM +0100, Christian Borntraeger wrote:
>> Also adding Cornelia.
>>
>>
>> On 06.02.20 23:17, Michael S. Tsirkin wrote:
>>> On Thu, Feb 06, 2020 at 04:12:21PM +0100, Christian Borntraeger wrote:
>>>>
>>>>
>>>> On 06.02.20 15:22, eperezma@redhat.com wrote:
>>>>> Hi Christian.
>>>>>
>>>>> Could you try this patch on top of ("38ced0208491 vhost: use batched version by default")?
>>>>>
>>>>> It will not solve your first random crash but it should help with the lost of network connectivity.
>>>>>
>>>>> Please let me know how does it goes.
>>>>
>>>>
>>>> 38ced0208491 + this seem to be ok.
>>>>
>>>> Not sure if you can make out anything of this (and the previous git bisect log)
>>>
>>> Yes it does - that this is just bad split-up of patches, and there's
>>> still a real bug that caused worse crashes :)
>>>
>>> So I just pushed batch-v4.
>>> I expect that will fail, and bisect to give us
>>>     vhost: batching fetches
>>> Can you try that please?
>>>
>>
>> yes.
>>
>> eccb852f1fe6bede630e2e4f1a121a81e34354ab is the first bad commit
>> commit eccb852f1fe6bede630e2e4f1a121a81e34354ab
>> Author: Michael S. Tsirkin <mst@redhat.com>
>> Date:   Mon Oct 7 06:11:18 2019 -0400
>>
>>     vhost: batching fetches
>>     
>>     With this patch applied, new and old code perform identically.
>>     
>>     Lots of extra optimizations are now possible, e.g.
>>     we can fetch multiple heads with copy_from/to_user now.
>>     We can get rid of maintaining the log array.  Etc etc.
>>     
>>     Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
>>
>>  drivers/vhost/test.c  |  2 +-
>>  drivers/vhost/vhost.c | 39 ++++++++++++++++++++++++++++++++++-----
>>  drivers/vhost/vhost.h |  4 +++-
>>  3 files changed, 38 insertions(+), 7 deletions(-)
>>
> 
> 
> And the symptom is still the same - random crashes
> after a bit of traffic, right?

random guest crashes after a reboot of the guests. As if vhost would still
write into now stale buffers.

Re: vhost changes (batched) in linux-next after 12/13 trigger random crashes in KVM guests after reboot

[Cornelia Huck]

On Fri, 7 Feb 2020 09:13:14 +0100
Christian Borntraeger <borntraeger@de.ibm.com> wrote:

> On 07.02.20 08:58, Michael S. Tsirkin wrote:
> > On Fri, Feb 07, 2020 at 08:47:14AM +0100, Christian Borntraeger wrote:  
> >> Also adding Cornelia.
> >>
> >>
> >> On 06.02.20 23:17, Michael S. Tsirkin wrote:  
> >>> On Thu, Feb 06, 2020 at 04:12:21PM +0100, Christian Borntraeger wrote:  
> >>>>
> >>>>
> >>>> On 06.02.20 15:22, eperezma@redhat.com wrote:  
> >>>>> Hi Christian.
> >>>>>
> >>>>> Could you try this patch on top of ("38ced0208491 vhost: use batched version by default")?
> >>>>>
> >>>>> It will not solve your first random crash but it should help with the lost of network connectivity.
> >>>>>
> >>>>> Please let me know how does it goes.  
> >>>>
> >>>>
> >>>> 38ced0208491 + this seem to be ok.
> >>>>
> >>>> Not sure if you can make out anything of this (and the previous git bisect log)  
> >>>
> >>> Yes it does - that this is just bad split-up of patches, and there's
> >>> still a real bug that caused worse crashes :)
> >>>
> >>> So I just pushed batch-v4.
> >>> I expect that will fail, and bisect to give us
> >>>     vhost: batching fetches
> >>> Can you try that please?
> >>>  
> >>
> >> yes.
> >>
> >> eccb852f1fe6bede630e2e4f1a121a81e34354ab is the first bad commit
> >> commit eccb852f1fe6bede630e2e4f1a121a81e34354ab
> >> Author: Michael S. Tsirkin <mst@redhat.com>
> >> Date:   Mon Oct 7 06:11:18 2019 -0400
> >>
> >>     vhost: batching fetches
> >>     
> >>     With this patch applied, new and old code perform identically.
> >>     
> >>     Lots of extra optimizations are now possible, e.g.
> >>     we can fetch multiple heads with copy_from/to_user now.
> >>     We can get rid of maintaining the log array.  Etc etc.
> >>     
> >>     Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
> >>
> >>  drivers/vhost/test.c  |  2 +-
> >>  drivers/vhost/vhost.c | 39 ++++++++++++++++++++++++++++++++++-----
> >>  drivers/vhost/vhost.h |  4 +++-
> >>  3 files changed, 38 insertions(+), 7 deletions(-)
> >>  
> > 
> > 
> > And the symptom is still the same - random crashes
> > after a bit of traffic, right?  
> 
> random guest crashes after a reboot of the guests. As if vhost would still
> write into now stale buffers.
> 

I'm late to the party; but where is that commit located? Or has it been
dropped again already?

Re: vhost changes (batched) in linux-next after 12/13 trigger random crashes in KVM guests after reboot

[Michael S. Tsirkin]

On Fri, Feb 07, 2020 at 09:53:53AM +0100, Cornelia Huck wrote:
> On Fri, 7 Feb 2020 09:13:14 +0100
> Christian Borntraeger <borntraeger@de.ibm.com> wrote:
> 
> > On 07.02.20 08:58, Michael S. Tsirkin wrote:
> > > On Fri, Feb 07, 2020 at 08:47:14AM +0100, Christian Borntraeger wrote:  
> > >> Also adding Cornelia.
> > >>
> > >>
> > >> On 06.02.20 23:17, Michael S. Tsirkin wrote:  
> > >>> On Thu, Feb 06, 2020 at 04:12:21PM +0100, Christian Borntraeger wrote:  
> > >>>>
> > >>>>
> > >>>> On 06.02.20 15:22, eperezma@redhat.com wrote:  
> > >>>>> Hi Christian.
> > >>>>>
> > >>>>> Could you try this patch on top of ("38ced0208491 vhost: use batched version by default")?
> > >>>>>
> > >>>>> It will not solve your first random crash but it should help with the lost of network connectivity.
> > >>>>>
> > >>>>> Please let me know how does it goes.  
> > >>>>
> > >>>>
> > >>>> 38ced0208491 + this seem to be ok.
> > >>>>
> > >>>> Not sure if you can make out anything of this (and the previous git bisect log)  
> > >>>
> > >>> Yes it does - that this is just bad split-up of patches, and there's
> > >>> still a real bug that caused worse crashes :)
> > >>>
> > >>> So I just pushed batch-v4.
> > >>> I expect that will fail, and bisect to give us
> > >>>     vhost: batching fetches
> > >>> Can you try that please?
> > >>>  
> > >>
> > >> yes.
> > >>
> > >> eccb852f1fe6bede630e2e4f1a121a81e34354ab is the first bad commit
> > >> commit eccb852f1fe6bede630e2e4f1a121a81e34354ab
> > >> Author: Michael S. Tsirkin <mst@redhat.com>
> > >> Date:   Mon Oct 7 06:11:18 2019 -0400
> > >>
> > >>     vhost: batching fetches
> > >>     
> > >>     With this patch applied, new and old code perform identically.
> > >>     
> > >>     Lots of extra optimizations are now possible, e.g.
> > >>     we can fetch multiple heads with copy_from/to_user now.
> > >>     We can get rid of maintaining the log array.  Etc etc.
> > >>     
> > >>     Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
> > >>
> > >>  drivers/vhost/test.c  |  2 +-
> > >>  drivers/vhost/vhost.c | 39 ++++++++++++++++++++++++++++++++++-----
> > >>  drivers/vhost/vhost.h |  4 +++-
> > >>  3 files changed, 38 insertions(+), 7 deletions(-)
> > >>  
> > > 
> > > 
> > > And the symptom is still the same - random crashes
> > > after a bit of traffic, right?  
> > 
> > random guest crashes after a reboot of the guests. As if vhost would still
> > write into now stale buffers.
> > 
> 
> I'm late to the party; but where is that commit located? Or has it been
> dropped again already?

my vhost tree. Tag batch-v4.

Re: vhost changes (batched) in linux-next after 12/13 trigger random crashes in KVM guests after reboot

[Christian Borntraeger]

On 10.02.20 10:40, Eugenio Perez Martin wrote:
> Hi Christian.
> 
> I'm not able to reproduce the failure with eccb852f1fe6bede630e2e4f1a121a81e34354ab commit. Could you add more data? Your configuration (libvirt or qemu line), and host's dmesg output if any?

I do the following in the guest:
ping -c 200 -f somevalidip; reboot
sometimes I need to do that multiple times and sometimes I do not get a guest crash but host dmesg like

Guest moved used index from 0 to 292

xml is pretty simple

    <interface type='direct'>
      <mac address='52:54:00:7c:2c:f3'/>
      <source dev='encbd00' mode='bridge'/>
      <model type='virtio'/>
      <driver name='vhost'/>
      <address type='ccw' cssid='0xfe' ssid='0x0' devno='0x0001'/>
    </interface>


Reverting this patch seems to make both problems go away.


> 
> Thanks!
> 
> On Fri, Feb 7, 2020 at 9:13 AM Christian Borntraeger <borntraeger@de.ibm.com <mailto:borntraeger@de.ibm.com>> wrote:
> 
> 
> 
>     On 07.02.20 08:58, Michael S. Tsirkin wrote:
>     > On Fri, Feb 07, 2020 at 08:47:14AM +0100, Christian Borntraeger wrote:
>     >> Also adding Cornelia.
>     >>
>     >>
>     >> On 06.02.20 23:17, Michael S. Tsirkin wrote:
>     >>> On Thu, Feb 06, 2020 at 04:12:21PM +0100, Christian Borntraeger wrote:
>     >>>>
>     >>>>
>     >>>> On 06.02.20 15:22, eperezma@redhat.com <mailto:eperezma@redhat.com> wrote:
>     >>>>> Hi Christian.
>     >>>>>
>     >>>>> Could you try this patch on top of ("38ced0208491 vhost: use batched version by default")?
>     >>>>>
>     >>>>> It will not solve your first random crash but it should help with the lost of network connectivity.
>     >>>>>
>     >>>>> Please let me know how does it goes.
>     >>>>
>     >>>>
>     >>>> 38ced0208491 + this seem to be ok.
>     >>>>
>     >>>> Not sure if you can make out anything of this (and the previous git bisect log)
>     >>>
>     >>> Yes it does - that this is just bad split-up of patches, and there's
>     >>> still a real bug that caused worse crashes :)
>     >>>
>     >>> So I just pushed batch-v4.
>     >>> I expect that will fail, and bisect to give us
>     >>>     vhost: batching fetches
>     >>> Can you try that please?
>     >>>
>     >>
>     >> yes.
>     >>
>     >> eccb852f1fe6bede630e2e4f1a121a81e34354ab is the first bad commit
>     >> commit eccb852f1fe6bede630e2e4f1a121a81e34354ab
>     >> Author: Michael S. Tsirkin <mst@redhat.com <mailto:mst@redhat.com>>
>     >> Date:   Mon Oct 7 06:11:18 2019 -0400
>     >>
>     >>     vhost: batching fetches
>     >>     
>     >>     With this patch applied, new and old code perform identically.
>     >>     
>     >>     Lots of extra optimizations are now possible, e.g.
>     >>     we can fetch multiple heads with copy_from/to_user now.
>     >>     We can get rid of maintaining the log array.  Etc etc.
>     >>     
>     >>     Signed-off-by: Michael S. Tsirkin <mst@redhat.com <mailto:mst@redhat.com>>
>     >>
>     >>  drivers/vhost/test.c  |  2 +-
>     >>  drivers/vhost/vhost.c | 39 ++++++++++++++++++++++++++++++++++-----
>     >>  drivers/vhost/vhost.h |  4 +++-
>     >>  3 files changed, 38 insertions(+), 7 deletions(-)
>     >>
>     >
>     >
>     > And the symptom is still the same - random crashes
>     > after a bit of traffic, right?
> 
>     random guest crashes after a reboot of the guests. As if vhost would still
>     write into now stale buffers.
> 

Re: vhost changes (batched) in linux-next after 12/13 trigger random crashes in KVM guests after reboot

[Christian Borntraeger]

On 10.02.20 10:47, Eugenio Perez Martin wrote:
> Hi Christian.
> 
> I'm not able to reproduce the failure with eccb852f1fe6bede630e2e4f1a121a81e34354ab commit. Could you add more data? Your configuration (libvirt or qemu line), and host's dmesg output if any?
> 
> Thanks!

If it was not obvious, this is on s390x, a big endian system.

Re: vhost changes (batched) in linux-next after 12/13 trigger random crashes in KVM guests after reboot

[Eugenio Pérez]

On Mon, 2020-02-10 at 12:01 +0100, Christian Borntraeger wrote:
> 
> On 10.02.20 10:47, Eugenio Perez Martin wrote:
> > Hi Christian.
> > 
> > I'm not able to reproduce the failure with eccb852f1fe6bede630e2e4f1a121a81e34354ab commit. Could you add more data?
> > Your configuration (libvirt or qemu line), and host's dmesg output if any?
> > 
> > Thanks!
> 
> If it was not obvious, this is on s390x, a big endian system.
> 

Hi Christian. Thank you very much for the hints.

Could we add some debug traces? Something like the inline patch should give us some clues.

Thanks!

From a8d65d5f0ae3d305443ee84b4842b7c712a1ac1d Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Eugenio=20P=C3=A9rez?= <eperezma@redhat.com>
Date: Tue, 11 Feb 2020 10:29:01 +0100
Subject: [PATCH] Add some traces

---
 drivers/vhost/vhost.c | 9 +++++++--
 1 file changed, 7 insertions(+), 2 deletions(-)

diff --git a/drivers/vhost/vhost.c b/drivers/vhost/vhost.c
index b5a51b1f2e79..60c048eebe4d 100644
--- a/drivers/vhost/vhost.c
+++ b/drivers/vhost/vhost.c
@@ -2239,8 +2239,8 @@ static int fetch_buf(struct vhost_virtqueue *vq)
 		vq->avail_idx = vhost16_to_cpu(vq, avail_idx);
 
 		if (unlikely((u16)(vq->avail_idx - last_avail_idx) > vq->num)) {
-			vq_err(vq, "Guest moved used index from %u to %u",
-				last_avail_idx, vq->avail_idx);
+			vq_err(vq, "Guest moved vq %p used index from %u to %u",
+				vq, last_avail_idx, vq->avail_idx);
 			return -EFAULT;
 		}
 
@@ -2336,6 +2336,8 @@ static int fetch_descs(struct vhost_virtqueue *vq)
 	while (!ret && vq->ndescs <= vq->batch_descs)
 		ret = fetch_buf(vq);
 
+	pr_debug("[vq=%p][vq->ndescs=%d]", vq, vq->ndescs);
+
 	return vq->ndescs ? 0 : ret;
 }
 
@@ -2416,6 +2418,8 @@ int vhost_get_vq_desc(struct vhost_virtqueue *vq,
 	}
 
 	vq->first_desc = i + 1;
+	pr_debug("[vq=%p][vq->ndescs=%d][vq->first_desc=%d]", vq, vq->ndescs,
+		 vq->first_desc);
 
 	return ret;
 
@@ -2459,6 +2463,7 @@ static int __vhost_add_used_n(struct vhost_virtqueue *vq,
 
 	start = vq->last_used_idx & (vq->num - 1);
 	used = vq->used->ring + start;
+	pr_debug("[vq=%p][start=%d][count=%u]", vq, start, count);
 	if (vhost_put_used(vq, heads, start, count)) {
 		vq_err(vq, "Failed to write used");
 		return -EFAULT;
-- 
2.18.1

Re: vhost changes (batched) in linux-next after 12/13 trigger random crashes in KVM guests after reboot

[Christian Borntraeger]

[-- Attachment #1: Type: text/plain, Size: 689 bytes --]



On 11.02.20 10:33, Eugenio Pérez wrote:
> On Mon, 2020-02-10 at 12:01 +0100, Christian Borntraeger wrote:
>>
>> On 10.02.20 10:47, Eugenio Perez Martin wrote:
>>> Hi Christian.
>>>
>>> I'm not able to reproduce the failure with eccb852f1fe6bede630e2e4f1a121a81e34354ab commit. Could you add more data?
>>> Your configuration (libvirt or qemu line), and host's dmesg output if any?
>>>
>>> Thanks!
>>
>> If it was not obvious, this is on s390x, a big endian system.
>>
> 
> Hi Christian. Thank you very much for the hints.
> 
> Could we add some debug traces? Something like the inline patch should give us some clues.

attached. this time the guest crashed (and no guest moved message)

[-- Attachment #2: log --]
[-- Type: text/plain, Size: 79115 bytes --]

[    0.140906] Linux version 5.5.0+ (cborntra@m83lp52.lnxne.boe) (gcc version 9.2.1 20190827 (Red Hat 9.2.1-1) (GCC)) #18 SMP Tue Feb 11 10:51:14 CET 2020
[    0.140908] setup: Linux is running natively in 64-bit mode
[    0.140950] setup: The maximum memory size is 131072MB
[    0.140955] setup: Reserving 1024MB of memory at 130048MB for crashkernel (System RAM: 130048MB)
[    0.140967] numa: NUMA mode: plain
[    0.141060] cpu: 64 configured CPUs, 0 standby CPUs
[    0.141107] cpu: The CPU configuration topology of the machine is: 0 0 4 2 3 10 / 4
[    0.141626] Write protected kernel read-only data: 13420k
[    0.142408] Zone ranges:
[    0.142409]   DMA      [mem 0x0000000000000000-0x000000007fffffff]
[    0.142411]   Normal   [mem 0x0000000080000000-0x0000001fffffffff]
[    0.142412] Movable zone start for each node
[    0.142413] Early memory node ranges
[    0.142414]   node   0: [mem 0x0000000000000000-0x0000001fffffffff]
[    0.142421] Initmem setup node 0 [mem 0x0000000000000000-0x0000001fffffffff]
[    0.142422] On node 0 totalpages: 33554432
[    0.142423]   DMA zone: 8192 pages used for memmap
[    0.142423]   DMA zone: 0 pages reserved
[    0.142424]   DMA zone: 524288 pages, LIFO batch:63
[    0.159790]   Normal zone: 516096 pages used for memmap
[    0.159791]   Normal zone: 33030144 pages, LIFO batch:63
[    0.175978] percpu: Embedded 33 pages/cpu s97024 r8192 d29952 u135168
[    0.175986] pcpu-alloc: s97024 r8192 d29952 u135168 alloc=33*4096
[    0.175987] pcpu-alloc: [0] 000 [0] 001 [0] 002 [0] 003 
[    0.175988] pcpu-alloc: [0] 004 [0] 005 [0] 006 [0] 007 
[    0.175989] pcpu-alloc: [0] 008 [0] 009 [0] 010 [0] 011 
[    0.175991] pcpu-alloc: [0] 012 [0] 013 [0] 014 [0] 015 
[    0.175992] pcpu-alloc: [0] 016 [0] 017 [0] 018 [0] 019 
[    0.175993] pcpu-alloc: [0] 020 [0] 021 [0] 022 [0] 023 
[    0.175994] pcpu-alloc: [0] 024 [0] 025 [0] 026 [0] 027 
[    0.175996] pcpu-alloc: [0] 028 [0] 029 [0] 030 [0] 031 
[    0.175997] pcpu-alloc: [0] 032 [0] 033 [0] 034 [0] 035 
[    0.175998] pcpu-alloc: [0] 036 [0] 037 [0] 038 [0] 039 
[    0.175999] pcpu-alloc: [0] 040 [0] 041 [0] 042 [0] 043 
[    0.176000] pcpu-alloc: [0] 044 [0] 045 [0] 046 [0] 047 
[    0.176001] pcpu-alloc: [0] 048 [0] 049 [0] 050 [0] 051 
[    0.176003] pcpu-alloc: [0] 052 [0] 053 [0] 054 [0] 055 
[    0.176004] pcpu-alloc: [0] 056 [0] 057 [0] 058 [0] 059 
[    0.176005] pcpu-alloc: [0] 060 [0] 061 [0] 062 [0] 063 
[    0.176006] pcpu-alloc: [0] 064 [0] 065 [0] 066 [0] 067 
[    0.176007] pcpu-alloc: [0] 068 [0] 069 [0] 070 [0] 071 
[    0.176008] pcpu-alloc: [0] 072 [0] 073 [0] 074 [0] 075 
[    0.176010] pcpu-alloc: [0] 076 [0] 077 [0] 078 [0] 079 
[    0.176011] pcpu-alloc: [0] 080 [0] 081 [0] 082 [0] 083 
[    0.176012] pcpu-alloc: [0] 084 [0] 085 [0] 086 [0] 087 
[    0.176013] pcpu-alloc: [0] 088 [0] 089 [0] 090 [0] 091 
[    0.176015] pcpu-alloc: [0] 092 [0] 093 [0] 094 [0] 095 
[    0.176016] pcpu-alloc: [0] 096 [0] 097 [0] 098 [0] 099 
[    0.176017] pcpu-alloc: [0] 100 [0] 101 [0] 102 [0] 103 
[    0.176019] pcpu-alloc: [0] 104 [0] 105 [0] 106 [0] 107 
[    0.176020] pcpu-alloc: [0] 108 [0] 109 [0] 110 [0] 111 
[    0.176021] pcpu-alloc: [0] 112 [0] 113 [0] 114 [0] 115 
[    0.176022] pcpu-alloc: [0] 116 [0] 117 [0] 118 [0] 119 
[    0.176024] pcpu-alloc: [0] 120 [0] 121 [0] 122 [0] 123 
[    0.176025] pcpu-alloc: [0] 124 [0] 125 [0] 126 [0] 127 
[    0.176026] pcpu-alloc: [0] 128 [0] 129 [0] 130 [0] 131 
[    0.176028] pcpu-alloc: [0] 132 [0] 133 [0] 134 [0] 135 
[    0.176029] pcpu-alloc: [0] 136 [0] 137 [0] 138 [0] 139 
[    0.176030] pcpu-alloc: [0] 140 [0] 141 [0] 142 [0] 143 
[    0.176031] pcpu-alloc: [0] 144 [0] 145 [0] 146 [0] 147 
[    0.176033] pcpu-alloc: [0] 148 [0] 149 [0] 150 [0] 151 
[    0.176034] pcpu-alloc: [0] 152 [0] 153 [0] 154 [0] 155 
[    0.176035] pcpu-alloc: [0] 156 [0] 157 [0] 158 [0] 159 
[    0.176036] pcpu-alloc: [0] 160 [0] 161 [0] 162 [0] 163 
[    0.176037] pcpu-alloc: [0] 164 [0] 165 [0] 166 [0] 167 
[    0.176039] pcpu-alloc: [0] 168 [0] 169 [0] 170 [0] 171 
[    0.176040] pcpu-alloc: [0] 172 [0] 173 [0] 174 [0] 175 
[    0.176041] pcpu-alloc: [0] 176 [0] 177 [0] 178 [0] 179 
[    0.176042] pcpu-alloc: [0] 180 [0] 181 [0] 182 [0] 183 
[    0.176044] pcpu-alloc: [0] 184 [0] 185 [0] 186 [0] 187 
[    0.176045] pcpu-alloc: [0] 188 [0] 189 [0] 190 [0] 191 
[    0.176046] pcpu-alloc: [0] 192 [0] 193 [0] 194 [0] 195 
[    0.176047] pcpu-alloc: [0] 196 [0] 197 [0] 198 [0] 199 
[    0.176048] pcpu-alloc: [0] 200 [0] 201 [0] 202 [0] 203 
[    0.176049] pcpu-alloc: [0] 204 [0] 205 [0] 206 [0] 207 
[    0.176051] pcpu-alloc: [0] 208 [0] 209 [0] 210 [0] 211 
[    0.176052] pcpu-alloc: [0] 212 [0] 213 [0] 214 [0] 215 
[    0.176053] pcpu-alloc: [0] 216 [0] 217 [0] 218 [0] 219 
[    0.176054] pcpu-alloc: [0] 220 [0] 221 [0] 222 [0] 223 
[    0.176055] pcpu-alloc: [0] 224 [0] 225 [0] 226 [0] 227 
[    0.176056] pcpu-alloc: [0] 228 [0] 229 [0] 230 [0] 231 
[    0.176058] pcpu-alloc: [0] 232 [0] 233 [0] 234 [0] 235 
[    0.176059] pcpu-alloc: [0] 236 [0] 237 [0] 238 [0] 239 
[    0.176060] pcpu-alloc: [0] 240 [0] 241 [0] 242 [0] 243 
[    0.176061] pcpu-alloc: [0] 244 [0] 245 [0] 246 [0] 247 
[    0.176062] pcpu-alloc: [0] 248 [0] 249 [0] 250 [0] 251 
[    0.176064] pcpu-alloc: [0] 252 [0] 253 [0] 254 [0] 255 
[    0.176065] pcpu-alloc: [0] 256 [0] 257 [0] 258 [0] 259 
[    0.176066] pcpu-alloc: [0] 260 [0] 261 [0] 262 [0] 263 
[    0.176067] pcpu-alloc: [0] 264 [0] 265 [0] 266 [0] 267 
[    0.176068] pcpu-alloc: [0] 268 [0] 269 [0] 270 [0] 271 
[    0.176069] pcpu-alloc: [0] 272 [0] 273 [0] 274 [0] 275 
[    0.176071] pcpu-alloc: [0] 276 [0] 277 [0] 278 [0] 279 
[    0.176072] pcpu-alloc: [0] 280 [0] 281 [0] 282 [0] 283 
[    0.176073] pcpu-alloc: [0] 284 [0] 285 [0] 286 [0] 287 
[    0.176074] pcpu-alloc: [0] 288 [0] 289 [0] 290 [0] 291 
[    0.176075] pcpu-alloc: [0] 292 [0] 293 [0] 294 [0] 295 
[    0.176077] pcpu-alloc: [0] 296 [0] 297 [0] 298 [0] 299 
[    0.176078] pcpu-alloc: [0] 300 [0] 301 [0] 302 [0] 303 
[    0.176079] pcpu-alloc: [0] 304 [0] 305 [0] 306 [0] 307 
[    0.176080] pcpu-alloc: [0] 308 [0] 309 [0] 310 [0] 311 
[    0.176081] pcpu-alloc: [0] 312 [0] 313 [0] 314 [0] 315 
[    0.176082] pcpu-alloc: [0] 316 [0] 317 [0] 318 [0] 319 
[    0.176084] pcpu-alloc: [0] 320 [0] 321 [0] 322 [0] 323 
[    0.176085] pcpu-alloc: [0] 324 [0] 325 [0] 326 [0] 327 
[    0.176086] pcpu-alloc: [0] 328 [0] 329 [0] 330 [0] 331 
[    0.176087] pcpu-alloc: [0] 332 [0] 333 [0] 334 [0] 335 
[    0.176088] pcpu-alloc: [0] 336 [0] 337 [0] 338 [0] 339 
[    0.176109] Built 1 zonelists, mobility grouping on.  Total pages: 33030144
[    0.176110] Policy zone: Normal
[    0.176111] Kernel command line: root=/dev/disk/by-path/ccw-0.0.3318-part1 rd.dasd=0.0.3318 cio_ignore=all,!condev rd.znet=qeth,0.0.bd00,0.0.bd01,0.0.bd02,layer2=1,portno=0,portname=OSAPORT zfcp.allow_lun_scan=0 BOOT_IMAGE=0 crashkernel=1G BOOT_IMAGE=0
[    0.177105] printk: log_buf_len individual max cpu contribution: 4096 bytes
[    0.177106] printk: log_buf_len total cpu_extra contributions: 1388544 bytes
[    0.177106] printk: log_buf_len min size: 131072 bytes
[    0.177403] printk: log_buf_len: 2097152 bytes
[    0.177404] printk: early log buf free: 123868(94%)
[    0.186473] Dentry cache hash table entries: 8388608 (order: 14, 67108864 bytes, linear)
[    0.190998] Inode-cache hash table entries: 4194304 (order: 13, 33554432 bytes, linear)
[    0.191000] mem auto-init: stack:off, heap alloc:off, heap free:off
[    0.223325] Memory: 2314920K/134217728K available (10364K kernel code, 1904K rwdata, 3056K rodata, 3912K init, 856K bss, 3355756K reserved, 0K cma-reserved)
[    0.223685] SLUB: HWalign=256, Order=0-3, MinObjects=0, CPUs=340, Nodes=1
[    0.223724] ftrace: allocating 31322 entries in 123 pages
[    0.228303] ftrace: allocated 123 pages with 6 groups
[    0.229076] rcu: Hierarchical RCU implementation.
[    0.229076] rcu: 	RCU event tracing is enabled.
[    0.229077] rcu: 	RCU restricting CPUs from NR_CPUS=512 to nr_cpu_ids=340.
[    0.229077] 	Tasks RCU enabled.
[    0.229078] rcu: RCU calculated value of scheduler-enlistment delay is 11 jiffies.
[    0.229079] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=340
[    0.231937] NR_IRQS: 3, nr_irqs: 3, preallocated irqs: 3
[    0.232042] clocksource: tod: mask: 0xffffffffffffffff max_cycles: 0x3b0a9be803b0a9, max_idle_ns: 1805497147909793 ns
[    0.232242] Console: colour dummy device 80x25
[    0.410136] printk: console [ttyS0] enabled
[    0.503299] Calibrating delay loop (skipped)... 21881.00 BogoMIPS preset
[    0.503300] pid_max: default: 348160 minimum: 2720
[    0.503432] LSM: Security Framework initializing
[    0.503451] SELinux:  Initializing.
[    0.503695] Mount-cache hash table entries: 131072 (order: 8, 1048576 bytes, linear)
[    0.503841] Mountpoint-cache hash table entries: 131072 (order: 8, 1048576 bytes, linear)
[    0.504709] rcu: Hierarchical SRCU implementation.
[    0.507419] smp: Bringing up secondary CPUs ...
[    0.520947] smp: Brought up 1 node, 64 CPUs
[    1.416671] node 0 initialised, 32136763 pages in 890ms
[    1.443532] devtmpfs: initialized
[    1.444269] random: get_random_u32 called from bucket_table_alloc.isra.0+0x82/0x120 with crng_init=0
[    1.444808] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604462750000 ns
[    1.445430] futex hash table entries: 131072 (order: 13, 33554432 bytes, vmalloc)
[    1.449946] xor: automatically using best checksumming function   xc        
[    1.450107] NET: Registered protocol family 16
[    1.450143] audit: initializing netlink subsys (disabled)
[    1.450207] audit: type=2000 audit(1581414814.421:1): state=initialized audit_enabled=0 res=1
[    1.450331] Spectre V2 mitigation: etokens
[    1.451141] random: fast init done
[    1.480154] HugeTLB registered 1.00 MiB page size, pre-allocated 0 pages
[    1.722111] raid6: vx128x8  gen() 23433 MB/s
[    1.892050] raid6: vx128x8  xor() 13777 MB/s
[    1.892052] raid6: using algorithm vx128x8 gen() 23433 MB/s
[    1.892053] raid6: .... xor() 13777 MB/s, rmw enabled
[    1.892054] raid6: using s390xc recovery algorithm
[    1.892402] iommu: Default domain type: Translated 
[    1.892573] SCSI subsystem initialized
[    1.960177] PCI host bridge to bus 0000:00
[    1.960183] pci_bus 0000:00: root bus resource [mem 0x8000000000000000-0x8000000007ffffff 64bit pref]
[    1.960185] pci_bus 0000:00: No busn resource found for root bus, will use [bus 00-ff]
[    1.960246] pci 0000:00:00.0: [1014:044b] type 00 class 0x120000
[    1.960296] pci 0000:00:00.0: reg 0x10: [mem 0xffffd80008000000-0xffffd8000fffffff 64bit pref]
[    1.960563] pci 0000:00:00.0: 0.000 Gb/s available PCIe bandwidth, limited by Unknown speed x0 link at 0000:00:00.0 (capable of 32.000 Gb/s with 5 GT/s x8 link)
[    1.960599] pci 0000:00:00.0: Adding to iommu group 0
[    1.960608] pci_bus 0000:00: busn_res: [bus 00-ff] end is updated to 00
[    1.962518] PCI host bridge to bus 0001:00
[    1.962520] pci_bus 0001:00: root bus resource [mem 0x8001000000000000-0x80010000000fffff 64bit pref]
[    1.962522] pci_bus 0001:00: No busn resource found for root bus, will use [bus 00-ff]
[    1.962612] pci 0001:00:00.0: [15b3:1016] type 00 class 0x020000
[    1.962713] pci 0001:00:00.0: reg 0x10: [mem 0xffffd40002000000-0xffffd400020fffff 64bit pref]
[    1.962865] pci 0001:00:00.0: enabling Extended Tags
[    1.963360] pci 0001:00:00.0: 0.000 Gb/s available PCIe bandwidth, limited by Unknown speed x0 link at 0001:00:00.0 (capable of 63.008 Gb/s with 8 GT/s x8 link)
[    1.963410] pci 0001:00:00.0: Adding to iommu group 1
[    1.963418] pci_bus 0001:00: busn_res: [bus 00-ff] end is updated to 00
[    1.965352] PCI host bridge to bus 0002:00
[    1.965354] pci_bus 0002:00: root bus resource [mem 0x8002000000000000-0x80020000000fffff 64bit pref]
[    1.965356] pci_bus 0002:00: No busn resource found for root bus, will use [bus 00-ff]
[    1.965443] pci 0002:00:00.0: [15b3:1016] type 00 class 0x020000
[    1.965541] pci 0002:00:00.0: reg 0x10: [mem 0xffffd40008000000-0xffffd400080fffff 64bit pref]
[    1.965692] pci 0002:00:00.0: enabling Extended Tags
[    1.966190] pci 0002:00:00.0: 0.000 Gb/s available PCIe bandwidth, limited by Unknown speed x0 link at 0002:00:00.0 (capable of 63.008 Gb/s with 8 GT/s x8 link)
[    1.966223] pci 0002:00:00.0: Adding to iommu group 2
[    1.966230] pci_bus 0002:00: busn_res: [bus 00-ff] end is updated to 00
[    2.455601] VFS: Disk quotas dquot_6.6.0
[    2.455644] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
[    2.457224] NET: Registered protocol family 2
[    2.457763] random: crng init done
[    2.458056] tcp_listen_portaddr_hash hash table entries: 65536 (order: 8, 1048576 bytes, linear)
[    2.458607] TCP established hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc)
[    2.460722] TCP bind hash table entries: 65536 (order: 8, 1048576 bytes, linear)
[    2.461169] TCP: Hash tables configured (established 524288 bind 65536)
[    2.461517] UDP hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc)
[    2.462534] UDP-Lite hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc)
[    2.463929] NET: Registered protocol family 1
[    2.464136] Trying to unpack rootfs image as initramfs...
[    3.031779] Freeing initrd memory: 43772K
[    3.032996] alg: No test for crc32be (crc32be-vx)
[    3.037140] Initialise system trusted keyrings
[    3.037188] workingset: timestamp_bits=42 max_order=25 bucket_order=0
[    3.038293] fuse: init (API version 7.31)
[    3.038367] SGI XFS with ACLs, security attributes, realtime, quota, no debug enabled
[    3.046520] Key type asymmetric registered
[    3.046522] Asymmetric key parser 'x509' registered
[    3.046529] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 251)
[    3.046738] io scheduler mq-deadline registered
[    3.046740] io scheduler kyber registered
[    3.046761] io scheduler bfq registered
[    3.047604] atomic64_test: passed
[    3.047665] hvc_iucv: The z/VM IUCV HVC device driver cannot be used without z/VM
[    3.053492] brd: module loaded
[    3.053875] cio: Channel measurement facility initialized using format extended (mode autodetected)
[    3.054146] Discipline DIAG cannot be used without z/VM
[    4.874424] sclp_sd: No data is available for the config data entity
[    5.345780] qeth: loading core functions
[    5.345842] qeth: register layer 2 discipline
[    5.345843] qeth: register layer 3 discipline
[    5.346294] NET: Registered protocol family 10
[    5.347206] Segment Routing with IPv6
[    5.347223] NET: Registered protocol family 17
[    5.347237] Key type dns_resolver registered
[    5.347314] registered taskstats version 1
[    5.347319] Loading compiled-in X.509 certificates
[    5.387008] Loaded X.509 cert 'Build time autogenerated kernel key: c46ba92ee388c82c5891ee836c9c20b752cdfac5'
[    5.387643] zswap: default zpool zbud not available
[    5.387645] zswap: pool creation failed
[    5.388151] Key type ._fscrypt registered
[    5.388153] Key type .fscrypt registered
[    5.388421] Btrfs loaded, crc32c=crc32c-vx
[    5.392827] Key type big_key registered
[    5.392833] ima: No TPM chip found, activating TPM-bypass!
[    5.392837] ima: Allocated hash algorithm: sha256
[    5.392845] ima: No architecture policies found
[    5.393985] Freeing unused kernel memory: 3912K
[    5.492121] Write protected read-only-after-init data: 68k
[    5.492123] Run /init as init process
[    5.506237] systemd[1]: Inserted module 'autofs4'
[    5.507585] systemd[1]: systemd v243.5-1.fc31 running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=unified)
[    5.507985] systemd[1]: Detected architecture s390x.
[    5.507987] systemd[1]: Running in initial RAM disk.
[    5.508036] systemd[1]: Set hostname to <m83lp52.lnxne.boe>.
[    5.509031] systemd[1]: Couldn't move remaining userspace processes, ignoring: Input/output error
[    5.542708] systemd[1]: Reached target Local File Systems.
[    5.542751] systemd[1]: Reached target Slices.
[    5.542770] systemd[1]: Reached target Swap.
[    5.542787] systemd[1]: Reached target Timers.
[    5.542868] systemd[1]: Listening on Journal Audit Socket.
[    5.812348] audit: type=1130 audit(1581414818.791:2): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=kernel msg='unit=systemd-journald comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[    5.819297] audit: type=1130 audit(1581414818.791:3): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=kernel msg='unit=systemd-tmpfiles-setup comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[    6.765404] audit: type=1130 audit(1581414819.741:4): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=kernel msg='unit=dracut-cmdline comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[    6.780267] audit: type=1130 audit(1581414819.751:5): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=kernel msg='unit=dracut-pre-udev comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[    7.005557] audit: type=1130 audit(1581414819.981:6): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=kernel msg='unit=systemd-udevd comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[    7.009701] qeth 0.0.bd00: Priority Queueing not supported
[    7.010265] qeth 0.0.bd00: portname is deprecated and is ignored
[    7.011335] dasd-eckd 0.0.3319: A channel path to the device has become operational
[    7.011795] dasd-eckd 0.0.3318: A channel path to the device has become operational
[    7.012163] dasd-eckd 0.0.331a: A channel path to the device has become operational
[    7.012327] dasd-eckd 0.0.331b: A channel path to the device has become operational
[    7.015361] qdio: 0.0.bd02 OSA on SC 159b using AI:1 QEBSM:0 PRI:1 TDD:1 SIGA: W AP
[    7.021855] dasd-eckd 0.0.3319: New DASD 3390/0E (CU 3990/01) with 262668 cylinders, 15 heads, 224 sectors
[    7.024637] dasd-eckd 0.0.3319: DASD with 4 KB/block, 189120960 KB total size, 48 KB/track, compatible disk layout
[    7.025745]  dasda:VOL1/  0X3319: dasda1
[    7.026758] dasd-eckd 0.0.3318: New DASD 3390/0E (CU 3990/01) with 262668 cylinders, 15 heads, 224 sectors
[    7.029512] dasd-eckd 0.0.3318: DASD with 4 KB/block, 189120960 KB total size, 48 KB/track, compatible disk layout
[    7.030552]  dasdb:VOL1/  0X3318: dasdb1
[    7.031661] dasd-eckd 0.0.331a: New DASD 3390/0C (CU 3990/01) with 30051 cylinders, 15 heads, 224 sectors
[    7.034797] dasd-eckd 0.0.331a: DASD with 4 KB/block, 21636720 KB total size, 48 KB/track, compatible disk layout
[    7.035495]  dasdc:VOL1/  0X331A:
[    7.036623] dasd-eckd 0.0.331b: New DASD 3390/0C (CU 3990/01) with 30051 cylinders, 15 heads, 224 sectors
[    7.039286] dasd-eckd 0.0.331b: DASD with 4 KB/block, 21636720 KB total size, 48 KB/track, compatible disk layout
[    7.039991]  dasdd:VOL1/  0X331B:
[    7.046991] audit: type=1130 audit(1581414820.021:7): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=kernel msg='unit=systemd-udev-trigger comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[    7.055127] qeth 0.0.bd00: QDIO data connection isolation is deactivated
[    7.055590] qeth 0.0.bd00: The device represents a Bridge Capable Port
[    7.058856] qeth 0.0.bd00: MAC address 0e:87:75:01:5c:5b successfully registered
[    7.059383] qeth 0.0.bd00: Device is a OSD Express card (level: 0198)
               with link type OSD_10GIG.
[    7.062125] audit: type=1130 audit(1581414820.041:8): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=kernel msg='unit=plymouth-start comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[    7.089046] qeth 0.0.bd00: MAC address de:45:d7:61:c4:13 successfully registered
[    7.091260] qeth 0.0.bd00 encbd00: renamed from eth0
[    7.273398] mlx5_core 0001:00:00.0: enabling device (0000 -> 0002)
[    7.273485] mlx5_core 0001:00:00.0: firmware version: 14.23.1020
[    7.340188] dasdconf.sh Warning: 0.0.331b is already online, not configuring
[    7.353716] dasdconf.sh Warning: 0.0.3319 is already online, not configuring
[    7.353908] dasdconf.sh Warning: 0.0.3318 is already online, not configuring
[    7.487167] dasdconf.sh Warning: 0.0.331a is already online, not configuring
[    7.716124] mlx5_core 0002:00:00.0: enabling device (0000 -> 0002)
[    7.716210] mlx5_core 0002:00:00.0: firmware version: 14.23.1020
[    8.163850] mlx5_core 0001:00:00.0: MLX5E: StrdRq(0) RqSz(1024) StrdSz(256) RxCqeCmprss(0)
[    8.312389] mlx5_core 0002:00:00.0: MLX5E: StrdRq(0) RqSz(1024) StrdSz(256) RxCqeCmprss(0)
[    8.468505] mlx5_core 0001:00:00.0 enP1s519: renamed from eth0
[    8.782875] mlx5_core 0002:00:00.0 enP2s564: renamed from eth1
[    8.973475] audit: type=1130 audit(1581414821.951:9): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=kernel msg='unit=dracut-initqueue comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[    8.990598] audit: type=1130 audit(1581414821.961:10): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=kernel msg='unit=systemd-fsck-root comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[    9.000284] EXT4-fs (dasdb1): mounted filesystem with ordered data mode. Opts: (null)
[    9.107788] audit: type=1130 audit(1581414822.081:11): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=kernel msg='unit=initrd-parse-etc comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[    9.107792] audit: type=1131 audit(1581414822.081:12): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=kernel msg='unit=initrd-parse-etc comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[    9.152972] audit: type=1130 audit(1581414822.131:13): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=kernel msg='unit=dracut-pre-pivot comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[    9.356930] systemd-journald[542]: Received SIGTERM from PID 1 (systemd).
[    9.371415] printk: systemd: 20 output lines suppressed due to ratelimiting
[    9.689903] SELinux:  Permission watch in class filesystem not defined in policy.
[    9.689908] SELinux:  Permission watch in class file not defined in policy.
[    9.689909] SELinux:  Permission watch_mount in class file not defined in policy.
[    9.689910] SELinux:  Permission watch_sb in class file not defined in policy.
[    9.689911] SELinux:  Permission watch_with_perm in class file not defined in policy.
[    9.689912] SELinux:  Permission watch_reads in class file not defined in policy.
[    9.689915] SELinux:  Permission watch in class dir not defined in policy.
[    9.689915] SELinux:  Permission watch_mount in class dir not defined in policy.
[    9.689916] SELinux:  Permission watch_sb in class dir not defined in policy.
[    9.689917] SELinux:  Permission watch_with_perm in class dir not defined in policy.
[    9.689918] SELinux:  Permission watch_reads in class dir not defined in policy.
[    9.689921] SELinux:  Permission watch in class lnk_file not defined in policy.
[    9.689922] SELinux:  Permission watch_mount in class lnk_file not defined in policy.
[    9.689923] SELinux:  Permission watch_sb in class lnk_file not defined in policy.
[    9.689924] SELinux:  Permission watch_with_perm in class lnk_file not defined in policy.
[    9.689925] SELinux:  Permission watch_reads in class lnk_file not defined in policy.
[    9.689927] SELinux:  Permission watch in class chr_file not defined in policy.
[    9.689941] SELinux:  Permission watch_mount in class chr_file not defined in policy.
[    9.689942] SELinux:  Permission watch_sb in class chr_file not defined in policy.
[    9.689943] SELinux:  Permission watch_with_perm in class chr_file not defined in policy.
[    9.689944] SELinux:  Permission watch_reads in class chr_file not defined in policy.
[    9.689946] SELinux:  Permission watch in class blk_file not defined in policy.
[    9.689947] SELinux:  Permission watch_mount in class blk_file not defined in policy.
[    9.689948] SELinux:  Permission watch_sb in class blk_file not defined in policy.
[    9.689949] SELinux:  Permission watch_with_perm in class blk_file not defined in policy.
[    9.689950] SELinux:  Permission watch_reads in class blk_file not defined in policy.
[    9.689952] SELinux:  Permission watch in class sock_file not defined in policy.
[    9.689953] SELinux:  Permission watch_mount in class sock_file not defined in policy.
[    9.689954] SELinux:  Permission watch_sb in class sock_file not defined in policy.
[    9.689955] SELinux:  Permission watch_with_perm in class sock_file not defined in policy.
[    9.689956] SELinux:  Permission watch_reads in class sock_file not defined in policy.
[    9.689958] SELinux:  Permission watch in class fifo_file not defined in policy.
[    9.689959] SELinux:  Permission watch_mount in class fifo_file not defined in policy.
[    9.689961] SELinux:  Permission watch_sb in class fifo_file not defined in policy.
[    9.689962] SELinux:  Permission watch_with_perm in class fifo_file not defined in policy.
[    9.689963] SELinux:  Permission watch_reads in class fifo_file not defined in policy.
[    9.690042] SELinux:  Class perf_event not defined in policy.
[    9.690043] SELinux: the above unknown classes and permissions will be allowed
[    9.690045] SELinux:  policy capability network_peer_controls=1
[    9.690046] SELinux:  policy capability open_perms=1
[    9.690047] SELinux:  policy capability extended_socket_class=1
[    9.690048] SELinux:  policy capability always_check_network=0
[    9.690048] SELinux:  policy capability cgroup_seclabel=1
[    9.690049] SELinux:  policy capability nnp_nosuid_transition=1
[    9.766488] systemd[1]: Successfully loaded SELinux policy in 336.854ms.
[    9.805966] systemd[1]: Relabelled /dev, /dev/shm, /run, /sys/fs/cgroup in 12.884ms.
[    9.808232] systemd[1]: systemd v243.5-1.fc31 running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=unified)
[    9.808649] systemd[1]: Detected architecture s390x.
[    9.810801] systemd[1]: Set hostname to <m83lp52.lnxne.boe>.
[    9.813360] systemd[1]: Couldn't move remaining userspace processes, ignoring: Input/output error
[    9.907681] systemd[1]: /usr/lib/systemd/system/sssd.service:12: PIDFile= references a path below legacy directory /var/run/, updating /var/run/sssd.pid → /run/sssd.pid; please update the unit file accordingly.
[    9.911744] systemd[1]: /usr/lib/systemd/system/iscsid.service:11: PIDFile= references a path below legacy directory /var/run/, updating /var/run/iscsid.pid → /run/iscsid.pid; please update the unit file accordingly.
[    9.911947] systemd[1]: /usr/lib/systemd/system/iscsiuio.service:13: PIDFile= references a path below legacy directory /var/run/, updating /var/run/iscsiuio.pid → /run/iscsiuio.pid; please update the unit file accordingly.
[    9.939442] systemd[1]: /usr/lib/systemd/system/sssd-kcm.socket:7: ListenStream= references a path below legacy directory /var/run/, updating /var/run/.heim_org.h5l.kcm-socket → /run/.heim_org.h5l.kcm-socket; please update the unit file accordingly.
[    9.966989] systemd[1]: initrd-switch-root.service: Succeeded.
[    9.967102] systemd[1]: Stopped Switch Root.
[    9.992173] EXT4-fs (dasdb1): re-mounted. Opts: (null)
[   10.269295] systemd-journald[1086]: Received client request to flush runtime journal.
[   10.448707] VFIO - User Level meta-driver version: 0.3
[   10.487288] dasdconf.sh Warning: 0.0.3319 is already online, not configuring
[   10.488092] dasdconf.sh Warning: 0.0.3318 is already online, not configuring
[   10.488683] dasdconf.sh Warning: 0.0.331a is already online, not configuring
[   10.489424] genwqe 0000:00:00.0: enabling device (0000 -> 0002)
[   10.489591] dasdconf.sh Warning: 0.0.331b is already online, not configuring
[   10.756189] mlx5_ib: Mellanox Connect-IB Infiniband driver v5.0-0
[   10.882438] RPC: Registered named UNIX socket transport module.
[   10.882441] RPC: Registered udp transport module.
[   10.882443] RPC: Registered tcp transport module.
[   10.882443] RPC: Registered tcp NFSv4.1 backchannel transport module.
[   10.912247] RPC: Registered rdma transport module.
[   10.912249] RPC: Registered rdma backchannel transport module.
[   10.975284] XFS (dasda1): Mounting V5 Filesystem
[   10.998984] XFS (dasda1): Ending clean mount
[   11.001618] xfs filesystem being mounted at /home supports timestamps until 2038 (0x7fffffff)
[   11.958825] mlx5_core 0001:00:00.0 enP1s519: Link up
[   11.960449] IPv6: ADDRCONF(NETDEV_CHANGE): enP1s519: link becomes ready
[   12.069226] mlx5_core 0002:00:00.0 enP2s564: Link up
[   12.132744] bridge: filtering via arp/ip/ip6tables is no longer available by default. Update your scripts to load br_netfilter if you need this.
[   12.140433] tun: Universal TUN/TAP device driver, 1.6
[   12.140993] virbr0: port 1(virbr0-nic) entered blocking state
[   12.140995] virbr0: port 1(virbr0-nic) entered disabled state
[   12.141059] device virbr0-nic entered promiscuous mode
[   12.836770] virbr0: port 1(virbr0-nic) entered blocking state
[   12.836773] virbr0: port 1(virbr0-nic) entered listening state
[   12.860218] virbr0: port 1(virbr0-nic) entered disabled state
[   13.032109] IPv6: ADDRCONF(NETDEV_CHANGE): enP2s564: link becomes ready
[   43.323371] [vq=00000000abd097cd][vq->ndescs=0]
[   43.323373] [vq=000000008be108ff][vq->ndescs=0]
[   44.331333] [vq=00000000abd097cd][vq->ndescs=65]
[   44.331336] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=1]
[   44.331339] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=2]
[   44.331341] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=3]
[   44.331342] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=4]
[   44.331344] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=5]
[   44.331346] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=6]
[   44.331347] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=7]
[   44.331349] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=8]
[   44.331351] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=9]
[   44.331353] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=10]
[   44.331355] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=11]
[   44.331357] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=12]
[   44.331358] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=13]
[   44.331360] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=14]
[   44.331361] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=15]
[   44.331363] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=16]
[   44.331365] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=17]
[   44.331367] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=18]
[   44.331369] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=19]
[   44.331370] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=20]
[   44.331372] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=21]
[   44.331373] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=22]
[   44.331375] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=23]
[   44.331376] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=24]
[   44.331378] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=25]
[   44.331393] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=26]
[   44.331395] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=27]
[   44.331396] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=28]
[   44.331398] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=29]
[   44.331399] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=30]
[   44.331401] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=31]
[   44.331402] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=32]
[   44.331403] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=33]
[   44.331405] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=34]
[   44.331406] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=35]
[   44.331407] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=36]
[   44.331408] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=37]
[   44.331410] [vq=00000000abd097cd][start=0][count=37]
[   44.348284] [vq=000000008be108ff][vq->ndescs=1]
[   44.348287] [vq=000000008be108ff][vq->ndescs=1][vq->first_desc=1]
[   44.348290] [vq=000000008be108ff][vq->ndescs=0]
[   44.348297] [vq=000000008be108ff][start=0][count=1]
[   44.362505] [vq=000000008be108ff][vq->ndescs=1]
[   44.362506] [vq=000000008be108ff][vq->ndescs=1][vq->first_desc=1]
[   44.362507] [vq=000000008be108ff][vq->ndescs=0]
[   44.362510] [vq=000000008be108ff][start=1][count=1]
[   44.363079] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=38]
[   44.363081] [vq=00000000abd097cd][start=37][count=1]
[   44.363102] [vq=000000008be108ff][vq->ndescs=2]
[   44.363104] [vq=000000008be108ff][vq->ndescs=2][vq->first_desc=1]
[   44.363105] [vq=000000008be108ff][vq->ndescs=2][vq->first_desc=2]
[   44.363107] [vq=000000008be108ff][vq->ndescs=0]
[   44.363110] [vq=000000008be108ff][start=2][count=2]
[   44.365821] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=39]
[   44.365823] [vq=00000000abd097cd][start=38][count=1]
[   44.365946] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=40]
[   44.365947] [vq=00000000abd097cd][start=39][count=1]
[   44.372309] [vq=000000008be108ff][vq->ndescs=1]
[   44.372311] [vq=000000008be108ff][vq->ndescs=1][vq->first_desc=1]
[   44.372312] [vq=000000008be108ff][vq->ndescs=0]
[   44.372316] [vq=000000008be108ff][start=4][count=1]
[   44.422280] [vq=000000008be108ff][vq->ndescs=1]
[   44.422282] [vq=000000008be108ff][vq->ndescs=1][vq->first_desc=1]
[   44.422283] [vq=000000008be108ff][vq->ndescs=0]
[   44.422286] [vq=000000008be108ff][start=5][count=1]
[   44.569329] [vq=000000008be108ff][vq->ndescs=2]
[   44.569330] [vq=000000008be108ff][vq->ndescs=2][vq->first_desc=2]
[   44.569332] [vq=000000008be108ff][vq->ndescs=0]
[   44.569334] [vq=000000008be108ff][start=6][count=1]
[   44.582286] [vq=000000008be108ff][vq->ndescs=1]
[   44.582287] [vq=000000008be108ff][vq->ndescs=1][vq->first_desc=1]
[   44.582288] [vq=000000008be108ff][vq->ndescs=0]
[   44.582291] [vq=000000008be108ff][start=7][count=1]
[   44.582531] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=41]
[   44.582533] [vq=00000000abd097cd][start=40][count=1]
[   44.633175] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=42]
[   44.633177] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=43]
[   44.633178] [vq=00000000abd097cd][start=41][count=2]
[   44.701286] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=44]
[   44.701287] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=45]
[   44.701289] [vq=00000000abd097cd][start=43][count=2]
[   44.701557] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=46]
[   44.701560] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=47]
[   44.701562] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=48]
[   44.701563] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=49]
[   44.701564] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=50]
[   44.701565] [vq=00000000abd097cd][start=45][count=5]
[   44.701856] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=51]
[   44.701858] [vq=00000000abd097cd][start=50][count=1]
[   44.773183] [vq=000000008be108ff][vq->ndescs=2]
[   44.773185] [vq=000000008be108ff][vq->ndescs=2][vq->first_desc=2]
[   44.773187] [vq=000000008be108ff][vq->ndescs=0]
[   44.773197] [vq=000000008be108ff][start=8][count=1]
[   44.801919] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=52]
[   44.801923] [vq=00000000abd097cd][start=51][count=1]
[   44.911482] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=53]
[   44.911487] [vq=00000000abd097cd][start=52][count=1]
[   44.919072] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=54]
[   44.919074] [vq=00000000abd097cd][start=53][count=1]
[   44.953778] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=55]
[   44.953780] [vq=00000000abd097cd][start=54][count=1]
[   44.974158] [vq=000000008be108ff][vq->ndescs=2]
[   44.974159] [vq=000000008be108ff][vq->ndescs=2][vq->first_desc=2]
[   44.974161] [vq=000000008be108ff][vq->ndescs=0]
[   44.974169] [vq=000000008be108ff][start=9][count=1]
[   45.012038] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=56]
[   45.012040] [vq=00000000abd097cd][start=55][count=1]
[   45.129882] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=57]
[   45.129884] [vq=00000000abd097cd][start=56][count=1]
[   45.178065] [vq=000000008be108ff][vq->ndescs=2]
[   45.178066] [vq=000000008be108ff][vq->ndescs=2][vq->first_desc=2]
[   45.178067] [vq=000000008be108ff][vq->ndescs=0]
[   45.178070] [vq=000000008be108ff][start=10][count=1]
[   45.192279] [vq=000000008be108ff][vq->ndescs=1]
[   45.192280] [vq=000000008be108ff][vq->ndescs=1][vq->first_desc=1]
[   45.192282] [vq=000000008be108ff][vq->ndescs=0]
[   45.192286] [vq=000000008be108ff][start=11][count=1]
[   45.217428] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=58]
[   45.217430] [vq=00000000abd097cd][start=57][count=1]
[   45.427294] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=59]
[   45.427296] [vq=00000000abd097cd][start=58][count=1]
[   45.702447] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=60]
[   45.702449] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=61]
[   45.702450] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=62]
[   45.702451] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=63]
[   45.702453] [vq=00000000abd097cd][start=59][count=4]
[   45.702797] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=64]
[   45.702799] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=65]
[   45.702803] [vq=00000000abd097cd][vq->ndescs=65]
[   45.702804] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=1]
[   45.702806] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=2]
[   45.702807] [vq=00000000abd097cd][start=63][count=4]
[   45.736690] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=3]
[   45.736692] [vq=00000000abd097cd][start=67][count=1]
[   45.772319] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=4]
[   45.772320] [vq=00000000abd097cd][start=68][count=1]
[   45.952427] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=5]
[   45.952430] [vq=00000000abd097cd][start=69][count=1]
[   45.987279] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=6]
[   45.987281] [vq=00000000abd097cd][start=70][count=1]
[   46.232287] [vq=000000008be108ff][vq->ndescs=1]
[   46.232289] [vq=000000008be108ff][vq->ndescs=1][vq->first_desc=1]
[   46.232290] [vq=000000008be108ff][vq->ndescs=0]
[   46.232295] [vq=000000008be108ff][start=12][count=1]
[   46.262277] [vq=000000008be108ff][vq->ndescs=1]
[   46.262278] [vq=000000008be108ff][vq->ndescs=1][vq->first_desc=1]
[   46.262279] [vq=000000008be108ff][vq->ndescs=0]
[   46.262282] [vq=000000008be108ff][start=13][count=1]
[   46.312265] [vq=000000008be108ff][vq->ndescs=1]
[   46.312266] [vq=000000008be108ff][vq->ndescs=1][vq->first_desc=1]
[   46.312267] [vq=000000008be108ff][vq->ndescs=0]
[   46.312270] [vq=000000008be108ff][start=14][count=1]
[   46.348301] [vq=000000008be108ff][vq->ndescs=1]
[   46.348302] [vq=000000008be108ff][vq->ndescs=1][vq->first_desc=1]
[   46.348303] [vq=000000008be108ff][vq->ndescs=0]
[   46.348306] [vq=000000008be108ff][start=15][count=1]
[   46.464029] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=7]
[   46.464031] [vq=00000000abd097cd][start=71][count=1]
[   46.602312] [vq=000000008be108ff][vq->ndescs=1]
[   46.602313] [vq=000000008be108ff][vq->ndescs=1][vq->first_desc=1]
[   46.602314] [vq=000000008be108ff][vq->ndescs=0]
[   46.602318] [vq=000000008be108ff][start=16][count=1]
[   46.609109] [vq=000000008be108ff][vq->ndescs=2]
[   46.609110] [vq=000000008be108ff][vq->ndescs=2][vq->first_desc=2]
[   46.609111] [vq=000000008be108ff][vq->ndescs=0]
[   46.609114] [vq=000000008be108ff][start=17][count=1]
[   46.625327] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=8]
[   46.625328] [vq=00000000abd097cd][start=72][count=1]
[   46.652313] [vq=000000008be108ff][vq->ndescs=1]
[   46.652315] [vq=000000008be108ff][vq->ndescs=1][vq->first_desc=1]
[   46.652316] [vq=000000008be108ff][vq->ndescs=0]
[   46.652319] [vq=000000008be108ff][start=18][count=1]
[   46.703259] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=9]
[   46.703260] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=10]
[   46.703265] [vq=00000000abd097cd][start=73][count=2]
[   46.703685] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=11]
[   46.703688] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=12]
[   46.703690] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=13]
[   46.703691] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=14]
[   46.703692] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=15]
[   46.703693] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=16]
[   46.703695] [vq=00000000abd097cd][start=75][count=6]
[   46.776740] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=17]
[   46.776743] [vq=00000000abd097cd][start=81][count=1]
[   46.833623] [vq=000000008be108ff][vq->ndescs=2]
[   46.833624] [vq=000000008be108ff][vq->ndescs=2][vq->first_desc=2]
[   46.833626] [vq=000000008be108ff][vq->ndescs=0]
[   46.833631] [vq=000000008be108ff][start=19][count=1]
[   46.858225] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=18]
[   46.858227] [vq=00000000abd097cd][start=82][count=1]
[   46.991350] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=19]
[   46.991352] [vq=00000000abd097cd][start=83][count=1]
[   47.027160] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=20]
[   47.027162] [vq=00000000abd097cd][start=84][count=1]
[   47.051619] [vq=000000008be108ff][vq->ndescs=2]
[   47.051621] [vq=000000008be108ff][vq->ndescs=2][vq->first_desc=2]
[   47.051623] [vq=000000008be108ff][vq->ndescs=0]
[   47.051627] [vq=000000008be108ff][start=20][count=1]
[   47.091535] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=21]
[   47.091537] [vq=00000000abd097cd][start=85][count=1]
[   47.254867] [vq=000000008be108ff][vq->ndescs=2]
[   47.254868] [vq=000000008be108ff][vq->ndescs=2][vq->first_desc=2]
[   47.254870] [vq=000000008be108ff][vq->ndescs=0]
[   47.254874] [vq=000000008be108ff][start=21][count=1]
[   47.291144] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=22]
[   47.291146] [vq=00000000abd097cd][start=86][count=1]
[   47.503996] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=23]
[   47.503998] [vq=00000000abd097cd][start=87][count=1]
[   47.704153] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=24]
[   47.704155] [vq=00000000abd097cd][start=88][count=1]
[   47.704433] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=25]
[   47.704436] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=26]
[   47.704437] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=27]
[   47.704439] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=28]
[   47.704440] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=29]
[   47.704441] [vq=00000000abd097cd][start=89][count=5]
[   47.704582] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=30]
[   47.704584] [vq=00000000abd097cd][start=94][count=1]
[   47.748063] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=31]
[   47.748065] [vq=00000000abd097cd][start=95][count=1]
[   47.912449] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=32]
[   47.912452] [vq=00000000abd097cd][start=96][count=1]
[   48.031357] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=33]
[   48.031358] [vq=00000000abd097cd][start=97][count=1]
[   48.348365] [vq=000000008be108ff][vq->ndescs=1]
[   48.348366] [vq=000000008be108ff][vq->ndescs=1][vq->first_desc=1]
[   48.348368] [vq=000000008be108ff][vq->ndescs=0]
[   48.348373] [vq=000000008be108ff][start=22][count=1]
[   48.660191] [vq=000000008be108ff][vq->ndescs=2]
[   48.660193] [vq=000000008be108ff][vq->ndescs=2][vq->first_desc=2]
[   48.660194] [vq=000000008be108ff][vq->ndescs=0]
[   48.660197] [vq=000000008be108ff][start=23][count=1]
[   48.681124] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=34]
[   48.681126] [vq=00000000abd097cd][start=98][count=1]
[   48.703279] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=35]
[   48.703281] [vq=00000000abd097cd][start=99][count=1]
[   48.705144] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=36]
[   48.705146] [vq=00000000abd097cd][start=100][count=1]
[   48.705468] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=37]
[   48.705469] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=38]
[   48.705471] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=39]
[   48.705472] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=40]
[   48.705473] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=41]
[   48.705475] [vq=00000000abd097cd][start=101][count=5]
[   48.705657] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=42]
[   48.705658] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=43]
[   48.705660] [vq=00000000abd097cd][start=106][count=2]
[   48.863171] [vq=000000008be108ff][vq->ndescs=2]
[   48.863173] [vq=000000008be108ff][vq->ndescs=2][vq->first_desc=2]
[   48.863175] [vq=000000008be108ff][vq->ndescs=0]
[   48.863182] [vq=000000008be108ff][start=24][count=1]
[   48.888847] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=44]
[   48.888858] [vq=00000000abd097cd][start=108][count=1]
[   49.117382] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=45]
[   49.117386] [vq=00000000abd097cd][start=109][count=1]
[   49.119562] [vq=000000008be108ff][vq->ndescs=2]
[   49.119563] [vq=000000008be108ff][vq->ndescs=2][vq->first_desc=2]
[   49.119565] [vq=000000008be108ff][vq->ndescs=0]
[   49.119572] [vq=000000008be108ff][start=25][count=1]
[   49.157488] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=46]
[   49.157490] [vq=00000000abd097cd][start=110][count=1]
[   49.321369] [vq=000000008be108ff][vq->ndescs=2]
[   49.321371] [vq=000000008be108ff][vq->ndescs=2][vq->first_desc=2]
[   49.321372] [vq=000000008be108ff][vq->ndescs=0]
[   49.321375] [vq=000000008be108ff][start=26][count=1]
[   49.353755] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=47]
[   49.353758] [vq=00000000abd097cd][start=111][count=1]
[   49.370006] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=48]
[   49.370008] [vq=00000000abd097cd][start=112][count=1]
[   49.370033] [vq=000000008be108ff][vq->ndescs=1]
[   49.370035] [vq=000000008be108ff][vq->ndescs=1][vq->first_desc=1]
[   49.370036] [vq=000000008be108ff][vq->ndescs=0]
[   49.370040] [vq=000000008be108ff][start=27][count=1]
[   49.429924] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=49]
[   49.429927] [vq=00000000abd097cd][start=113][count=1]
[   49.706234] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=50]
[   49.706239] [vq=00000000abd097cd][start=114][count=1]
[   49.706554] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=51]
[   49.706556] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=52]
[   49.706558] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=53]
[   49.706559] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=54]
[   49.706561] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=55]
[   49.706563] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=56]
[   49.706564] [vq=00000000abd097cd][start=115][count=6]
[   49.706714] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=57]
[   49.706716] [vq=00000000abd097cd][start=121][count=1]
[   49.736717] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=58]
[   49.736719] [vq=00000000abd097cd][start=122][count=1]
[   49.855886] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=59]
[   49.855890] [vq=00000000abd097cd][start=123][count=1]
[   50.157344] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=60]
[   50.157348] [vq=00000000abd097cd][start=124][count=1]
[   50.464012] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=61]
[   50.464014] [vq=00000000abd097cd][start=125][count=1]
[   50.646328] [vq=000000008be108ff][vq->ndescs=1]
[   50.646330] [vq=000000008be108ff][vq->ndescs=1][vq->first_desc=1]
[   50.646331] [vq=000000008be108ff][vq->ndescs=0]
[   50.646337] [vq=000000008be108ff][start=28][count=1]
[   50.703164] [vq=000000008be108ff][vq->ndescs=2]
[   50.703165] [vq=000000008be108ff][vq->ndescs=2][vq->first_desc=2]
[   50.703166] [vq=000000008be108ff][vq->ndescs=0]
[   50.703169] [vq=000000008be108ff][start=29][count=1]
[   50.707140] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=62]
[   50.707142] [vq=00000000abd097cd][start=126][count=1]
[   50.707471] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=63]
[   50.707473] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=64]
[   50.707475] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=65]
[   50.707480] [vq=00000000abd097cd][vq->ndescs=65]
[   50.707481] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=1]
[   50.707483] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=2]
[   50.707484] [vq=00000000abd097cd][start=127][count=5]
[   50.707633] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=3]
[   50.707635] [vq=00000000abd097cd][start=132][count=1]
[   50.720693] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=4]
[   50.720695] [vq=00000000abd097cd][start=133][count=1]
[   50.776744] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=5]
[   50.776746] [vq=00000000abd097cd][start=134][count=1]
[   50.821119] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=6]
[   50.821121] [vq=00000000abd097cd][start=135][count=1]
[   50.829012] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=7]
[   50.829014] [vq=00000000abd097cd][start=136][count=1]
[   50.904533] [vq=000000008be108ff][vq->ndescs=2]
[   50.904535] [vq=000000008be108ff][vq->ndescs=2][vq->first_desc=2]
[   50.904536] [vq=000000008be108ff][vq->ndescs=0]
[   50.904540] [vq=000000008be108ff][start=30][count=1]
[   50.921538] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=8]
[   50.921540] [vq=00000000abd097cd][start=137][count=1]
[   50.930125] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=9]
[   50.930127] [vq=00000000abd097cd][start=138][count=1]
[   51.180313] [vq=000000008be108ff][vq->ndescs=2]
[   51.180315] [vq=000000008be108ff][vq->ndescs=2][vq->first_desc=2]
[   51.180316] [vq=000000008be108ff][vq->ndescs=0]
[   51.180320] [vq=000000008be108ff][start=31][count=1]
[   51.197225] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=10]
[   51.197226] [vq=00000000abd097cd][start=139][count=1]
[   51.218861] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=11]
[   51.218863] [vq=00000000abd097cd][start=140][count=1]
[   51.382675] [vq=000000008be108ff][vq->ndescs=2]
[   51.382676] [vq=000000008be108ff][vq->ndescs=2][vq->first_desc=2]
[   51.382680] [vq=000000008be108ff][vq->ndescs=0]
[   51.382682] [vq=000000008be108ff][start=32][count=1]
[   51.410246] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=12]
[   51.410247] [vq=00000000abd097cd][start=141][count=1]
[   51.503908] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=13]
[   51.503910] [vq=00000000abd097cd][start=142][count=1]
[   51.626662] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=14]
[   51.626666] [vq=00000000abd097cd][start=143][count=1]
[   51.708185] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=15]
[   51.708187] [vq=00000000abd097cd][start=144][count=1]
[   51.708530] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=16]
[   51.708532] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=17]
[   51.708534] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=18]
[   51.708536] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=19]
[   51.708538] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=20]
[   51.708540] [vq=00000000abd097cd][start=145][count=5]
[   51.708676] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=21]
[   51.708678] [vq=00000000abd097cd][start=150][count=1]
[   51.863910] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=22]
[   51.863912] [vq=00000000abd097cd][start=151][count=1]
[   51.951289] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=23]
[   51.951291] [vq=00000000abd097cd][start=152][count=1]
[   52.561591] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=24]
[   52.561594] [vq=00000000abd097cd][start=153][count=1]
[   52.564073] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=25]
[   52.564076] [vq=00000000abd097cd][start=154][count=1]
[   52.705924] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=26]
[   52.705926] [vq=00000000abd097cd][start=155][count=1]
[   52.709187] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=27]
[   52.709191] [vq=00000000abd097cd][start=156][count=1]
[   52.709509] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=28]
[   52.709511] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=29]
[   52.709513] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=30]
[   52.709515] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=31]
[   52.709516] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=32]
[   52.709518] [vq=00000000abd097cd][start=157][count=5]
[   52.709666] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=33]
[   52.709668] [vq=00000000abd097cd][start=162][count=1]
[   52.783101] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=34]
[   52.783103] [vq=00000000abd097cd][start=163][count=1]
[   52.992797] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=35]
[   52.992801] [vq=00000000abd097cd][start=164][count=1]
[   53.433538] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=36]
[   53.433542] [vq=00000000abd097cd][start=165][count=1]
[   53.710292] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=37]
[   53.710295] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=38]
[   53.710298] [vq=00000000abd097cd][start=166][count=2]
[   53.710609] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=39]
[   53.710611] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=40]
[   53.710613] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=41]
[   53.710615] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=42]
[   53.710617] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=43]
[   53.710618] [vq=00000000abd097cd][start=168][count=5]
[   53.736687] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=44]
[   53.736689] [vq=00000000abd097cd][start=173][count=1]
[   53.802369] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=45]
[   53.802372] [vq=00000000abd097cd][start=174][count=1]
[   54.014274] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=46]
[   54.014277] [vq=00000000abd097cd][start=175][count=1]
[   54.041379] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=47]
[   54.041381] [vq=00000000abd097cd][start=176][count=1]
[   54.463998] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=48]
[   54.464001] [vq=00000000abd097cd][start=177][count=1]
[   54.465467] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=49]
[   54.465469] [vq=00000000abd097cd][start=178][count=1]
[   54.646933] [vq=000000008be108ff][vq->ndescs=1]
[   54.646934] [vq=000000008be108ff][vq->ndescs=1][vq->first_desc=1]
[   54.646936] [vq=000000008be108ff][vq->ndescs=0]
[   54.646942] [vq=000000008be108ff][start=33][count=1]
[   54.711370] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=50]
[   54.711373] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=51]
[   54.711375] [vq=00000000abd097cd][start=179][count=2]
[   54.711633] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=52]
[   54.711634] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=53]
[   54.711636] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=54]
[   54.711637] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=55]
[   54.711639] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=56]
[   54.711640] [vq=00000000abd097cd][start=181][count=5]
[   54.776734] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=57]
[   54.776736] [vq=00000000abd097cd][start=186][count=1]
[   55.081427] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=58]
[   55.081430] [vq=00000000abd097cd][start=187][count=1]
[   55.121658] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=59]
[   55.121661] [vq=00000000abd097cd][start=188][count=1]
[   55.503925] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=60]
[   55.503927] [vq=00000000abd097cd][start=189][count=1]
[   55.712305] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=61]
[   55.712309] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=62]
[   55.712311] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=63]
[   55.712313] [vq=00000000abd097cd][start=190][count=3]
[   55.712437] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=64]
[   55.712438] [vq=00000000abd097cd][start=193][count=1]
[   56.034990] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=65]
[   56.034993] [vq=00000000abd097cd][start=194][count=1]
[   56.308036] [vq=00000000abd097cd][vq->ndescs=61]
[   56.308037] [vq=00000000abd097cd][vq->ndescs=61][vq->first_desc=1]
[   56.308040] [vq=00000000abd097cd][start=195][count=1]
[   56.512620] [vq=00000000abd097cd][vq->ndescs=61][vq->first_desc=2]
[   56.512622] [vq=00000000abd097cd][start=196][count=1]
[   56.707754] [vq=00000000abd097cd][vq->ndescs=61][vq->first_desc=3]
[   56.707758] [vq=00000000abd097cd][start=197][count=1]
[   56.713242] [vq=00000000abd097cd][vq->ndescs=61][vq->first_desc=4]
[   56.713244] [vq=00000000abd097cd][vq->ndescs=61][vq->first_desc=5]
[   56.713246] [vq=00000000abd097cd][start=198][count=2]
[   56.713463] [vq=00000000abd097cd][vq->ndescs=61][vq->first_desc=6]
[   56.713465] [vq=00000000abd097cd][vq->ndescs=61][vq->first_desc=7]
[   56.713467] [vq=00000000abd097cd][start=200][count=2]
[   57.238741] [vq=00000000abd097cd][vq->ndescs=61][vq->first_desc=8]
[   57.238747] [vq=00000000abd097cd][start=202][count=1]
[   57.281662] [vq=00000000abd097cd][vq->ndescs=61][vq->first_desc=9]
[   57.281664] [vq=00000000abd097cd][start=203][count=1]
[   57.347235] [vq=00000000abd097cd][vq->ndescs=61][vq->first_desc=10]
[   57.347237] [vq=00000000abd097cd][start=204][count=1]
[   57.436044] [vq=00000000abd097cd][vq->ndescs=61][vq->first_desc=11]
[   57.436047] [vq=00000000abd097cd][start=205][count=1]
[   57.714242] [vq=00000000abd097cd][vq->ndescs=61][vq->first_desc=12]
[   57.714245] [vq=00000000abd097cd][start=206][count=1]
[   57.714470] [vq=00000000abd097cd][vq->ndescs=61][vq->first_desc=13]
[   57.714472] [vq=00000000abd097cd][vq->ndescs=61][vq->first_desc=14]
[   57.714474] [vq=00000000abd097cd][vq->ndescs=61][vq->first_desc=15]
[   57.714476] [vq=00000000abd097cd][vq->ndescs=61][vq->first_desc=16]
[   57.714478] [vq=00000000abd097cd][start=207][count=4]
[   57.736863] [vq=00000000abd097cd][vq->ndescs=61][vq->first_desc=17]
[   57.736865] [vq=00000000abd097cd][start=211][count=1]
[   58.369315] [vq=00000000abd097cd][vq->ndescs=61][vq->first_desc=18]
[   58.369319] [vq=00000000abd097cd][start=212][count=1]
[   58.387247] [vq=00000000abd097cd][vq->ndescs=61][vq->first_desc=19]
[   58.387249] [vq=00000000abd097cd][start=213][count=1]
[   58.464066] [vq=00000000abd097cd][vq->ndescs=61][vq->first_desc=20]
[   58.464071] [vq=00000000abd097cd][start=214][count=1]
[   58.715327] [vq=00000000abd097cd][vq->ndescs=61][vq->first_desc=21]
[   58.715330] [vq=00000000abd097cd][vq->ndescs=61][vq->first_desc=22]
[   58.715333] [vq=00000000abd097cd][vq->ndescs=61][vq->first_desc=23]
[   58.715335] [vq=00000000abd097cd][start=215][count=3]
[   58.715525] [vq=00000000abd097cd][vq->ndescs=61][vq->first_desc=24]
[   58.715527] [vq=00000000abd097cd][vq->ndescs=61][vq->first_desc=25]
[   58.715529] [vq=00000000abd097cd][start=218][count=2]
[   58.776675] [vq=00000000abd097cd][vq->ndescs=61][vq->first_desc=26]
[   58.776677] [vq=00000000abd097cd][start=220][count=1]
[   59.427685] [vq=00000000abd097cd][vq->ndescs=61][vq->first_desc=27]
[   59.427689] [vq=00000000abd097cd][start=221][count=1]
[   59.473816] [vq=00000000abd097cd][vq->ndescs=61][vq->first_desc=28]
[   59.473819] [vq=00000000abd097cd][start=222][count=1]
[   59.503898] [vq=00000000abd097cd][vq->ndescs=61][vq->first_desc=29]
[   59.503901] [vq=00000000abd097cd][start=223][count=1]
[   59.646298] [vq=00000000abd097cd][vq->ndescs=61][vq->first_desc=30]
[   59.646302] [vq=00000000abd097cd][start=224][count=1]
[   59.716224] [vq=00000000abd097cd][vq->ndescs=61][vq->first_desc=31]
[   59.716226] [vq=00000000abd097cd][start=225][count=1]
[   59.716457] [vq=00000000abd097cd][vq->ndescs=61][vq->first_desc=32]
[   59.716459] [vq=00000000abd097cd][vq->ndescs=61][vq->first_desc=33]
[   59.716461] [vq=00000000abd097cd][vq->ndescs=61][vq->first_desc=34]
[   59.716463] [vq=00000000abd097cd][start=226][count=3]
[   60.467226] [vq=00000000abd097cd][vq->ndescs=61][vq->first_desc=35]
[   60.467229] [vq=00000000abd097cd][start=229][count=1]
[   60.710063] [vq=00000000abd097cd][vq->ndescs=61][vq->first_desc=36]
[   60.710066] [vq=00000000abd097cd][start=230][count=1]
[   60.717198] [vq=00000000abd097cd][vq->ndescs=61][vq->first_desc=37]
[   60.717200] [vq=00000000abd097cd][vq->ndescs=61][vq->first_desc=38]
[   60.717201] [vq=00000000abd097cd][start=231][count=2]
[   60.717438] [vq=00000000abd097cd][vq->ndescs=61][vq->first_desc=39]
[   60.717440] [vq=00000000abd097cd][vq->ndescs=61][vq->first_desc=40]
[   60.717442] [vq=00000000abd097cd][start=233][count=2]
[   61.001117] [vq=00000000abd097cd][vq->ndescs=61][vq->first_desc=41]
[   61.001119] [vq=00000000abd097cd][start=235][count=1]
[   61.057080] [vq=00000000abd097cd][vq->ndescs=61][vq->first_desc=42]
[   61.057082] [vq=00000000abd097cd][start=236][count=1]
[   61.382703] [vq=00000000abd097cd][vq->ndescs=61][vq->first_desc=43]
[   61.382707] [vq=00000000abd097cd][start=237][count=1]
[   61.438023] [vq=00000000abd097cd][vq->ndescs=61][vq->first_desc=44]
[   61.438025] [vq=00000000abd097cd][start=238][count=1]
[   61.507231] [vq=00000000abd097cd][vq->ndescs=61][vq->first_desc=45]
[   61.507234] [vq=00000000abd097cd][start=239][count=1]
[   61.718344] [vq=00000000abd097cd][vq->ndescs=61][vq->first_desc=46]
[   61.718348] [vq=00000000abd097cd][vq->ndescs=61][vq->first_desc=47]
[   61.718349] [vq=00000000abd097cd][vq->ndescs=61][vq->first_desc=48]
[   61.718351] [vq=00000000abd097cd][start=240][count=3]
[   61.718483] [vq=00000000abd097cd][vq->ndescs=61][vq->first_desc=49]
[   61.718484] [vq=00000000abd097cd][start=243][count=1]
[   61.736698] [vq=00000000abd097cd][vq->ndescs=61][vq->first_desc=50]
[   61.736700] [vq=00000000abd097cd][start=244][count=1]
[   61.953423] [vq=00000000abd097cd][vq->ndescs=61][vq->first_desc=51]
[   61.953425] [vq=00000000abd097cd][start=245][count=1]
[   62.464045] [vq=00000000abd097cd][vq->ndescs=61][vq->first_desc=52]
[   62.464049] [vq=00000000abd097cd][start=246][count=1]
[   62.719113] [vq=00000000abd097cd][vq->ndescs=61][vq->first_desc=53]
[   62.719115] [vq=00000000abd097cd][start=247][count=1]
[   62.719328] [vq=00000000abd097cd][vq->ndescs=61][vq->first_desc=54]
[   62.719330] [vq=00000000abd097cd][vq->ndescs=61][vq->first_desc=55]
[   62.719331] [vq=00000000abd097cd][start=248][count=2]
[   62.776707] [vq=00000000abd097cd][vq->ndescs=61][vq->first_desc=56]
[   62.776709] [vq=00000000abd097cd][start=250][count=1]
[   63.503845] [vq=00000000abd097cd][vq->ndescs=61][vq->first_desc=57]
[   63.503849] [vq=00000000abd097cd][start=251][count=1]
[   63.720279] [vq=00000000abd097cd][vq->ndescs=61][vq->first_desc=58]
[   63.720282] [vq=00000000abd097cd][vq->ndescs=61][vq->first_desc=59]
[   63.720283] [vq=00000000abd097cd][start=252][count=2]
[   63.720442] [vq=00000000abd097cd][vq->ndescs=61][vq->first_desc=60]
[   63.720444] [vq=00000000abd097cd][start=254][count=1]
[   64.620475] [vq=00000000abd097cd][vq->ndescs=61][vq->first_desc=61]
[   64.620479] [vq=00000000abd097cd][start=255][count=1]
[   64.711456] [vq=00000000abd097cd][vq->ndescs=65]
[   64.711461] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=1]
[   64.711466] [vq=00000000abd097cd][start=0][count=1]
[   64.721336] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=2]
[   64.721338] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=3]
[   64.721340] [vq=00000000abd097cd][start=1][count=2]
[   65.440290] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=4]
[   65.440295] [vq=00000000abd097cd][start=3][count=1]
[   65.493920] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=5]
[   65.493922] [vq=00000000abd097cd][start=4][count=1]
[   65.722216] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=6]
[   65.722221] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=7]
[   65.722223] [vq=00000000abd097cd][start=5][count=2]
[   65.736673] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=8]
[   65.736675] [vq=00000000abd097cd][start=7][count=1]
[   65.812061] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=9]
[   65.812063] [vq=00000000abd097cd][start=8][count=1]
[   65.954015] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=10]
[   65.954017] [vq=00000000abd097cd][start=9][count=1]
[   66.463948] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=11]
[   66.463953] [vq=00000000abd097cd][start=10][count=1]
[   66.723295] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=12]
[   66.723297] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=13]
[   66.723298] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=14]
[   66.723300] [vq=00000000abd097cd][start=11][count=3]
[   66.776726] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=15]
[   66.776728] [vq=00000000abd097cd][start=14][count=1]
[   66.831446] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=16]
[   66.831449] [vq=00000000abd097cd][start=15][count=1]
[   67.154905] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=17]
[   67.154908] [vq=00000000abd097cd][start=16][count=1]
[   67.202734] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=18]
[   67.202737] [vq=00000000abd097cd][start=17][count=1]
[   67.299794] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=19]
[   67.299797] [vq=00000000abd097cd][start=18][count=1]
[   67.503931] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=20]
[   67.503934] [vq=00000000abd097cd][start=19][count=1]
[   67.724138] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=21]
[   67.724140] [vq=00000000abd097cd][start=20][count=1]
[   67.724307] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=22]
[   67.724308] [vq=00000000abd097cd][start=21][count=1]
[   67.871496] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=23]
[   67.871499] [vq=00000000abd097cd][start=22][count=1]
[   68.227397] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=24]
[   68.227399] [vq=00000000abd097cd][start=23][count=1]
[   68.625622] [vq=000000008be108ff][vq->ndescs=1]
[   68.625624] [vq=000000008be108ff][vq->ndescs=1][vq->first_desc=1]
[   68.625627] [vq=000000008be108ff][vq->ndescs=0]
[   68.625632] [vq=000000008be108ff][start=34][count=1]
[   68.625897] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=25]
[   68.625899] [vq=00000000abd097cd][start=24][count=1]
[   68.625920] [vq=000000008be108ff][vq->ndescs=1]
[   68.625921] [vq=000000008be108ff][vq->ndescs=1][vq->first_desc=1]
[   68.625923] [vq=000000008be108ff][vq->ndescs=0]
[   68.625925] [vq=000000008be108ff][start=35][count=1]
[   68.626108] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=26]
[   68.626109] [vq=00000000abd097cd][start=25][count=1]
[   68.626163] [vq=000000008be108ff][vq->ndescs=1]
[   68.626164] [vq=000000008be108ff][vq->ndescs=1][vq->first_desc=1]
[   68.626165] [vq=000000008be108ff][vq->ndescs=0]
[   68.626168] [vq=000000008be108ff][start=36][count=1]
[   68.626309] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=27]
[   68.626311] [vq=00000000abd097cd][start=26][count=1]
[   68.626348] [vq=000000008be108ff][vq->ndescs=1]
[   68.626349] [vq=000000008be108ff][vq->ndescs=1][vq->first_desc=1]
[   68.626351] [vq=000000008be108ff][vq->ndescs=0]
[   68.626353] [vq=000000008be108ff][start=37][count=1]
[   68.626489] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=28]
[   68.626491] [vq=00000000abd097cd][start=27][count=1]
[   68.626527] [vq=000000008be108ff][vq->ndescs=1]
[   68.626528] [vq=000000008be108ff][vq->ndescs=1][vq->first_desc=1]
[   68.626529] [vq=000000008be108ff][vq->ndescs=0]
[   68.626531] [vq=000000008be108ff][start=38][count=1]
[   68.626668] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=29]
[   68.626669] [vq=00000000abd097cd][start=28][count=1]
[   68.626705] [vq=000000008be108ff][vq->ndescs=1]
[   68.626706] [vq=000000008be108ff][vq->ndescs=1][vq->first_desc=1]
[   68.626707] [vq=000000008be108ff][vq->ndescs=0]
[   68.626710] [vq=000000008be108ff][start=39][count=1]
[   68.626848] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=30]
[   68.626850] [vq=00000000abd097cd][start=29][count=1]
[   68.626905] [vq=000000008be108ff][vq->ndescs=1]
[   68.626906] [vq=000000008be108ff][vq->ndescs=1][vq->first_desc=1]
[   68.626907] [vq=000000008be108ff][vq->ndescs=0]
[   68.626910] [vq=000000008be108ff][start=40][count=1]
[   68.627046] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=31]
[   68.627048] [vq=00000000abd097cd][start=30][count=1]
[   68.627083] [vq=000000008be108ff][vq->ndescs=1]
[   68.627084] [vq=000000008be108ff][vq->ndescs=1][vq->first_desc=1]
[   68.627086] [vq=000000008be108ff][vq->ndescs=0]
[   68.627088] [vq=000000008be108ff][start=41][count=1]
[   68.627225] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=32]
[   68.627227] [vq=00000000abd097cd][start=31][count=1]
[   68.627263] [vq=000000008be108ff][vq->ndescs=1]
[   68.627264] [vq=000000008be108ff][vq->ndescs=1][vq->first_desc=1]
[   68.627265] [vq=000000008be108ff][vq->ndescs=0]
[   68.627267] [vq=000000008be108ff][start=42][count=1]
[   68.627412] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=33]
[   68.627413] [vq=00000000abd097cd][start=32][count=1]
[   68.627455] [vq=000000008be108ff][vq->ndescs=1]
[   68.627456] [vq=000000008be108ff][vq->ndescs=1][vq->first_desc=1]
[   68.627458] [vq=000000008be108ff][vq->ndescs=0]
[   68.627461] [vq=000000008be108ff][start=43][count=1]
[   68.627597] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=34]
[   68.627598] [vq=00000000abd097cd][start=33][count=1]
[   68.627632] [vq=000000008be108ff][vq->ndescs=1]
[   68.627633] [vq=000000008be108ff][vq->ndescs=1][vq->first_desc=1]
[   68.627634] [vq=000000008be108ff][vq->ndescs=0]
[   68.627636] [vq=000000008be108ff][start=44][count=1]
[   68.627788] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=35]
[   68.627791] [vq=00000000abd097cd][start=34][count=1]
[   68.627840] [vq=000000008be108ff][vq->ndescs=1]
[   68.627841] [vq=000000008be108ff][vq->ndescs=1][vq->first_desc=1]
[   68.627843] [vq=000000008be108ff][vq->ndescs=0]
[   68.627845] [vq=000000008be108ff][start=45][count=1]
[   68.627976] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=36]
[   68.627978] [vq=00000000abd097cd][start=35][count=1]
[   68.628011] [vq=000000008be108ff][vq->ndescs=1]
[   68.628012] [vq=000000008be108ff][vq->ndescs=1][vq->first_desc=1]
[   68.628013] [vq=000000008be108ff][vq->ndescs=0]
[   68.628016] [vq=000000008be108ff][start=46][count=1]
[   68.628145] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=37]
[   68.628147] [vq=00000000abd097cd][start=36][count=1]
[   68.628178] [vq=000000008be108ff][vq->ndescs=1]
[   68.628179] [vq=000000008be108ff][vq->ndescs=1][vq->first_desc=1]
[   68.628181] [vq=000000008be108ff][vq->ndescs=0]
[   68.628183] [vq=000000008be108ff][start=47][count=1]
[   68.628311] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=38]
[   68.628313] [vq=00000000abd097cd][start=37][count=1]
[   68.628348] [vq=000000008be108ff][vq->ndescs=1]
[   68.628349] [vq=000000008be108ff][vq->ndescs=1][vq->first_desc=1]
[   68.628350] [vq=000000008be108ff][vq->ndescs=0]
[   68.628353] [vq=000000008be108ff][start=48][count=1]
[   68.628481] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=39]
[   68.628483] [vq=00000000abd097cd][start=38][count=1]
[   68.628517] [vq=000000008be108ff][vq->ndescs=1]
[   68.628518] [vq=000000008be108ff][vq->ndescs=1][vq->first_desc=1]
[   68.628520] [vq=000000008be108ff][vq->ndescs=0]
[   68.628522] [vq=000000008be108ff][start=49][count=1]
[   68.628648] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=40]
[   68.628650] [vq=00000000abd097cd][start=39][count=1]
[   68.628683] [vq=000000008be108ff][vq->ndescs=1]
[   68.628684] [vq=000000008be108ff][vq->ndescs=1][vq->first_desc=1]
[   68.628686] [vq=000000008be108ff][vq->ndescs=0]
[   68.628688] [vq=000000008be108ff][start=50][count=1]
[   68.628814] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=41]
[   68.628816] [vq=00000000abd097cd][start=40][count=1]
[   68.628847] [vq=000000008be108ff][vq->ndescs=1]
[   68.628848] [vq=000000008be108ff][vq->ndescs=1][vq->first_desc=1]
[   68.628849] [vq=000000008be108ff][vq->ndescs=0]
[   68.628851] [vq=000000008be108ff][start=51][count=1]
[   68.628977] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=42]
[   68.628979] [vq=00000000abd097cd][start=41][count=1]
[   68.629012] [vq=000000008be108ff][vq->ndescs=1]
[   68.629013] [vq=000000008be108ff][vq->ndescs=1][vq->first_desc=1]
[   68.629014] [vq=000000008be108ff][vq->ndescs=0]
[   68.629017] [vq=000000008be108ff][start=52][count=1]
[   68.629143] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=43]
[   68.629144] [vq=00000000abd097cd][start=42][count=1]
[   68.629177] [vq=000000008be108ff][vq->ndescs=1]
[   68.629178] [vq=000000008be108ff][vq->ndescs=1][vq->first_desc=1]
[   68.629179] [vq=000000008be108ff][vq->ndescs=0]
[   68.629182] [vq=000000008be108ff][start=53][count=1]
[   68.629310] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=44]
[   68.629311] [vq=00000000abd097cd][start=43][count=1]
[   68.629344] [vq=000000008be108ff][vq->ndescs=1]
[   68.629345] [vq=000000008be108ff][vq->ndescs=1][vq->first_desc=1]
[   68.629346] [vq=000000008be108ff][vq->ndescs=0]
[   68.629349] [vq=000000008be108ff][start=54][count=1]
[   68.629490] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=45]
[   68.629495] [vq=00000000abd097cd][start=44][count=1]
[   68.713536] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=46]
[   68.713539] [vq=00000000abd097cd][start=45][count=1]
[   68.725293] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=47]
[   68.725295] [vq=00000000abd097cd][start=46][count=1]
[   68.748355] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=48]
[   68.748359] [vq=00000000abd097cd][start=47][count=1]
[   68.805702] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=49]
[   68.805704] [vq=00000000abd097cd][start=48][count=1]
[   68.911555] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=50]
[   68.911558] [vq=00000000abd097cd][start=49][count=1]
[   69.133394] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=51]
[   69.133397] [vq=00000000abd097cd][start=50][count=1]
[   69.267403] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=52]
[   69.267405] [vq=00000000abd097cd][start=51][count=1]
[   69.442123] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=53]
[   69.442127] [vq=00000000abd097cd][start=52][count=1]
[   69.458741] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=54]
[   69.458743] [vq=00000000abd097cd][start=53][count=1]
[   69.726266] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=55]
[   69.726270] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=56]
[   69.726272] [vq=00000000abd097cd][start=54][count=2]
[   69.736761] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=57]
[   69.736764] [vq=00000000abd097cd][start=56][count=1]
[   69.826013] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=58]
[   69.826014] [vq=00000000abd097cd][start=57][count=1]
[   69.951440] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=59]
[   69.951442] [vq=00000000abd097cd][start=58][count=1]
[   70.110704] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=60]
[   70.110706] [vq=00000000abd097cd][start=59][count=1]
[   70.317450] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=61]
[   70.317452] [vq=00000000abd097cd][start=60][count=1]
[   70.463806] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=62]
[   70.463808] [vq=00000000abd097cd][start=61][count=1]
[   70.491573] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=63]
[   70.491575] [vq=00000000abd097cd][start=62][count=1]
[   70.727266] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=64]
[   70.727269] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=65]
[   70.727271] [vq=00000000abd097cd][start=63][count=2]
[   70.776670] [vq=00000000abd097cd][vq->ndescs=65]
[   70.776671] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=1]
[   70.776673] [vq=00000000abd097cd][start=65][count=1]
[   70.991537] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=2]
[   70.991539] [vq=00000000abd097cd][start=66][count=1]
[   71.208420] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=3]
[   71.208423] [vq=00000000abd097cd][start=67][count=1]
[   71.347437] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=4]
[   71.347439] [vq=00000000abd097cd][start=68][count=1]
[   71.503869] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=5]
[   71.503871] [vq=00000000abd097cd][start=69][count=1]
[   71.728241] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=6]
[   71.728244] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=7]
[   71.728246] [vq=00000000abd097cd][start=70][count=2]
[   74.661409] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=8]
[   74.661417] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=9]
[   74.661419] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=10]
[   74.661421] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=11]
[   74.661423] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=12]
[   74.661424] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=13]
[   74.661425] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=14]
[   74.661427] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=15]
[   74.661429] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=16]
[   74.661432] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=17]
[   74.661433] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=18]
[   74.661435] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=19]
[   74.661437] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=20]
[   74.661438] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=21]
[   74.661440] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=22]
[   74.661442] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=23]
[   74.661444] [vq=00000000abd097cd][vq->ndescs=65][vq->first_desc=24]
[   74.661446] [vq=00000000abd097cd][start=0][count=17]

Re: vhost changes (batched) in linux-next after 12/13 trigger random crashes in KVM guests after reboot

[Christian Borntraeger]

[-- Attachment #1: Type: text/plain, Size: 816 bytes --]



On 11.02.20 10:56, Christian Borntraeger wrote:
> 
> 
> On 11.02.20 10:33, Eugenio Pérez wrote:
>> On Mon, 2020-02-10 at 12:01 +0100, Christian Borntraeger wrote:
>>>
>>> On 10.02.20 10:47, Eugenio Perez Martin wrote:
>>>> Hi Christian.
>>>>
>>>> I'm not able to reproduce the failure with eccb852f1fe6bede630e2e4f1a121a81e34354ab commit. Could you add more data?
>>>> Your configuration (libvirt or qemu line), and host's dmesg output if any?
>>>>
>>>> Thanks!
>>>
>>> If it was not obvious, this is on s390x, a big endian system.
>>>
>>
>> Hi Christian. Thank you very much for the hints.
>>
>> Could we add some debug traces? Something like the inline patch should give us some clues.
> 
> attached. this time the guest crashed (and no guest moved message)
> 

And here with Guest moved vq after some reboots.

[-- Attachment #2: log2 --]
[-- Type: text/plain, Size: 72381 bytes --]

[   74.661450] [vq=000000008be108ff][vq->ndescs=0]
[  757.654136] [vq=00000000201c16ab][vq->ndescs=0]
[  757.654142] [vq=00000000465a3846][vq->ndescs=0]
[  758.283378] [vq=00000000201c16ab][vq->ndescs=65]
[  758.283381] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=1]
[  758.283385] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=2]
[  758.283386] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=3]
[  758.283388] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=4]
[  758.283390] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=5]
[  758.283391] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=6]
[  758.283393] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=7]
[  758.283394] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=8]
[  758.283397] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=9]
[  758.283399] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=10]
[  758.283401] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=11]
[  758.283403] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=12]
[  758.283406] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=13]
[  758.283407] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=14]
[  758.283409] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=15]
[  758.283410] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=16]
[  758.283413] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=17]
[  758.283414] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=18]
[  758.283417] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=19]
[  758.283418] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=20]
[  758.283420] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=21]
[  758.283422] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=22]
[  758.283423] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=23]
[  758.283425] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=24]
[  758.283426] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=25]
[  758.283428] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=26]
[  758.283429] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=27]
[  758.283431] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=28]
[  758.283433] [vq=00000000201c16ab][start=0][count=28]
[  758.344947] [vq=00000000465a3846][vq->ndescs=1]
[  758.344948] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  758.344951] [vq=00000000465a3846][vq->ndescs=0]
[  758.344962] [vq=00000000465a3846][start=0][count=1]
[  758.366414] [vq=00000000465a3846][vq->ndescs=1]
[  758.366415] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  758.366417] [vq=00000000465a3846][vq->ndescs=0]
[  758.366420] [vq=00000000465a3846][start=1][count=1]
[  758.366913] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=29]
[  758.366915] [vq=00000000201c16ab][start=28][count=1]
[  758.366917] [vq=00000000465a3846][vq->ndescs=1]
[  758.366918] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  758.366919] [vq=00000000465a3846][vq->ndescs=0]
[  758.366924] [vq=00000000465a3846][start=2][count=1]
[  758.366943] [vq=00000000465a3846][vq->ndescs=2]
[  758.366944] [vq=00000000465a3846][vq->ndescs=2][vq->first_desc=1]
[  758.366946] [vq=00000000465a3846][vq->ndescs=2][vq->first_desc=2]
[  758.366947] [vq=00000000465a3846][vq->ndescs=0]
[  758.366951] [vq=00000000465a3846][start=3][count=2]
[  758.369847] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=30]
[  758.369852] [vq=00000000201c16ab][start=29][count=1]
[  758.382143] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=31]
[  758.382145] [vq=00000000201c16ab][start=30][count=1]
[  758.421531] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=32]
[  758.421534] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=33]
[  758.421536] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=34]
[  758.421537] [vq=00000000201c16ab][start=31][count=3]
[  758.480414] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=35]
[  758.480418] [vq=00000000201c16ab][start=34][count=1]
[  758.584968] [vq=00000000465a3846][vq->ndescs=2]
[  758.584973] [vq=00000000465a3846][vq->ndescs=2][vq->first_desc=2]
[  758.584975] [vq=00000000465a3846][vq->ndescs=0]
[  758.584987] [vq=00000000465a3846][start=5][count=1]
[  758.629880] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=36]
[  758.629883] [vq=00000000201c16ab][start=35][count=1]
[  758.696982] [vq=00000000465a3846][vq->ndescs=1]
[  758.696985] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  758.696987] [vq=00000000465a3846][vq->ndescs=0]
[  758.696994] [vq=00000000465a3846][start=6][count=1]
[  758.788476] [vq=00000000465a3846][vq->ndescs=2]
[  758.788478] [vq=00000000465a3846][vq->ndescs=2][vq->first_desc=2]
[  758.788480] [vq=00000000465a3846][vq->ndescs=0]
[  758.788488] [vq=00000000465a3846][start=7][count=1]
[  758.818471] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=37]
[  758.818475] [vq=00000000201c16ab][start=36][count=1]
[  758.864048] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=38]
[  758.864051] [vq=00000000201c16ab][start=37][count=1]
[  758.912219] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=39]
[  758.912221] [vq=00000000201c16ab][start=38][count=1]
[  758.916937] [vq=00000000465a3846][vq->ndescs=1]
[  758.916939] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  758.916940] [vq=00000000465a3846][vq->ndescs=0]
[  758.916949] [vq=00000000465a3846][start=8][count=1]
[  758.992422] [vq=00000000465a3846][vq->ndescs=2]
[  758.992425] [vq=00000000465a3846][vq->ndescs=2][vq->first_desc=2]
[  758.992426] [vq=00000000465a3846][vq->ndescs=0]
[  758.992431] [vq=00000000465a3846][start=9][count=1]
[  759.026942] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=40]
[  759.026944] [vq=00000000201c16ab][start=39][count=1]
[  759.106290] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=41]
[  759.106293] [vq=00000000201c16ab][start=40][count=1]
[  759.176843] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=42]
[  759.176846] [vq=00000000201c16ab][start=41][count=1]
[  759.192881] [vq=00000000465a3846][vq->ndescs=2]
[  759.192884] [vq=00000000465a3846][vq->ndescs=2][vq->first_desc=2]
[  759.192886] [vq=00000000465a3846][vq->ndescs=0]
[  759.192896] [vq=00000000465a3846][start=10][count=1]
[  759.215142] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=43]
[  759.215144] [vq=00000000201c16ab][start=42][count=1]
[  759.277701] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=44]
[  759.277704] [vq=00000000201c16ab][start=43][count=1]
[  759.422515] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=45]
[  759.422518] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=46]
[  759.422520] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=47]
[  759.422521] [vq=00000000201c16ab][start=44][count=3]
[  759.422792] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=48]
[  759.422794] [vq=00000000201c16ab][start=47][count=1]
[  759.626994] [vq=00000000465a3846][vq->ndescs=1]
[  759.626997] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  759.626999] [vq=00000000465a3846][vq->ndescs=0]
[  759.627008] [vq=00000000465a3846][start=11][count=1]
[  759.904224] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=49]
[  759.904232] [vq=00000000201c16ab][start=48][count=1]
[  759.945942] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=50]
[  759.945944] [vq=00000000201c16ab][start=49][count=1]
[  759.951344] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=51]
[  759.951346] [vq=00000000201c16ab][start=50][count=1]
[  760.307838] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=52]
[  760.307841] [vq=00000000201c16ab][start=51][count=1]
[  760.344981] [vq=00000000465a3846][vq->ndescs=1]
[  760.344983] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  760.344985] [vq=00000000465a3846][vq->ndescs=0]
[  760.344994] [vq=00000000465a3846][start=12][count=1]
[  760.351941] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=53]
[  760.351943] [vq=00000000201c16ab][start=52][count=1]
[  760.423655] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=54]
[  760.423657] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=55]
[  760.423658] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=56]
[  760.423659] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=57]
[  760.423661] [vq=00000000201c16ab][start=53][count=4]
[  760.423903] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=58]
[  760.423905] [vq=00000000201c16ab][start=57][count=1]
[  760.633527] [vq=00000000465a3846][vq->ndescs=2]
[  760.633529] [vq=00000000465a3846][vq->ndescs=2][vq->first_desc=2]
[  760.633530] [vq=00000000465a3846][vq->ndescs=0]
[  760.633535] [vq=00000000465a3846][start=13][count=1]
[  760.666987] [vq=00000000465a3846][vq->ndescs=1]
[  760.666988] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  760.666990] [vq=00000000465a3846][vq->ndescs=0]
[  760.666993] [vq=00000000465a3846][start=14][count=1]
[  760.672771] [vq=00000000465a3846][vq->ndescs=1]
[  760.672772] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  760.672773] [vq=00000000465a3846][vq->ndescs=0]
[  760.672777] [vq=00000000465a3846][start=15][count=1]
[  760.673253] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=59]
[  760.673254] [vq=00000000201c16ab][start=58][count=1]
[  760.673277] [vq=00000000465a3846][vq->ndescs=1]
[  760.673278] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  760.673279] [vq=00000000465a3846][vq->ndescs=0]
[  760.673282] [vq=00000000465a3846][start=16][count=1]
[  760.673475] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=60]
[  760.673476] [vq=00000000201c16ab][start=59][count=1]
[  760.673592] [vq=00000000465a3846][vq->ndescs=1]
[  760.673593] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  760.673594] [vq=00000000465a3846][vq->ndescs=0]
[  760.673597] [vq=00000000465a3846][start=17][count=1]
[  760.673751] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=61]
[  760.673753] [vq=00000000201c16ab][start=60][count=1]
[  760.673800] [vq=00000000465a3846][vq->ndescs=1]
[  760.673801] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  760.673803] [vq=00000000465a3846][vq->ndescs=0]
[  760.673805] [vq=00000000465a3846][start=18][count=1]
[  760.673944] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=62]
[  760.673946] [vq=00000000201c16ab][start=61][count=1]
[  760.673986] [vq=00000000465a3846][vq->ndescs=1]
[  760.673987] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  760.673989] [vq=00000000465a3846][vq->ndescs=0]
[  760.673992] [vq=00000000465a3846][start=19][count=1]
[  760.674132] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=63]
[  760.674133] [vq=00000000201c16ab][start=62][count=1]
[  760.674175] [vq=00000000465a3846][vq->ndescs=1]
[  760.674176] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  760.674177] [vq=00000000465a3846][vq->ndescs=0]
[  760.674180] [vq=00000000465a3846][start=20][count=1]
[  760.674316] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=64]
[  760.674318] [vq=00000000201c16ab][start=63][count=1]
[  760.674358] [vq=00000000465a3846][vq->ndescs=1]
[  760.674359] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  760.674360] [vq=00000000465a3846][vq->ndescs=0]
[  760.674363] [vq=00000000465a3846][start=21][count=1]
[  760.674502] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=65]
[  760.674504] [vq=00000000201c16ab][start=64][count=1]
[  760.674548] [vq=00000000465a3846][vq->ndescs=1]
[  760.674549] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  760.674550] [vq=00000000465a3846][vq->ndescs=0]
[  760.674553] [vq=00000000465a3846][start=22][count=1]
[  760.674692] [vq=00000000201c16ab][vq->ndescs=65]
[  760.674693] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=1]
[  760.674694] [vq=00000000201c16ab][start=65][count=1]
[  760.674734] [vq=00000000465a3846][vq->ndescs=1]
[  760.674735] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  760.674737] [vq=00000000465a3846][vq->ndescs=0]
[  760.674739] [vq=00000000465a3846][start=23][count=1]
[  760.674877] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=2]
[  760.674878] [vq=00000000201c16ab][start=66][count=1]
[  760.674926] [vq=00000000465a3846][vq->ndescs=1]
[  760.674927] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  760.674929] [vq=00000000465a3846][vq->ndescs=0]
[  760.674931] [vq=00000000465a3846][start=24][count=1]
[  760.675082] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=3]
[  760.675083] [vq=00000000201c16ab][start=67][count=1]
[  760.675124] [vq=00000000465a3846][vq->ndescs=1]
[  760.675125] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  760.675127] [vq=00000000465a3846][vq->ndescs=0]
[  760.675129] [vq=00000000465a3846][start=25][count=1]
[  760.675266] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=4]
[  760.675268] [vq=00000000201c16ab][start=68][count=1]
[  760.675320] [vq=00000000465a3846][vq->ndescs=1]
[  760.675321] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  760.675322] [vq=00000000465a3846][vq->ndescs=0]
[  760.675325] [vq=00000000465a3846][start=26][count=1]
[  760.675465] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=5]
[  760.675466] [vq=00000000201c16ab][start=69][count=1]
[  760.675508] [vq=00000000465a3846][vq->ndescs=1]
[  760.675509] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  760.675510] [vq=00000000465a3846][vq->ndescs=0]
[  760.675513] [vq=00000000465a3846][start=27][count=1]
[  760.675659] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=6]
[  760.675661] [vq=00000000201c16ab][start=70][count=1]
[  760.675703] [vq=00000000465a3846][vq->ndescs=1]
[  760.675704] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  760.675706] [vq=00000000465a3846][vq->ndescs=0]
[  760.675708] [vq=00000000465a3846][start=28][count=1]
[  760.675845] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=7]
[  760.675846] [vq=00000000201c16ab][start=71][count=1]
[  760.675888] [vq=00000000465a3846][vq->ndescs=1]
[  760.675889] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  760.675890] [vq=00000000465a3846][vq->ndescs=0]
[  760.675893] [vq=00000000465a3846][start=29][count=1]
[  760.676031] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=8]
[  760.676032] [vq=00000000201c16ab][start=72][count=1]
[  760.676073] [vq=00000000465a3846][vq->ndescs=1]
[  760.676074] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  760.676076] [vq=00000000465a3846][vq->ndescs=0]
[  760.676079] [vq=00000000465a3846][start=30][count=1]
[  760.676220] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=9]
[  760.676221] [vq=00000000201c16ab][start=73][count=1]
[  760.676261] [vq=00000000465a3846][vq->ndescs=1]
[  760.676262] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  760.676263] [vq=00000000465a3846][vq->ndescs=0]
[  760.676266] [vq=00000000465a3846][start=31][count=1]
[  760.676406] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=10]
[  760.676407] [vq=00000000201c16ab][start=74][count=1]
[  760.676447] [vq=00000000465a3846][vq->ndescs=1]
[  760.676448] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  760.676450] [vq=00000000465a3846][vq->ndescs=0]
[  760.676452] [vq=00000000465a3846][start=32][count=1]
[  760.676590] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=11]
[  760.676591] [vq=00000000201c16ab][start=75][count=1]
[  760.676631] [vq=00000000465a3846][vq->ndescs=1]
[  760.676632] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  760.676634] [vq=00000000465a3846][vq->ndescs=0]
[  760.676636] [vq=00000000465a3846][start=33][count=1]
[  760.676775] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=12]
[  760.676777] [vq=00000000201c16ab][start=76][count=1]
[  760.676817] [vq=00000000465a3846][vq->ndescs=1]
[  760.676818] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  760.676819] [vq=00000000465a3846][vq->ndescs=0]
[  760.676822] [vq=00000000465a3846][start=34][count=1]
[  760.676961] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=13]
[  760.676963] [vq=00000000201c16ab][start=77][count=1]
[  760.677005] [vq=00000000465a3846][vq->ndescs=1]
[  760.677006] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  760.677007] [vq=00000000465a3846][vq->ndescs=0]
[  760.677011] [vq=00000000465a3846][start=35][count=1]
[  760.677152] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=14]
[  760.677153] [vq=00000000201c16ab][start=78][count=1]
[  760.678377] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=15]
[  760.678379] [vq=00000000201c16ab][start=79][count=1]
[  760.696916] [vq=00000000465a3846][vq->ndescs=1]
[  760.696918] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  760.696919] [vq=00000000465a3846][vq->ndescs=0]
[  760.696922] [vq=00000000465a3846][start=36][count=1]
[  760.731769] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=16]
[  760.731771] [vq=00000000201c16ab][start=80][count=1]
[  760.906976] [vq=00000000465a3846][vq->ndescs=1]
[  760.906980] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  760.906982] [vq=00000000465a3846][vq->ndescs=0]
[  760.906992] [vq=00000000465a3846][start=37][count=1]
[  761.001377] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=17]
[  761.001383] [vq=00000000201c16ab][start=81][count=1]
[  761.025300] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=18]
[  761.025304] [vq=00000000201c16ab][start=82][count=1]
[  761.088194] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=19]
[  761.088196] [vq=00000000201c16ab][start=83][count=1]
[  761.096984] [vq=00000000465a3846][vq->ndescs=1]
[  761.096985] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  761.096987] [vq=00000000465a3846][vq->ndescs=0]
[  761.096995] [vq=00000000465a3846][start=38][count=1]
[  761.347842] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=20]
[  761.347846] [vq=00000000201c16ab][start=84][count=1]
[  761.424800] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=21]
[  761.424803] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=22]
[  761.424805] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=23]
[  761.424807] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=24]
[  761.424809] [vq=00000000201c16ab][start=85][count=4]
[  761.848308] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=25]
[  761.848310] [vq=00000000201c16ab][start=89][count=1]
[  761.926827] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=26]
[  761.926829] [vq=00000000201c16ab][start=90][count=1]
[  761.960724] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=27]
[  761.960727] [vq=00000000201c16ab][start=91][count=1]
[  762.136662] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=28]
[  762.136664] [vq=00000000201c16ab][start=92][count=1]
[  762.387928] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=29]
[  762.387931] [vq=00000000201c16ab][start=93][count=1]
[  762.425545] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=30]
[  762.425547] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=31]
[  762.425548] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=32]
[  762.425550] [vq=00000000201c16ab][start=94][count=3]
[  762.425829] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=33]
[  762.425830] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=34]
[  762.425832] [vq=00000000201c16ab][start=97][count=2]
[  762.554053] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=35]
[  762.554055] [vq=00000000201c16ab][start=99][count=1]
[  762.804248] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=36]
[  762.804250] [vq=00000000201c16ab][start=100][count=1]
[  762.863934] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=37]
[  762.863936] [vq=00000000201c16ab][start=101][count=1]
[  763.176763] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=38]
[  763.176766] [vq=00000000201c16ab][start=102][count=1]
[  763.220580] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=39]
[  763.220582] [vq=00000000201c16ab][start=103][count=1]
[  763.371347] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=40]
[  763.371350] [vq=00000000201c16ab][start=104][count=1]
[  763.371376] [vq=00000000465a3846][vq->ndescs=1]
[  763.371379] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  763.371380] [vq=00000000465a3846][vq->ndescs=0]
[  763.371384] [vq=00000000465a3846][start=39][count=1]
[  763.426476] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=41]
[  763.426479] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=42]
[  763.426481] [vq=00000000201c16ab][start=105][count=2]
[  763.426783] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=43]
[  763.426785] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=44]
[  763.426786] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=45]
[  763.426788] [vq=00000000201c16ab][start=107][count=3]
[  763.784998] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=46]
[  763.785001] [vq=00000000201c16ab][start=110][count=1]
[  763.871337] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=47]
[  763.871340] [vq=00000000201c16ab][start=111][count=1]
[  763.903944] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=48]
[  763.903946] [vq=00000000201c16ab][start=112][count=1]
[  766.596032] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=49]
[  766.596036] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=50]
[  766.596038] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=51]
[  766.596040] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=52]
[  766.596041] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=53]
[  766.596043] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=54]
[  766.596044] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=55]
[  766.596046] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=56]
[  766.596047] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=57]
[  766.596049] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=58]
[  766.596051] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=59]
[  766.596053] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=60]
[  766.596054] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=61]
[  766.596056] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=62]
[  766.596058] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=63]
[  766.596059] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=64]
[  766.596061] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=65]
[  766.596063] [vq=00000000201c16ab][vq->ndescs=0]
[  766.596064] [vq=00000000201c16ab][start=0][count=17]
[  766.596067] [vq=00000000465a3846][vq->ndescs=0]
[  767.304752] [vq=00000000201c16ab][vq->ndescs=65]
[  767.304754] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=1]
[  767.304756] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=2]
[  767.304757] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=3]
[  767.304760] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=4]
[  767.304762] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=5]
[  767.304763] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=6]
[  767.304765] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=7]
[  767.304767] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=8]
[  767.304768] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=9]
[  767.304770] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=10]
[  767.304771] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=11]
[  767.304773] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=12]
[  767.304774] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=13]
[  767.304776] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=14]
[  767.304777] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=15]
[  767.304779] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=16]
[  767.304781] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=17]
[  767.304782] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=18]
[  767.304784] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=19]
[  767.304785] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=20]
[  767.304787] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=21]
[  767.304789] [vq=00000000201c16ab][start=17][count=21]
[  767.345024] [vq=00000000465a3846][vq->ndescs=1]
[  767.345025] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  767.345027] [vq=00000000465a3846][vq->ndescs=0]
[  767.345035] [vq=00000000465a3846][start=0][count=1]
[  767.364564] [vq=00000000465a3846][vq->ndescs=1]
[  767.364566] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  767.364567] [vq=00000000465a3846][vq->ndescs=0]
[  767.364571] [vq=00000000465a3846][start=1][count=1]
[  767.365202] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=22]
[  767.365209] [vq=00000000201c16ab][start=38][count=1]
[  767.365235] [vq=00000000465a3846][vq->ndescs=2]
[  767.365237] [vq=00000000465a3846][vq->ndescs=2][vq->first_desc=1]
[  767.365239] [vq=00000000465a3846][vq->ndescs=2][vq->first_desc=2]
[  767.365241] [vq=00000000465a3846][vq->ndescs=0]
[  767.365249] [vq=00000000465a3846][start=2][count=2]
[  767.365433] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=23]
[  767.365435] [vq=00000000201c16ab][start=39][count=1]
[  767.365448] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=24]
[  767.365449] [vq=00000000201c16ab][start=40][count=1]
[  767.370391] [vq=00000000465a3846][vq->ndescs=1]
[  767.370392] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  767.370394] [vq=00000000465a3846][vq->ndescs=0]
[  767.370398] [vq=00000000465a3846][start=4][count=1]
[  767.430616] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=25]
[  767.430620] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=26]
[  767.430622] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=27]
[  767.430624] [vq=00000000201c16ab][start=41][count=3]
[  767.430875] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=28]
[  767.430876] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=29]
[  767.430878] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=30]
[  767.430879] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=31]
[  767.430880] [vq=00000000201c16ab][start=44][count=4]
[  767.571839] [vq=00000000465a3846][vq->ndescs=2]
[  767.571842] [vq=00000000465a3846][vq->ndescs=2][vq->first_desc=2]
[  767.571844] [vq=00000000465a3846][vq->ndescs=0]
[  767.571851] [vq=00000000465a3846][start=5][count=1]
[  767.597747] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=32]
[  767.597750] [vq=00000000201c16ab][start=48][count=1]
[  767.650396] [vq=00000000465a3846][vq->ndescs=1]
[  767.650398] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  767.650399] [vq=00000000465a3846][vq->ndescs=0]
[  767.650406] [vq=00000000465a3846][start=6][count=1]
[  767.749947] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=33]
[  767.749952] [vq=00000000201c16ab][start=49][count=1]
[  767.773664] [vq=00000000465a3846][vq->ndescs=2]
[  767.773666] [vq=00000000465a3846][vq->ndescs=2][vq->first_desc=2]
[  767.773667] [vq=00000000465a3846][vq->ndescs=0]
[  767.773673] [vq=00000000465a3846][start=7][count=1]
[  767.807483] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=34]
[  767.807485] [vq=00000000201c16ab][start=50][count=1]
[  767.903813] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=35]
[  767.903816] [vq=00000000201c16ab][start=51][count=1]
[  767.975773] [vq=00000000465a3846][vq->ndescs=2]
[  767.975774] [vq=00000000465a3846][vq->ndescs=2][vq->first_desc=2]
[  767.975776] [vq=00000000465a3846][vq->ndescs=0]
[  767.975781] [vq=00000000465a3846][start=8][count=1]
[  768.007409] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=36]
[  768.007411] [vq=00000000201c16ab][start=52][count=1]
[  768.031512] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=37]
[  768.031514] [vq=00000000201c16ab][start=53][count=1]
[  768.067894] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=38]
[  768.067896] [vq=00000000201c16ab][start=54][count=1]
[  768.075648] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=39]
[  768.075650] [vq=00000000201c16ab][start=55][count=1]
[  768.090397] [vq=00000000465a3846][vq->ndescs=1]
[  768.090398] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  768.090400] [vq=00000000465a3846][vq->ndescs=0]
[  768.090406] [vq=00000000465a3846][start=9][count=1]
[  768.140369] [vq=00000000465a3846][vq->ndescs=1]
[  768.140371] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  768.140373] [vq=00000000465a3846][vq->ndescs=0]
[  768.140378] [vq=00000000465a3846][start=10][count=1]
[  768.176600] [vq=00000000465a3846][vq->ndescs=2]
[  768.176601] [vq=00000000465a3846][vq->ndescs=2][vq->first_desc=2]
[  768.176603] [vq=00000000465a3846][vq->ndescs=0]
[  768.176606] [vq=00000000465a3846][start=11][count=1]
[  768.217108] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=40]
[  768.217111] [vq=00000000201c16ab][start=56][count=1]
[  768.431553] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=41]
[  768.431560] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=42]
[  768.431563] [vq=00000000201c16ab][start=57][count=2]
[  768.431869] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=43]
[  768.431870] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=44]
[  768.431871] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=45]
[  768.431872] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=46]
[  768.431874] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=47]
[  768.431875] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=48]
[  768.431876] [vq=00000000201c16ab][start=59][count=6]
[  768.858404] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=49]
[  768.858412] [vq=00000000201c16ab][start=65][count=1]
[  769.065857] [vq=00000000465a3846][vq->ndescs=1]
[  769.065859] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  769.065862] [vq=00000000465a3846][vq->ndescs=0]
[  769.065871] [vq=00000000465a3846][start=12][count=1]
[  769.066299] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=50]
[  769.066301] [vq=00000000201c16ab][start=66][count=1]
[  769.066329] [vq=00000000465a3846][vq->ndescs=1]
[  769.066330] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  769.066331] [vq=00000000465a3846][vq->ndescs=0]
[  769.066335] [vq=00000000465a3846][start=13][count=1]
[  769.066514] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=51]
[  769.066515] [vq=00000000201c16ab][start=67][count=1]
[  769.066617] [vq=00000000465a3846][vq->ndescs=1]
[  769.066618] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  769.066619] [vq=00000000465a3846][vq->ndescs=0]
[  769.066622] [vq=00000000465a3846][start=14][count=1]
[  769.066765] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=52]
[  769.066769] [vq=00000000201c16ab][start=68][count=1]
[  769.066828] [vq=00000000465a3846][vq->ndescs=1]
[  769.066829] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  769.066831] [vq=00000000465a3846][vq->ndescs=0]
[  769.066835] [vq=00000000465a3846][start=15][count=1]
[  769.066967] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=53]
[  769.066968] [vq=00000000201c16ab][start=69][count=1]
[  769.067024] [vq=00000000465a3846][vq->ndescs=1]
[  769.067025] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  769.067026] [vq=00000000465a3846][vq->ndescs=0]
[  769.067029] [vq=00000000465a3846][start=16][count=1]
[  769.067170] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=54]
[  769.067172] [vq=00000000201c16ab][start=70][count=1]
[  769.067224] [vq=00000000465a3846][vq->ndescs=1]
[  769.067225] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  769.067226] [vq=00000000465a3846][vq->ndescs=0]
[  769.067229] [vq=00000000465a3846][start=17][count=1]
[  769.067365] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=55]
[  769.067366] [vq=00000000201c16ab][start=71][count=1]
[  769.067416] [vq=00000000465a3846][vq->ndescs=1]
[  769.067417] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  769.067418] [vq=00000000465a3846][vq->ndescs=0]
[  769.067421] [vq=00000000465a3846][start=18][count=1]
[  769.067557] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=56]
[  769.067558] [vq=00000000201c16ab][start=72][count=1]
[  769.067605] [vq=00000000465a3846][vq->ndescs=1]
[  769.067606] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  769.067607] [vq=00000000465a3846][vq->ndescs=0]
[  769.067610] [vq=00000000465a3846][start=19][count=1]
[  769.067758] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=57]
[  769.067759] [vq=00000000201c16ab][start=73][count=1]
[  769.067805] [vq=00000000465a3846][vq->ndescs=1]
[  769.067806] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  769.067807] [vq=00000000465a3846][vq->ndescs=0]
[  769.067811] [vq=00000000465a3846][start=20][count=1]
[  769.067959] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=58]
[  769.067961] [vq=00000000201c16ab][start=74][count=1]
[  769.068045] [vq=00000000465a3846][vq->ndescs=1]
[  769.068047] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  769.068049] [vq=00000000465a3846][vq->ndescs=0]
[  769.068052] [vq=00000000465a3846][start=21][count=1]
[  769.068181] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=59]
[  769.068186] [vq=00000000201c16ab][start=75][count=1]
[  769.068241] [vq=00000000465a3846][vq->ndescs=1]
[  769.068242] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  769.068244] [vq=00000000465a3846][vq->ndescs=0]
[  769.068250] [vq=00000000465a3846][start=22][count=1]
[  769.068379] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=60]
[  769.068381] [vq=00000000201c16ab][start=76][count=1]
[  769.068425] [vq=00000000465a3846][vq->ndescs=1]
[  769.068426] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  769.068428] [vq=00000000465a3846][vq->ndescs=0]
[  769.068432] [vq=00000000465a3846][start=23][count=1]
[  769.068555] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=61]
[  769.068557] [vq=00000000201c16ab][start=77][count=1]
[  769.068599] [vq=00000000465a3846][vq->ndescs=1]
[  769.068600] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  769.068605] [vq=00000000465a3846][vq->ndescs=0]
[  769.068608] [vq=00000000465a3846][start=24][count=1]
[  769.068737] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=62]
[  769.068739] [vq=00000000201c16ab][start=78][count=1]
[  769.068781] [vq=00000000465a3846][vq->ndescs=1]
[  769.068782] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  769.068784] [vq=00000000465a3846][vq->ndescs=0]
[  769.068787] [vq=00000000465a3846][start=25][count=1]
[  769.068914] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=63]
[  769.068916] [vq=00000000201c16ab][start=79][count=1]
[  769.068959] [vq=00000000465a3846][vq->ndescs=1]
[  769.068960] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  769.068962] [vq=00000000465a3846][vq->ndescs=0]
[  769.068965] [vq=00000000465a3846][start=26][count=1]
[  769.069094] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=64]
[  769.069097] [vq=00000000201c16ab][start=80][count=1]
[  769.069139] [vq=00000000465a3846][vq->ndescs=1]
[  769.069140] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  769.069143] [vq=00000000465a3846][vq->ndescs=0]
[  769.069145] [vq=00000000465a3846][start=27][count=1]
[  769.069271] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=65]
[  769.069274] [vq=00000000201c16ab][start=81][count=1]
[  769.069316] [vq=00000000465a3846][vq->ndescs=1]
[  769.069317] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  769.069319] [vq=00000000465a3846][vq->ndescs=0]
[  769.069322] [vq=00000000465a3846][start=28][count=1]
[  769.069452] [vq=00000000201c16ab][vq->ndescs=65]
[  769.069453] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=1]
[  769.069455] [vq=00000000201c16ab][start=82][count=1]
[  769.069499] [vq=00000000465a3846][vq->ndescs=1]
[  769.069500] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  769.069501] [vq=00000000465a3846][vq->ndescs=0]
[  769.069504] [vq=00000000465a3846][start=29][count=1]
[  769.069650] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=2]
[  769.069654] [vq=00000000201c16ab][start=83][count=1]
[  769.069711] [vq=00000000465a3846][vq->ndescs=1]
[  769.069712] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  769.069714] [vq=00000000465a3846][vq->ndescs=0]
[  769.069718] [vq=00000000465a3846][start=30][count=1]
[  769.069851] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=3]
[  769.069852] [vq=00000000201c16ab][start=84][count=1]
[  769.069905] [vq=00000000465a3846][vq->ndescs=1]
[  769.069906] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  769.069908] [vq=00000000465a3846][vq->ndescs=0]
[  769.069910] [vq=00000000465a3846][start=31][count=1]
[  769.070045] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=4]
[  769.070046] [vq=00000000201c16ab][start=85][count=1]
[  769.070115] [vq=00000000465a3846][vq->ndescs=1]
[  769.070116] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  769.070117] [vq=00000000465a3846][vq->ndescs=0]
[  769.070120] [vq=00000000465a3846][start=32][count=1]
[  769.070253] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=5]
[  769.070254] [vq=00000000201c16ab][start=86][count=1]
[  769.093873] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=6]
[  769.093878] [vq=00000000201c16ab][start=87][count=1]
[  769.107789] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=7]
[  769.107791] [vq=00000000201c16ab][start=88][count=1]
[  769.130423] [vq=00000000465a3846][vq->ndescs=1]
[  769.130424] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  769.130426] [vq=00000000465a3846][vq->ndescs=0]
[  769.130437] [vq=00000000465a3846][start=33][count=1]
[  769.133437] [vq=00000000465a3846][vq->ndescs=1]
[  769.133438] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  769.133440] [vq=00000000465a3846][vq->ndescs=0]
[  769.133443] [vq=00000000465a3846][start=34][count=1]
[  769.170381] [vq=00000000465a3846][vq->ndescs=1]
[  769.170383] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  769.170384] [vq=00000000465a3846][vq->ndescs=0]
[  769.170388] [vq=00000000465a3846][start=35][count=1]
[  769.370375] [vq=00000000465a3846][vq->ndescs=1]
[  769.370377] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  769.370380] [vq=00000000465a3846][vq->ndescs=0]
[  769.370386] [vq=00000000465a3846][start=36][count=1]
[  769.432770] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=8]
[  769.432773] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=9]
[  769.432775] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=10]
[  769.432776] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=11]
[  769.432778] [vq=00000000201c16ab][start=89][count=4]
[  769.432999] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=12]
[  769.433001] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=13]
[  769.433003] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=14]
[  769.433005] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=15]
[  769.433006] [vq=00000000201c16ab][start=93][count=4]
[  769.500358] [vq=00000000465a3846][vq->ndescs=1]
[  769.500361] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  769.500363] [vq=00000000465a3846][vq->ndescs=0]
[  769.500368] [vq=00000000465a3846][start=37][count=1]
[  769.855119] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=16]
[  769.855123] [vq=00000000201c16ab][start=97][count=1]
[  769.855886] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=17]
[  769.855888] [vq=00000000201c16ab][start=98][count=1]
[  770.136903] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=18]
[  770.136906] [vq=00000000201c16ab][start=99][count=1]
[  770.147613] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=19]
[  770.147616] [vq=00000000201c16ab][start=100][count=1]
[  770.433405] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=20]
[  770.433408] [vq=00000000201c16ab][start=101][count=1]
[  770.433716] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=21]
[  770.433718] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=22]
[  770.433720] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=23]
[  770.433721] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=24]
[  770.433723] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=25]
[  770.433724] [vq=00000000201c16ab][start=102][count=5]
[  770.433903] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=26]
[  770.433904] [vq=00000000201c16ab][start=107][count=1]
[  770.830869] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=27]
[  770.830874] [vq=00000000201c16ab][start=108][count=1]
[  770.863937] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=28]
[  770.863939] [vq=00000000201c16ab][start=109][count=1]
[  770.921316] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=29]
[  770.921319] [vq=00000000201c16ab][start=110][count=1]
[  771.176836] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=30]
[  771.176839] [vq=00000000201c16ab][start=111][count=1]
[  771.434412] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=31]
[  771.434415] [vq=00000000201c16ab][start=112][count=1]
[  771.434701] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=32]
[  771.434704] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=33]
[  771.434705] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=34]
[  771.434707] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=35]
[  771.434708] [vq=00000000201c16ab][start=113][count=4]
[  771.434885] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=36]
[  771.434887] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=37]
[  771.434888] [vq=00000000201c16ab][start=117][count=2]
[  771.904063] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=38]
[  771.904066] [vq=00000000201c16ab][start=119][count=1]
[  771.961232] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=39]
[  771.961234] [vq=00000000201c16ab][start=120][count=1]
[  772.435587] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=40]
[  772.435590] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=41]
[  772.435592] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=42]
[  772.435593] [vq=00000000201c16ab][start=121][count=3]
[  772.435862] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=43]
[  772.435864] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=44]
[  772.435865] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=45]
[  772.435867] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=46]
[  772.435868] [vq=00000000201c16ab][start=124][count=4]
[  775.057134] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=47]
[  775.057139] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=48]
[  775.057141] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=49]
[  775.057143] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=50]
[  775.057144] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=51]
[  775.057146] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=52]
[  775.057148] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=53]
[  775.057149] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=54]
[  775.057151] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=55]
[  775.057153] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=56]
[  775.057155] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=57]
[  775.057156] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=58]
[  775.057158] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=59]
[  775.057160] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=60]
[  775.057162] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=61]
[  775.057163] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=62]
[  775.057165] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=63]
[  775.057166] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=64]
[  775.057168] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=65]
[  775.057170] [vq=00000000201c16ab][vq->ndescs=0]
[  775.057172] [vq=00000000201c16ab][start=0][count=19]
[  775.057175] [vq=00000000465a3846][vq->ndescs=0]
[  775.746716] [vq=00000000201c16ab][vq->ndescs=65]
[  775.746717] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=1]
[  775.746719] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=2]
[  775.746721] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=3]
[  775.746722] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=4]
[  775.746723] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=5]
[  775.746725] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=6]
[  775.746726] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=7]
[  775.746727] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=8]
[  775.746729] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=9]
[  775.746730] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=10]
[  775.746731] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=11]
[  775.746732] [vq=00000000201c16ab][start=19][count=11]
[  775.779189] [vq=00000000465a3846][vq->ndescs=1]
[  775.779191] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  775.779193] [vq=00000000465a3846][vq->ndescs=0]
[  775.779201] [vq=00000000465a3846][start=0][count=1]
[  775.794833] [vq=00000000465a3846][vq->ndescs=1]
[  775.794835] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  775.794836] [vq=00000000465a3846][vq->ndescs=0]
[  775.794842] [vq=00000000465a3846][start=1][count=1]
[  775.803247] [vq=00000000465a3846][vq->ndescs=1]
[  775.803248] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  775.803249] [vq=00000000465a3846][vq->ndescs=0]
[  775.803253] [vq=00000000465a3846][start=2][count=1]
[  775.803932] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=12]
[  775.803938] [vq=00000000201c16ab][start=30][count=1]
[  775.803977] [vq=00000000465a3846][vq->ndescs=2]
[  775.803998] [vq=00000000465a3846][vq->ndescs=2][vq->first_desc=1]
[  775.804000] [vq=00000000465a3846][vq->ndescs=2][vq->first_desc=2]
[  775.804002] [vq=00000000465a3846][vq->ndescs=0]
[  775.804010] [vq=00000000465a3846][start=3][count=2]
[  775.804194] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=13]
[  775.804196] [vq=00000000201c16ab][start=31][count=1]
[  775.804205] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=14]
[  775.804207] [vq=00000000201c16ab][start=32][count=1]
[  775.913969] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=15]
[  775.913974] [vq=00000000201c16ab][start=33][count=1]
[  775.997471] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=16]
[  775.997474] [vq=00000000201c16ab][start=34][count=1]
[  776.006405] [vq=00000000465a3846][vq->ndescs=2]
[  776.006407] [vq=00000000465a3846][vq->ndescs=2][vq->first_desc=2]
[  776.006409] [vq=00000000465a3846][vq->ndescs=0]
[  776.006416] [vq=00000000465a3846][start=5][count=1]
[  776.036919] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=17]
[  776.036924] [vq=00000000201c16ab][start=35][count=1]
[  776.046852] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=18]
[  776.046856] [vq=00000000201c16ab][start=36][count=1]
[  776.061540] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=19]
[  776.061542] [vq=00000000201c16ab][start=37][count=1]
[  776.111415] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=20]
[  776.111419] [vq=00000000201c16ab][start=38][count=1]
[  776.154841] [vq=00000000465a3846][vq->ndescs=1]
[  776.154843] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  776.154845] [vq=00000000465a3846][vq->ndescs=0]
[  776.154850] [vq=00000000465a3846][start=6][count=1]
[  776.209078] [vq=00000000465a3846][vq->ndescs=2]
[  776.209079] [vq=00000000465a3846][vq->ndescs=2][vq->first_desc=2]
[  776.209080] [vq=00000000465a3846][vq->ndescs=0]
[  776.209083] [vq=00000000465a3846][start=7][count=1]
[  776.234725] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=21]
[  776.234727] [vq=00000000201c16ab][start=39][count=1]
[  776.306637] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=22]
[  776.306639] [vq=00000000201c16ab][start=40][count=1]
[  776.394831] [vq=00000000465a3846][vq->ndescs=1]
[  776.394832] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  776.394833] [vq=00000000465a3846][vq->ndescs=0]
[  776.394837] [vq=00000000465a3846][start=8][count=1]
[  776.411908] [vq=00000000465a3846][vq->ndescs=2]
[  776.411910] [vq=00000000465a3846][vq->ndescs=2][vq->first_desc=2]
[  776.411911] [vq=00000000465a3846][vq->ndescs=0]
[  776.411914] [vq=00000000465a3846][start=9][count=1]
[  776.439659] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=23]
[  776.439660] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=24]
[  776.439661] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=25]
[  776.439663] [vq=00000000201c16ab][start=41][count=3]
[  776.439929] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=26]
[  776.439931] [vq=00000000201c16ab][start=44][count=1]
[  776.448982] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=27]
[  776.448984] [vq=00000000201c16ab][start=45][count=1]
[  776.504845] [vq=00000000465a3846][vq->ndescs=1]
[  776.504846] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  776.504848] [vq=00000000465a3846][vq->ndescs=0]
[  776.504851] [vq=00000000465a3846][start=10][count=1]
[  776.615546] [vq=00000000465a3846][vq->ndescs=2]
[  776.615548] [vq=00000000465a3846][vq->ndescs=2][vq->first_desc=2]
[  776.615549] [vq=00000000465a3846][vq->ndescs=0]
[  776.615551] [vq=00000000465a3846][start=11][count=1]
[  776.650494] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=28]
[  776.650495] [vq=00000000201c16ab][start=46][count=1]
[  777.099061] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=29]
[  777.099068] [vq=00000000201c16ab][start=47][count=1]
[  777.151245] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=30]
[  777.151247] [vq=00000000201c16ab][start=48][count=1]
[  777.283424] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=31]
[  777.283427] [vq=00000000201c16ab][start=49][count=1]
[  777.337441] [vq=00000000465a3846][vq->ndescs=1]
[  777.337442] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  777.337443] [vq=00000000465a3846][vq->ndescs=0]
[  777.337447] [vq=00000000465a3846][start=12][count=1]
[  777.337944] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=32]
[  777.337948] [vq=00000000201c16ab][start=50][count=1]
[  777.337968] [vq=00000000465a3846][vq->ndescs=1]
[  777.337969] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  777.337971] [vq=00000000465a3846][vq->ndescs=0]
[  777.337975] [vq=00000000465a3846][start=13][count=1]
[  777.338197] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=33]
[  777.338198] [vq=00000000201c16ab][start=51][count=1]
[  777.338238] [vq=00000000465a3846][vq->ndescs=1]
[  777.338239] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  777.338241] [vq=00000000465a3846][vq->ndescs=0]
[  777.338244] [vq=00000000465a3846][start=14][count=1]
[  777.338382] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=34]
[  777.338383] [vq=00000000201c16ab][start=52][count=1]
[  777.338415] [vq=00000000465a3846][vq->ndescs=1]
[  777.338416] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  777.338417] [vq=00000000465a3846][vq->ndescs=0]
[  777.338419] [vq=00000000465a3846][start=15][count=1]
[  777.338556] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=35]
[  777.338557] [vq=00000000201c16ab][start=53][count=1]
[  777.338588] [vq=00000000465a3846][vq->ndescs=1]
[  777.338589] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  777.338600] [vq=00000000465a3846][vq->ndescs=0]
[  777.338603] [vq=00000000465a3846][start=16][count=1]
[  777.338760] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=36]
[  777.338762] [vq=00000000201c16ab][start=54][count=1]
[  777.338795] [vq=00000000465a3846][vq->ndescs=1]
[  777.338796] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  777.338797] [vq=00000000465a3846][vq->ndescs=0]
[  777.338799] [vq=00000000465a3846][start=17][count=1]
[  777.338939] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=37]
[  777.338940] [vq=00000000201c16ab][start=55][count=1]
[  777.338975] [vq=00000000465a3846][vq->ndescs=1]
[  777.338977] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  777.338979] [vq=00000000465a3846][vq->ndescs=0]
[  777.338983] [vq=00000000465a3846][start=18][count=1]
[  777.339124] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=38]
[  777.339126] [vq=00000000201c16ab][start=56][count=1]
[  777.339158] [vq=00000000465a3846][vq->ndescs=1]
[  777.339159] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  777.339160] [vq=00000000465a3846][vq->ndescs=0]
[  777.339163] [vq=00000000465a3846][start=19][count=1]
[  777.339314] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=39]
[  777.339316] [vq=00000000201c16ab][start=57][count=1]
[  777.339345] [vq=00000000465a3846][vq->ndescs=1]
[  777.339346] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  777.339348] [vq=00000000465a3846][vq->ndescs=0]
[  777.339350] [vq=00000000465a3846][start=20][count=1]
[  777.339489] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=40]
[  777.339490] [vq=00000000201c16ab][start=58][count=1]
[  777.339521] [vq=00000000465a3846][vq->ndescs=1]
[  777.339522] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  777.339524] [vq=00000000465a3846][vq->ndescs=0]
[  777.339526] [vq=00000000465a3846][start=21][count=1]
[  777.339662] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=41]
[  777.339664] [vq=00000000201c16ab][start=59][count=1]
[  777.339694] [vq=00000000465a3846][vq->ndescs=1]
[  777.339695] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  777.339696] [vq=00000000465a3846][vq->ndescs=0]
[  777.339699] [vq=00000000465a3846][start=22][count=1]
[  777.339842] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=42]
[  777.339843] [vq=00000000201c16ab][start=60][count=1]
[  777.339874] [vq=00000000465a3846][vq->ndescs=1]
[  777.339876] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  777.339877] [vq=00000000465a3846][vq->ndescs=0]
[  777.339879] [vq=00000000465a3846][start=23][count=1]
[  777.340015] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=43]
[  777.340016] [vq=00000000201c16ab][start=61][count=1]
[  777.340045] [vq=00000000465a3846][vq->ndescs=1]
[  777.340046] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  777.340048] [vq=00000000465a3846][vq->ndescs=0]
[  777.340050] [vq=00000000465a3846][start=24][count=1]
[  777.340187] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=44]
[  777.340188] [vq=00000000201c16ab][start=62][count=1]
[  777.340217] [vq=00000000465a3846][vq->ndescs=1]
[  777.340218] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  777.340220] [vq=00000000465a3846][vq->ndescs=0]
[  777.340222] [vq=00000000465a3846][start=25][count=1]
[  777.340358] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=45]
[  777.340360] [vq=00000000201c16ab][start=63][count=1]
[  777.340389] [vq=00000000465a3846][vq->ndescs=1]
[  777.340390] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  777.340391] [vq=00000000465a3846][vq->ndescs=0]
[  777.340393] [vq=00000000465a3846][start=26][count=1]
[  777.340528] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=46]
[  777.340529] [vq=00000000201c16ab][start=64][count=1]
[  777.340559] [vq=00000000465a3846][vq->ndescs=1]
[  777.340560] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  777.340561] [vq=00000000465a3846][vq->ndescs=0]
[  777.340564] [vq=00000000465a3846][start=27][count=1]
[  777.340697] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=47]
[  777.340698] [vq=00000000201c16ab][start=65][count=1]
[  777.340730] [vq=00000000465a3846][vq->ndescs=1]
[  777.340731] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  777.340732] [vq=00000000465a3846][vq->ndescs=0]
[  777.340734] [vq=00000000465a3846][start=28][count=1]
[  777.340870] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=48]
[  777.340871] [vq=00000000201c16ab][start=66][count=1]
[  777.340900] [vq=00000000465a3846][vq->ndescs=1]
[  777.340901] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  777.340902] [vq=00000000465a3846][vq->ndescs=0]
[  777.340904] [vq=00000000465a3846][start=29][count=1]
[  777.341038] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=49]
[  777.341039] [vq=00000000201c16ab][start=67][count=1]
[  777.341068] [vq=00000000465a3846][vq->ndescs=1]
[  777.341069] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  777.341070] [vq=00000000465a3846][vq->ndescs=0]
[  777.341073] [vq=00000000465a3846][start=30][count=1]
[  777.341209] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=50]
[  777.341210] [vq=00000000201c16ab][start=68][count=1]
[  777.341239] [vq=00000000465a3846][vq->ndescs=1]
[  777.341240] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  777.341242] [vq=00000000465a3846][vq->ndescs=0]
[  777.341244] [vq=00000000465a3846][start=31][count=1]
[  777.341379] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=51]
[  777.341381] [vq=00000000201c16ab][start=69][count=1]
[  777.341410] [vq=00000000465a3846][vq->ndescs=1]
[  777.341411] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  777.341412] [vq=00000000465a3846][vq->ndescs=0]
[  777.341415] [vq=00000000465a3846][start=32][count=1]
[  777.341550] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=52]
[  777.341551] [vq=00000000201c16ab][start=70][count=1]
[  777.347869] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=53]
[  777.347874] [vq=00000000201c16ab][start=71][count=1]
[  777.404861] [vq=00000000465a3846][vq->ndescs=1]
[  777.404863] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  777.404864] [vq=00000000465a3846][vq->ndescs=0]
[  777.404872] [vq=00000000465a3846][start=33][count=1]
[  777.434829] [vq=00000000465a3846][vq->ndescs=1]
[  777.434831] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  777.434832] [vq=00000000465a3846][vq->ndescs=0]
[  777.434836] [vq=00000000465a3846][start=34][count=1]
[  777.440676] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=54]
[  777.440678] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=55]
[  777.440679] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=56]
[  777.440681] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=57]
[  777.440682] [vq=00000000201c16ab][start=72][count=4]
[  777.440939] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=58]
[  777.440940] [vq=00000000201c16ab][start=76][count=1]
[  777.844890] [vq=00000000465a3846][vq->ndescs=1]
[  777.844893] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  777.844896] [vq=00000000465a3846][vq->ndescs=0]
[  777.844905] [vq=00000000465a3846][start=35][count=1]
[  777.861124] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=59]
[  777.861127] [vq=00000000201c16ab][start=77][count=1]
[  777.892947] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=60]
[  777.892950] [vq=00000000201c16ab][start=78][count=1]
[  778.136716] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=61]
[  778.136718] [vq=00000000201c16ab][start=79][count=1]
[  778.201376] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=62]
[  778.201378] [vq=00000000201c16ab][start=80][count=1]
[  778.204884] [vq=00000000465a3846][vq->ndescs=1]
[  778.204885] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  778.204887] [vq=00000000465a3846][vq->ndescs=0]
[  778.204892] [vq=00000000465a3846][start=36][count=1]
[  778.335800] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=63]
[  778.335803] [vq=00000000201c16ab][start=81][count=1]
[  778.387704] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=64]
[  778.387706] [vq=00000000201c16ab][start=82][count=1]
[  778.441538] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=65]
[  778.441543] [vq=00000000201c16ab][vq->ndescs=65]
[  778.441545] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=1]
[  778.441546] [vq=00000000201c16ab][start=83][count=2]
[  778.441804] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=2]
[  778.441805] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=3]
[  778.441807] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=4]
[  778.441808] [vq=00000000201c16ab][start=85][count=3]
[  778.808385] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=5]
[  778.808390] [vq=00000000201c16ab][start=88][count=1]
[  778.863990] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=6]
[  778.863992] [vq=00000000201c16ab][start=89][count=1]
[  778.907345] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=7]
[  778.907348] [vq=00000000201c16ab][start=90][count=1]
[  779.176815] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=8]
[  779.176819] [vq=00000000201c16ab][start=91][count=1]
[  779.358511] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=9]
[  779.358513] [vq=00000000201c16ab][start=92][count=1]
[  779.437933] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=10]
[  779.437935] [vq=00000000201c16ab][start=93][count=1]
[  779.442422] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=11]
[  779.442424] [vq=00000000201c16ab][start=94][count=1]
[  779.442665] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=12]
[  779.442667] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=13]
[  779.442668] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=14]
[  779.442669] [vq=00000000201c16ab][start=95][count=3]
[  779.904029] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=15]
[  779.904032] [vq=00000000201c16ab][start=98][count=1]
[  780.443688] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=16]
[  780.443691] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=17]
[  780.443692] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=18]
[  780.443694] [vq=00000000201c16ab][start=99][count=3]
[  780.443985] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=19]
[  780.443987] [vq=00000000201c16ab][start=102][count=1]
[  780.467631] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=20]
[  780.467633] [vq=00000000201c16ab][start=103][count=1]
[  780.651623] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=21]
[  780.651626] [vq=00000000201c16ab][start=104][count=1]
[  783.326504] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=22]
[  783.326510] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=23]
[  783.326512] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=24]
[  783.326514] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=25]
[  783.326515] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=26]
[  783.326517] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=27]
[  783.326518] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=28]
[  783.326520] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=29]
[  783.326522] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=30]
[  783.326523] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=31]
[  783.326524] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=32]
[  783.326526] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=33]
[  783.326528] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=34]
[  783.326530] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=35]
[  783.326532] [vq=00000000201c16ab][start=0][count=14]
[  783.326535] [vq=00000000465a3846][vq->ndescs=0]
[  783.446605] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=36]
[  783.446608] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=37]
[  783.446610] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=38]
[  783.446611] [vq=00000000201c16ab][start=14][count=3]
[  783.903893] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=39]
[  783.903896] [vq=00000000201c16ab][start=17][count=1]
[  784.022786] [vq=00000000465a3846][vq->ndescs=1]
[  784.022788] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  784.022790] [vq=00000000465a3846][vq->ndescs=0]
[  784.022797] [vq=00000000465a3846][start=0][count=1]
[  784.039300] [vq=00000000465a3846][vq->ndescs=1]
[  784.039301] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  784.039303] [vq=00000000465a3846][vq->ndescs=0]
[  784.039306] [vq=00000000465a3846][start=1][count=1]
[  784.039827] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=40]
[  784.039829] [vq=00000000201c16ab][start=18][count=1]
[  784.044823] [vq=00000000465a3846][vq->ndescs=1]
[  784.044824] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  784.044826] [vq=00000000465a3846][vq->ndescs=0]
[  784.044831] [vq=00000000465a3846][start=2][count=1]
[  784.167484] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=41]
[  784.167486] [vq=00000000201c16ab][start=19][count=1]
[  784.329413] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=42]
[  784.329416] [vq=00000000201c16ab][start=20][count=1]
[  784.414819] [vq=00000000465a3846][vq->ndescs=1]
[  784.414820] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  784.414822] [vq=00000000465a3846][vq->ndescs=0]
[  784.414826] [vq=00000000465a3846][start=3][count=1]
[  784.447615] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=43]
[  784.447618] [vq=00000000201c16ab][start=21][count=1]
[  784.447900] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=44]
[  784.447902] [vq=00000000201c16ab][start=22][count=1]
[  784.934838] [vq=00000000465a3846][vq->ndescs=1]
[  784.934840] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  784.934841] [vq=00000000465a3846][vq->ndescs=0]
[  784.934845] [vq=00000000465a3846][start=4][count=1]
[  785.100943] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=45]
[  785.100946] [vq=00000000201c16ab][start=23][count=1]
[  785.104788] [vq=00000000465a3846][vq->ndescs=1]
[  785.104789] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  785.104791] [vq=00000000465a3846][vq->ndescs=0]
[  785.104797] [vq=00000000465a3846][start=5][count=1]
[  785.105315] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=46]
[  785.105316] [vq=00000000201c16ab][start=24][count=1]
[  785.264810] [vq=00000000465a3846][vq->ndescs=1]
[  785.264812] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  785.264813] [vq=00000000465a3846][vq->ndescs=0]
[  785.264817] [vq=00000000465a3846][start=6][count=1]
[  785.448591] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=47]
[  785.448593] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=48]
[  785.448595] [vq=00000000201c16ab][start=25][count=2]
[  785.812091] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=49]
[  785.812093] [vq=00000000201c16ab][start=27][count=1]
[  785.865622] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=50]
[  785.865624] [vq=00000000201c16ab][start=28][count=1]
[  785.956012] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=51]
[  785.956014] [vq=00000000201c16ab][start=29][count=1]
[  785.984841] [vq=00000000465a3846][vq->ndescs=1]
[  785.984843] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  785.984844] [vq=00000000465a3846][vq->ndescs=0]
[  785.984849] [vq=00000000465a3846][start=7][count=1]
[  785.985751] [vq=00000000465a3846][vq->ndescs=1]
[  785.985752] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  785.985753] [vq=00000000465a3846][vq->ndescs=0]
[  785.985756] [vq=00000000465a3846][start=8][count=1]
[  786.022926] [vq=00000000465a3846][vq->ndescs=1]
[  786.022927] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  786.022928] [vq=00000000465a3846][vq->ndescs=0]
[  786.022931] [vq=00000000465a3846][start=9][count=1]
[  786.024797] [vq=00000000465a3846][vq->ndescs=1]
[  786.024798] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  786.024799] [vq=00000000465a3846][vq->ndescs=0]
[  786.024802] [vq=00000000465a3846][start=10][count=1]
[  786.054795] [vq=00000000465a3846][vq->ndescs=1]
[  786.054796] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  786.054798] [vq=00000000465a3846][vq->ndescs=0]
[  786.054802] [vq=00000000465a3846][start=11][count=1]
[  786.136759] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=52]
[  786.136761] [vq=00000000201c16ab][start=30][count=1]
[  786.144783] [vq=00000000465a3846][vq->ndescs=1]
[  786.144784] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  786.144786] [vq=00000000465a3846][vq->ndescs=0]
[  786.144790] [vq=00000000465a3846][start=12][count=1]
[  786.145388] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=53]
[  786.145389] [vq=00000000201c16ab][start=31][count=1]
[  786.166419] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=54]
[  786.166421] [vq=00000000201c16ab][start=32][count=1]
[  786.449605] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=55]
[  786.449608] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=56]
[  786.449610] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=57]
[  786.449611] [vq=00000000201c16ab][start=33][count=3]
[  786.831294] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=58]
[  786.831296] [vq=00000000201c16ab][start=36][count=1]
[  786.943969] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=59]
[  786.943971] [vq=00000000201c16ab][start=37][count=1]
[  786.944849] [vq=00000000465a3846][vq->ndescs=1]
[  786.944850] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  786.944853] [vq=00000000465a3846][vq->ndescs=0]
[  786.944858] [vq=00000000465a3846][start=13][count=1]
[  786.980009] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=60]
[  786.980011] [vq=00000000201c16ab][start=38][count=1]
[  787.155382] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=61]
[  787.155386] [vq=00000000201c16ab][start=39][count=1]
[  787.176736] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=62]
[  787.176738] [vq=00000000201c16ab][start=40][count=1]
[  787.185039] [vq=00000000465a3846][vq->ndescs=1]
[  787.185040] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  787.185042] [vq=00000000465a3846][vq->ndescs=0]
[  787.185050] [vq=00000000465a3846][start=14][count=1]
[  787.185639] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=63]
[  787.185641] [vq=00000000201c16ab][start=41][count=1]
[  787.450691] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=64]
[  787.450693] [vq=00000000201c16ab][vq->ndescs=65][vq->first_desc=65]
[  787.450695] [vq=00000000201c16ab][start=42][count=2]
[  787.485512] Guest moved vq 00000000201c16ab used index from 0 to 289
[  787.485513] [vq=00000000201c16ab][vq->ndescs=0]
[  788.023034] [vq=00000000465a3846][vq->ndescs=1]
[  788.023036] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  788.023037] [vq=00000000465a3846][vq->ndescs=0]
[  788.023042] [vq=00000000465a3846][start=15][count=1]
[  788.224798] [vq=00000000465a3846][vq->ndescs=1]
[  788.224800] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  788.224802] [vq=00000000465a3846][vq->ndescs=0]
[  788.224806] [vq=00000000465a3846][start=16][count=1]
[  789.264807] [vq=00000000465a3846][vq->ndescs=1]
[  789.264809] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  789.264811] [vq=00000000465a3846][vq->ndescs=0]
[  789.264816] [vq=00000000465a3846][start=17][count=1]
[  789.885117] [vq=00000000465a3846][vq->ndescs=1]
[  789.885118] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  789.885120] [vq=00000000465a3846][vq->ndescs=0]
[  789.885124] [vq=00000000465a3846][start=18][count=1]
[  790.304957] [vq=00000000465a3846][vq->ndescs=1]
[  790.304959] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  790.304961] [vq=00000000465a3846][vq->ndescs=0]
[  790.304967] [vq=00000000465a3846][start=19][count=1]
[  791.344839] [vq=00000000465a3846][vq->ndescs=1]
[  791.344842] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  791.344843] [vq=00000000465a3846][vq->ndescs=0]
[  791.344850] [vq=00000000465a3846][start=20][count=1]
[  792.384804] [vq=00000000465a3846][vq->ndescs=1]
[  792.384807] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  792.384808] [vq=00000000465a3846][vq->ndescs=0]
[  792.384814] [vq=00000000465a3846][start=21][count=1]
[  793.424946] [vq=00000000465a3846][vq->ndescs=1]
[  793.424948] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  793.424950] [vq=00000000465a3846][vq->ndescs=0]
[  793.424955] [vq=00000000465a3846][start=22][count=1]
[  793.886441] [vq=00000000465a3846][vq->ndescs=1]
[  793.886442] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  793.886444] [vq=00000000465a3846][vq->ndescs=0]
[  793.886449] [vq=00000000465a3846][start=23][count=1]
[  794.464802] [vq=00000000465a3846][vq->ndescs=1]
[  794.464804] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  794.464806] [vq=00000000465a3846][vq->ndescs=0]
[  794.464811] [vq=00000000465a3846][start=24][count=1]
[  795.504831] [vq=00000000465a3846][vq->ndescs=1]
[  795.504835] [vq=00000000465a3846][vq->ndescs=1][vq->first_desc=1]
[  795.504838] [vq=00000000465a3846][vq->ndescs=0]

Re: vhost changes (batched) in linux-next after 12/13 trigger random crashes in KVM guests after reboot

[Eugenio Pérez]

On Mon, 2020-02-10 at 12:01 +0100, Christian Borntraeger wrote:
> 
> On 10.02.20 10:47, Eugenio Perez Martin wrote:
> > Hi Christian.
> > 
> > I'm not able to reproduce the failure with eccb852f1fe6bede630e2e4f1a121a81e34354ab commit. Could you add more data?
> > Your configuration (libvirt or qemu line), and host's dmesg output if any?
> > 
> > Thanks!
> 
> If it was not obvious, this is on s390x, a big endian system.
> 

Hi Christian. Thank you very much for your fast responses.

Could you try this patch on top of eccb852f1fe6bede630e2e4f1a121a81e34354ab?

Thanks!

From 71d0f9108a18aa894cc0c0c1c7efbad39f465a27 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Eugenio=20P=C3=A9rez?= <
eperezma@redhat.com>
Date: Tue, 11 Feb 2020 13:19:10 +0100
Subject: [PATCH] vhost: fix return value of vhost_get_vq_desc

Before of the batch change, it was the chain's head. Need to keep that
way or we will not be able to free a chain of descriptors.

Fixes: eccb852f1fe6 ("vhost: batching fetches")
---
 drivers/vhost/vhost.c | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)

diff --git a/drivers/vhost/vhost.c b/drivers/vhost/vhost.c
index b5a51b1f2e79..fc422c3e5c08 100644
--- a/drivers/vhost/vhost.c
+++ b/drivers/vhost/vhost.c
@@ -2409,12 +2409,11 @@ int vhost_get_vq_desc(struct vhost_virtqueue *vq,
 			*out_num += ret;
 		}
 
-		ret = desc->id;
-
 		if (!(desc->flags & VRING_DESC_F_NEXT))
 			break;
 	}
 
+	ret = vq->descs[vq->first_desc].id;
 	vq->first_desc = i + 1;
 
 	return ret;
-- 
2.18.1

Re: vhost changes (batched) in linux-next after 12/13 trigger random crashes in KVM guests after reboot

[Christian Borntraeger]

On 11.02.20 14:04, Eugenio Pérez wrote:
> On Mon, 2020-02-10 at 12:01 +0100, Christian Borntraeger wrote:
>>
>> On 10.02.20 10:47, Eugenio Perez Martin wrote:
>>> Hi Christian.
>>>
>>> I'm not able to reproduce the failure with eccb852f1fe6bede630e2e4f1a121a81e34354ab commit. Could you add more data?
>>> Your configuration (libvirt or qemu line), and host's dmesg output if any?
>>>
>>> Thanks!
>>
>> If it was not obvious, this is on s390x, a big endian system.
>>
> 
> Hi Christian. Thank you very much for your fast responses.
> 
> Could you try this patch on top of eccb852f1fe6bede630e2e4f1a121a81e34354ab?

I still get 
[   43.665145] Guest moved used index from 0 to 289
after some reboots.


> 
> Thanks!
> 
> From 71d0f9108a18aa894cc0c0c1c7efbad39f465a27 Mon Sep 17 00:00:00 2001
> From: =?UTF-8?q?Eugenio=20P=C3=A9rez?= <
> eperezma@redhat.com>
> Date: Tue, 11 Feb 2020 13:19:10 +0100
> Subject: [PATCH] vhost: fix return value of vhost_get_vq_desc
> 
> Before of the batch change, it was the chain's head. Need to keep that
> way or we will not be able to free a chain of descriptors.
> 
> Fixes: eccb852f1fe6 ("vhost: batching fetches")
> ---
>  drivers/vhost/vhost.c | 3 +--
>  1 file changed, 1 insertion(+), 2 deletions(-)
> 
> diff --git a/drivers/vhost/vhost.c b/drivers/vhost/vhost.c
> index b5a51b1f2e79..fc422c3e5c08 100644
> --- a/drivers/vhost/vhost.c
> +++ b/drivers/vhost/vhost.c
> @@ -2409,12 +2409,11 @@ int vhost_get_vq_desc(struct vhost_virtqueue *vq,
>  			*out_num += ret;
>  		}
>  
> -		ret = desc->id;
> -
>  		if (!(desc->flags & VRING_DESC_F_NEXT))
>  			break;
>  	}
>  
> +	ret = vq->descs[vq->first_desc].id;
>  	vq->first_desc = i + 1;
>  
>  	return ret;
> 

Re: vhost changes (batched) in linux-next after 12/13 trigger random crashes in KVM guests after reboot

[Eugenio Pérez]

On Tue, 2020-02-11 at 14:13 +0100, Christian Borntraeger wrote:
> 
> On 11.02.20 14:04, Eugenio Pérez wrote:
> > On Mon, 2020-02-10 at 12:01 +0100, Christian Borntraeger wrote:
> > > On 10.02.20 10:47, Eugenio Perez Martin wrote:
> > > > Hi Christian.
> > > > 
> > > > I'm not able to reproduce the failure with eccb852f1fe6bede630e2e4f1a121a81e34354ab commit. Could you add more
> > > > data?
> > > > Your configuration (libvirt or qemu line), and host's dmesg output if any?
> > > > 
> > > > Thanks!
> > > 
> > > If it was not obvious, this is on s390x, a big endian system.
> > > 
> > 
> > Hi Christian. Thank you very much for your fast responses.
> > 
> > Could you try this patch on top of eccb852f1fe6bede630e2e4f1a121a81e34354ab?
> 
> I still get 
> [   43.665145] Guest moved used index from 0 to 289
> after some reboots.
> 
> 
> > Thanks!
> > 
> > From 71d0f9108a18aa894cc0c0c1c7efbad39f465a27 Mon Sep 17 00:00:00 2001
> > From: =?UTF-8?q?Eugenio=20P=C3=A9rez?= <
> > eperezma@redhat.com>
> > Date: Tue, 11 Feb 2020 13:19:10 +0100
> > Subject: [PATCH] vhost: fix return value of vhost_get_vq_desc
> > 
> > Before of the batch change, it was the chain's head. Need to keep that
> > way or we will not be able to free a chain of descriptors.
> > 
> > Fixes: eccb852f1fe6 ("vhost: batching fetches")
> > ---
> >  drivers/vhost/vhost.c | 3 +--
> >  1 file changed, 1 insertion(+), 2 deletions(-)
> > 
> > diff --git a/drivers/vhost/vhost.c b/drivers/vhost/vhost.c
> > index b5a51b1f2e79..fc422c3e5c08 100644
> > --- a/drivers/vhost/vhost.c
> > +++ b/drivers/vhost/vhost.c
> > @@ -2409,12 +2409,11 @@ int vhost_get_vq_desc(struct vhost_virtqueue *vq,
> >  			*out_num += ret;
> >  		}
> >  
> > -		ret = desc->id;
> > -
> >  		if (!(desc->flags & VRING_DESC_F_NEXT))
> >  			break;
> >  	}
> >  
> > +	ret = vq->descs[vq->first_desc].id;
> >  	vq->first_desc = i + 1;
> >  
> >  	return ret;
> > 

Sorry, still not able to reproduce the issue.

Could we try to disable all the vhost features?

Thanks!

diff --git a/drivers/vhost/vhost.h b/drivers/vhost/vhost.h
index 661088ae6dc7..08f6d2ccb697 100644
--- a/drivers/vhost/vhost.h
+++ b/drivers/vhost/vhost.h
@@ -250,11 +250,11 @@ int vhost_init_device_iotlb(struct vhost_dev *d, bool enabled);
        } while (0)
 
 enum {
-       VHOST_FEATURES = (1ULL << VIRTIO_F_NOTIFY_ON_EMPTY) |
-                        (1ULL << VIRTIO_RING_F_INDIRECT_DESC) |
-                        (1ULL << VIRTIO_RING_F_EVENT_IDX) |
-                        (1ULL << VHOST_F_LOG_ALL) |
-                        (1ULL << VIRTIO_F_ANY_LAYOUT) |
+       VHOST_FEATURES = /* (1ULL << VIRTIO_F_NOTIFY_ON_EMPTY) | */
+                        /* (1ULL << VIRTIO_RING_F_INDIRECT_DESC) | */
+                        /* (1ULL << VIRTIO_RING_F_EVENT_IDX) | */
+                        /* (1ULL << VHOST_F_LOG_ALL) | */
+                        /* (1ULL << VIRTIO_F_ANY_LAYOUT) | */
                         (1ULL << VIRTIO_F_VERSION_1)
 };

Re: vhost changes (batched) in linux-next after 12/13 trigger random crashes in KVM guests after reboot

[Christian Borntraeger]

On 12.02.20 17:34, Eugenio Pérez wrote:
> On Tue, 2020-02-11 at 14:13 +0100, Christian Borntraeger wrote:
>>
>> On 11.02.20 14:04, Eugenio Pérez wrote:
>>> On Mon, 2020-02-10 at 12:01 +0100, Christian Borntraeger wrote:
>>>> On 10.02.20 10:47, Eugenio Perez Martin wrote:
>>>>> Hi Christian.
>>>>>
>>>>> I'm not able to reproduce the failure with eccb852f1fe6bede630e2e4f1a121a81e34354ab commit. Could you add more
>>>>> data?
>>>>> Your configuration (libvirt or qemu line), and host's dmesg output if any?
>>>>>
>>>>> Thanks!
>>>>
>>>> If it was not obvious, this is on s390x, a big endian system.
>>>>
>>>
>>> Hi Christian. Thank you very much for your fast responses.
>>>
>>> Could you try this patch on top of eccb852f1fe6bede630e2e4f1a121a81e34354ab?
>>
>> I still get 
>> [   43.665145] Guest moved used index from 0 to 289
>> after some reboots.
>>
>>
>>> Thanks!
>>>
>>> From 71d0f9108a18aa894cc0c0c1c7efbad39f465a27 Mon Sep 17 00:00:00 2001
>>> From: =?UTF-8?q?Eugenio=20P=C3=A9rez?= <
>>> eperezma@redhat.com>
>>> Date: Tue, 11 Feb 2020 13:19:10 +0100
>>> Subject: [PATCH] vhost: fix return value of vhost_get_vq_desc
>>>
>>> Before of the batch change, it was the chain's head. Need to keep that
>>> way or we will not be able to free a chain of descriptors.
>>>
>>> Fixes: eccb852f1fe6 ("vhost: batching fetches")
>>> ---
>>>  drivers/vhost/vhost.c | 3 +--
>>>  1 file changed, 1 insertion(+), 2 deletions(-)
>>>
>>> diff --git a/drivers/vhost/vhost.c b/drivers/vhost/vhost.c
>>> index b5a51b1f2e79..fc422c3e5c08 100644
>>> --- a/drivers/vhost/vhost.c
>>> +++ b/drivers/vhost/vhost.c
>>> @@ -2409,12 +2409,11 @@ int vhost_get_vq_desc(struct vhost_virtqueue *vq,
>>>  			*out_num += ret;
>>>  		}
>>>  
>>> -		ret = desc->id;
>>> -
>>>  		if (!(desc->flags & VRING_DESC_F_NEXT))
>>>  			break;
>>>  	}
>>>  
>>> +	ret = vq->descs[vq->first_desc].id;
>>>  	vq->first_desc = i + 1;
>>>  
>>>  	return ret;
>>>
> 
> Sorry, still not able to reproduce the issue.
> 
> Could we try to disable all the vhost features?
> 
> Thanks!
> 
> diff --git a/drivers/vhost/vhost.h b/drivers/vhost/vhost.h
> index 661088ae6dc7..08f6d2ccb697 100644
> --- a/drivers/vhost/vhost.h
> +++ b/drivers/vhost/vhost.h
> @@ -250,11 +250,11 @@ int vhost_init_device_iotlb(struct vhost_dev *d, bool enabled);
>         } while (0)
>  
>  enum {
> -       VHOST_FEATURES = (1ULL << VIRTIO_F_NOTIFY_ON_EMPTY) |
> -                        (1ULL << VIRTIO_RING_F_INDIRECT_DESC) |
> -                        (1ULL << VIRTIO_RING_F_EVENT_IDX) |
> -                        (1ULL << VHOST_F_LOG_ALL) |
> -                        (1ULL << VIRTIO_F_ANY_LAYOUT) |
> +       VHOST_FEATURES = /* (1ULL << VIRTIO_F_NOTIFY_ON_EMPTY) | */
> +                        /* (1ULL << VIRTIO_RING_F_INDIRECT_DESC) | */
> +                        /* (1ULL << VIRTIO_RING_F_EVENT_IDX) | */
> +                        /* (1ULL << VHOST_F_LOG_ALL) | */
> +                        /* (1ULL << VIRTIO_F_ANY_LAYOUT) | */
>                          (1ULL << VIRTIO_F_VERSION_1)
>  };
> 


I still get  guest crashes with this on top of eccb852f1fe6. (The patch did not
apply, I had to manually comment out these things)

Re: vhost changes (batched) in linux-next after 12/13 trigger random crashes in KVM guests after reboot

[Eugenio Pérez]

On Thu, 2020-02-13 at 10:30 +0100, Christian Borntraeger wrote:
> 
> On 12.02.20 17:34, Eugenio Pérez wrote:
> > On Tue, 2020-02-11 at 14:13 +0100, Christian Borntraeger wrote:
> > > On 11.02.20 14:04, Eugenio Pérez wrote:
> > > > On Mon, 2020-02-10 at 12:01 +0100, Christian Borntraeger wrote:
> > > > > On 10.02.20 10:47, Eugenio Perez Martin wrote:
> > > > > > Hi Christian.
> > > > > > 
> > > > > > I'm not able to reproduce the failure with eccb852f1fe6bede630e2e4f1a121a81e34354ab commit. Could you add
> > > > > > more
> > > > > > data?
> > > > > > Your configuration (libvirt or qemu line), and host's dmesg output if any?
> > > > > > 
> > > > > > Thanks!
> > > > > 
> > > > > If it was not obvious, this is on s390x, a big endian system.
> > > > > 
> > > > 
> > > > Hi Christian. Thank you very much for your fast responses.
> > > > 
> > > > Could you try this patch on top of eccb852f1fe6bede630e2e4f1a121a81e34354ab?
> > > 
> > > I still get 
> > > [   43.665145] Guest moved used index from 0 to 289
> > > after some reboots.
> > > 
> > > 
> > > > Thanks!
> > > > 
> > > > From 71d0f9108a18aa894cc0c0c1c7efbad39f465a27 Mon Sep 17 00:00:00 2001
> > > > From: =?UTF-8?q?Eugenio=20P=C3=A9rez?= <
> > > > eperezma@redhat.com>
> > > > Date: Tue, 11 Feb 2020 13:19:10 +0100
> > > > Subject: [PATCH] vhost: fix return value of vhost_get_vq_desc
> > > > 
> > > > Before of the batch change, it was the chain's head. Need to keep that
> > > > way or we will not be able to free a chain of descriptors.
> > > > 
> > > > Fixes: eccb852f1fe6 ("vhost: batching fetches")
> > > > ---
> > > >  drivers/vhost/vhost.c | 3 +--
> > > >  1 file changed, 1 insertion(+), 2 deletions(-)
> > > > 
> > > > diff --git a/drivers/vhost/vhost.c b/drivers/vhost/vhost.c
> > > > index b5a51b1f2e79..fc422c3e5c08 100644
> > > > --- a/drivers/vhost/vhost.c
> > > > +++ b/drivers/vhost/vhost.c
> > > > @@ -2409,12 +2409,11 @@ int vhost_get_vq_desc(struct vhost_virtqueue *vq,
> > > >  			*out_num += ret;
> > > >  		}
> > > >  
> > > > -		ret = desc->id;
> > > > -
> > > >  		if (!(desc->flags & VRING_DESC_F_NEXT))
> > > >  			break;
> > > >  	}
> > > >  
> > > > +	ret = vq->descs[vq->first_desc].id;
> > > >  	vq->first_desc = i + 1;
> > > >  
> > > >  	return ret;
> > > > 
> > 
> > Sorry, still not able to reproduce the issue.
> > 
> > Could we try to disable all the vhost features?
> > 
> > Thanks!
> > 
> > diff --git a/drivers/vhost/vhost.h b/drivers/vhost/vhost.h
> > index 661088ae6dc7..08f6d2ccb697 100644
> > --- a/drivers/vhost/vhost.h
> > +++ b/drivers/vhost/vhost.h
> > @@ -250,11 +250,11 @@ int vhost_init_device_iotlb(struct vhost_dev *d, bool enabled);
> >         } while (0)
> >  
> >  enum {
> > -       VHOST_FEATURES = (1ULL << VIRTIO_F_NOTIFY_ON_EMPTY) |
> > -                        (1ULL << VIRTIO_RING_F_INDIRECT_DESC) |
> > -                        (1ULL << VIRTIO_RING_F_EVENT_IDX) |
> > -                        (1ULL << VHOST_F_LOG_ALL) |
> > -                        (1ULL << VIRTIO_F_ANY_LAYOUT) |
> > +       VHOST_FEATURES = /* (1ULL << VIRTIO_F_NOTIFY_ON_EMPTY) | */
> > +                        /* (1ULL << VIRTIO_RING_F_INDIRECT_DESC) | */
> > +                        /* (1ULL << VIRTIO_RING_F_EVENT_IDX) | */
> > +                        /* (1ULL << VHOST_F_LOG_ALL) | */
> > +                        /* (1ULL << VIRTIO_F_ANY_LAYOUT) | */
> >                          (1ULL << VIRTIO_F_VERSION_1)
> >  };
> > 
> 
> I still get  guest crashes with this on top of eccb852f1fe6. (The patch did not
> apply, I had to manually comment out these things)
> 

Sorry about that, I C&P transformed tabs to spaces.

Can we try tracing last_avail_idx with the attached patch? Can you enable also line and thread id (dyndbg='+plt')?

Thanks!

From f7012e8b9db711b12d36e6e97411e7afa34bf768 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Eugenio=20P=C3=A9rez?= <eperezma@redhat.com>
Date: Thu, 13 Feb 2020 11:26:06 +0100
Subject: [PATCH] vhost: disable all features and trace last_avail_idx

---
 drivers/vhost/vhost.c | 13 +++++++++++--
 drivers/vhost/vhost.h | 10 +++++-----
 2 files changed, 16 insertions(+), 7 deletions(-)

diff --git a/drivers/vhost/vhost.c b/drivers/vhost/vhost.c
index fc422c3e5c08..021d70bed015 100644
--- a/drivers/vhost/vhost.c
+++ b/drivers/vhost/vhost.c
@@ -1645,6 +1645,9 @@ long vhost_vring_ioctl(struct vhost_dev *d, unsigned int ioctl, void __user *arg
 		vq->last_avail_idx = s.num;
 		/* Forget the cached index value. */
 		vq->avail_idx = vq->last_avail_idx;
+		pr_debug(
+			"VHOST_SET_VRING_BASE [vq=%p][vq->last_avail_idx=%u][vq->avail_idx=%u]",
+			vq, vq->last_avail_idx, vq->avail_idx);
 		break;
 	case VHOST_GET_VRING_BASE:
 		s.index = idx;
@@ -2239,8 +2242,8 @@ static int fetch_buf(struct vhost_virtqueue *vq)
 		vq->avail_idx = vhost16_to_cpu(vq, avail_idx);
 
 		if (unlikely((u16)(vq->avail_idx - last_avail_idx) > vq->num)) {
-			vq_err(vq, "Guest moved used index from %u to %u",
-				last_avail_idx, vq->avail_idx);
+			vq_err(vq, "Guest moved vq %p used index from %u to %u",
+				vq, last_avail_idx, vq->avail_idx);
 			return -EFAULT;
 		}
 
@@ -2316,6 +2319,9 @@ static int fetch_buf(struct vhost_virtqueue *vq)
 	BUG_ON(!(vq->used_flags & VRING_USED_F_NO_NOTIFY));
 
 	/* On success, increment avail index. */
+	pr_debug(
+		"[vq=%p][vq->last_avail_idx=%u][vq->avail_idx=%u][vq->ndescs=%d][vq->first_desc=%d]",
+		vq, vq->last_avail_idx, vq->avail_idx, vq->ndescs, vq->first_desc);
 	vq->last_avail_idx++;
 
 	return 0;
@@ -2431,6 +2437,9 @@ EXPORT_SYMBOL_GPL(vhost_get_vq_desc);
 /* Reverse the effect of vhost_get_vq_desc. Useful for error handling. */
 void vhost_discard_vq_desc(struct vhost_virtqueue *vq, int n)
 {
+	pr_debug(
+		"DISCARD [vq=%p][vq->last_avail_idx=%u][vq->avail_idx=%u][n=%d]",
+		vq, vq->last_avail_idx, vq->avail_idx, n);
 	vq->last_avail_idx -= n;
 }
 EXPORT_SYMBOL_GPL(vhost_discard_vq_desc);
diff --git a/drivers/vhost/vhost.h b/drivers/vhost/vhost.h
index 661088ae6dc7..08f6d2ccb697 100644
--- a/drivers/vhost/vhost.h
+++ b/drivers/vhost/vhost.h
@@ -250,11 +250,11 @@ int vhost_init_device_iotlb(struct vhost_dev *d, bool enabled);
 	} while (0)
 
 enum {
-	VHOST_FEATURES = (1ULL << VIRTIO_F_NOTIFY_ON_EMPTY) |
-			 (1ULL << VIRTIO_RING_F_INDIRECT_DESC) |
-			 (1ULL << VIRTIO_RING_F_EVENT_IDX) |
-			 (1ULL << VHOST_F_LOG_ALL) |
-			 (1ULL << VIRTIO_F_ANY_LAYOUT) |
+	VHOST_FEATURES = /* (1ULL << VIRTIO_F_NOTIFY_ON_EMPTY) | */
+			 /* (1ULL << VIRTIO_RING_F_INDIRECT_DESC) | */
+			 /* (1ULL << VIRTIO_RING_F_EVENT_IDX) | */
+			 /* (1ULL << VHOST_F_LOG_ALL) | */
+			 /* (1ULL << VIRTIO_F_ANY_LAYOUT) | */
 			 (1ULL << VIRTIO_F_VERSION_1)
 };
 
-- 
2.18.1

Re: vhost changes (batched) in linux-next after 12/13 trigger random crashes in KVM guests after reboot

[Christian Borntraeger]

[-- Attachment #1: Type: text/plain, Size: 169 bytes --]



On 13.02.20 11:47, Eugenio Pérez wrote:
> 
> Can we try tracing last_avail_idx with the attached patch? Can you enable also line and thread id (dyndbg='+plt')?







[-- Attachment #2: log --]
[-- Type: text/plain, Size: 23398 bytes --]

[  326.584446] [2127] 1648: VHOST_SET_VRING_BASE [vq=0000000036fdfcb7][vq->last_avail_idx=0][vq->avail_idx=0]
[  326.584457] [2127] 1648: VHOST_SET_VRING_BASE [vq=0000000060d7302b][vq->last_avail_idx=0][vq->avail_idx=0]
[  326.584474] [2124] 2441: DISCARD [vq=0000000036fdfcb7][vq->last_avail_idx=0][vq->avail_idx=0][n=0]
[  327.281901] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=0][vq->avail_idx=256][vq->ndescs=1][vq->first_desc=0]
[  327.281902] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=1][vq->avail_idx=256][vq->ndescs=2][vq->first_desc=0]
[  327.281903] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=2][vq->avail_idx=256][vq->ndescs=3][vq->first_desc=0]
[  327.281904] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=3][vq->avail_idx=256][vq->ndescs=4][vq->first_desc=0]
[  327.281905] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=4][vq->avail_idx=256][vq->ndescs=5][vq->first_desc=0]
[  327.281906] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=5][vq->avail_idx=256][vq->ndescs=6][vq->first_desc=0]
[  327.281907] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=6][vq->avail_idx=256][vq->ndescs=7][vq->first_desc=0]
[  327.281908] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=7][vq->avail_idx=256][vq->ndescs=8][vq->first_desc=0]
[  327.281909] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=8][vq->avail_idx=256][vq->ndescs=9][vq->first_desc=0]
[  327.281910] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=9][vq->avail_idx=256][vq->ndescs=10][vq->first_desc=0]
[  327.281911] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=10][vq->avail_idx=256][vq->ndescs=11][vq->first_desc=0]
[  327.281911] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=11][vq->avail_idx=256][vq->ndescs=12][vq->first_desc=0]
[  327.281912] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=12][vq->avail_idx=256][vq->ndescs=13][vq->first_desc=0]
[  327.281913] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=13][vq->avail_idx=256][vq->ndescs=14][vq->first_desc=0]
[  327.281914] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=14][vq->avail_idx=256][vq->ndescs=15][vq->first_desc=0]
[  327.281915] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=15][vq->avail_idx=256][vq->ndescs=16][vq->first_desc=0]
[  327.281916] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=16][vq->avail_idx=256][vq->ndescs=17][vq->first_desc=0]
[  327.281916] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=17][vq->avail_idx=256][vq->ndescs=18][vq->first_desc=0]
[  327.281917] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=18][vq->avail_idx=256][vq->ndescs=19][vq->first_desc=0]
[  327.281918] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=19][vq->avail_idx=256][vq->ndescs=20][vq->first_desc=0]
[  327.281919] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=20][vq->avail_idx=256][vq->ndescs=21][vq->first_desc=0]
[  327.281920] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=21][vq->avail_idx=256][vq->ndescs=22][vq->first_desc=0]
[  327.281921] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=22][vq->avail_idx=256][vq->ndescs=23][vq->first_desc=0]
[  327.281921] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=23][vq->avail_idx=256][vq->ndescs=24][vq->first_desc=0]
[  327.281922] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=24][vq->avail_idx=256][vq->ndescs=25][vq->first_desc=0]
[  327.281923] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=25][vq->avail_idx=256][vq->ndescs=26][vq->first_desc=0]
[  327.281924] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=26][vq->avail_idx=256][vq->ndescs=27][vq->first_desc=0]
[  327.281925] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=27][vq->avail_idx=256][vq->ndescs=28][vq->first_desc=0]
[  327.281926] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=28][vq->avail_idx=256][vq->ndescs=29][vq->first_desc=0]
[  327.281926] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=29][vq->avail_idx=256][vq->ndescs=30][vq->first_desc=0]
[  327.281927] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=30][vq->avail_idx=256][vq->ndescs=31][vq->first_desc=0]
[  327.281928] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=31][vq->avail_idx=256][vq->ndescs=32][vq->first_desc=0]
[  327.281929] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=32][vq->avail_idx=256][vq->ndescs=33][vq->first_desc=0]
[  327.281930] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=33][vq->avail_idx=256][vq->ndescs=34][vq->first_desc=0]
[  327.281931] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=34][vq->avail_idx=256][vq->ndescs=35][vq->first_desc=0]
[  327.281932] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=35][vq->avail_idx=256][vq->ndescs=36][vq->first_desc=0]
[  327.281933] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=36][vq->avail_idx=256][vq->ndescs=37][vq->first_desc=0]
[  327.281934] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=37][vq->avail_idx=256][vq->ndescs=38][vq->first_desc=0]
[  327.281934] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=38][vq->avail_idx=256][vq->ndescs=39][vq->first_desc=0]
[  327.281935] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=39][vq->avail_idx=256][vq->ndescs=40][vq->first_desc=0]
[  327.281936] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=40][vq->avail_idx=256][vq->ndescs=41][vq->first_desc=0]
[  327.281937] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=41][vq->avail_idx=256][vq->ndescs=42][vq->first_desc=0]
[  327.281938] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=42][vq->avail_idx=256][vq->ndescs=43][vq->first_desc=0]
[  327.281939] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=43][vq->avail_idx=256][vq->ndescs=44][vq->first_desc=0]
[  327.281939] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=44][vq->avail_idx=256][vq->ndescs=45][vq->first_desc=0]
[  327.281940] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=45][vq->avail_idx=256][vq->ndescs=46][vq->first_desc=0]
[  327.281941] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=46][vq->avail_idx=256][vq->ndescs=47][vq->first_desc=0]
[  327.281942] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=47][vq->avail_idx=256][vq->ndescs=48][vq->first_desc=0]
[  327.281943] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=48][vq->avail_idx=256][vq->ndescs=49][vq->first_desc=0]
[  327.281944] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=49][vq->avail_idx=256][vq->ndescs=50][vq->first_desc=0]
[  327.281945] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=50][vq->avail_idx=256][vq->ndescs=51][vq->first_desc=0]
[  327.281946] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=51][vq->avail_idx=256][vq->ndescs=52][vq->first_desc=0]
[  327.281947] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=52][vq->avail_idx=256][vq->ndescs=53][vq->first_desc=0]
[  327.281948] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=53][vq->avail_idx=256][vq->ndescs=54][vq->first_desc=0]
[  327.281948] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=54][vq->avail_idx=256][vq->ndescs=55][vq->first_desc=0]
[  327.281949] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=55][vq->avail_idx=256][vq->ndescs=56][vq->first_desc=0]
[  327.281950] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=56][vq->avail_idx=256][vq->ndescs=57][vq->first_desc=0]
[  327.281951] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=57][vq->avail_idx=256][vq->ndescs=58][vq->first_desc=0]
[  327.281952] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=58][vq->avail_idx=256][vq->ndescs=59][vq->first_desc=0]
[  327.281953] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=59][vq->avail_idx=256][vq->ndescs=60][vq->first_desc=0]
[  327.281954] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=60][vq->avail_idx=256][vq->ndescs=61][vq->first_desc=0]
[  327.281955] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=61][vq->avail_idx=256][vq->ndescs=62][vq->first_desc=0]
[  327.281955] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=62][vq->avail_idx=256][vq->ndescs=63][vq->first_desc=0]
[  327.281956] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=63][vq->avail_idx=256][vq->ndescs=64][vq->first_desc=0]
[  327.281958] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=64][vq->avail_idx=256][vq->ndescs=65][vq->first_desc=0]
[  327.295289] [2124] 2322: [vq=0000000060d7302b][vq->last_avail_idx=0][vq->avail_idx=1][vq->ndescs=1][vq->first_desc=0]
[  327.313733] [2124] 2322: [vq=0000000060d7302b][vq->last_avail_idx=1][vq->avail_idx=2][vq->ndescs=1][vq->first_desc=0]
[  327.328259] [2124] 2322: [vq=0000000060d7302b][vq->last_avail_idx=2][vq->avail_idx=3][vq->ndescs=1][vq->first_desc=0]
[  327.328816] [2124] 2322: [vq=0000000060d7302b][vq->last_avail_idx=3][vq->avail_idx=5][vq->ndescs=1][vq->first_desc=0]
[  327.328818] [2124] 2322: [vq=0000000060d7302b][vq->last_avail_idx=4][vq->avail_idx=5][vq->ndescs=2][vq->first_desc=0]
[  327.553778] [2124] 2322: [vq=0000000060d7302b][vq->last_avail_idx=5][vq->avail_idx=6][vq->ndescs=1][vq->first_desc=0]
[  327.663779] [2124] 2322: [vq=0000000060d7302b][vq->last_avail_idx=6][vq->avail_idx=7][vq->ndescs=1][vq->first_desc=0]
[  328.433781] [2124] 2322: [vq=0000000060d7302b][vq->last_avail_idx=7][vq->avail_idx=8][vq->ndescs=1][vq->first_desc=0]
[  328.522897] [2124] 2322: [vq=0000000060d7302b][vq->last_avail_idx=8][vq->avail_idx=9][vq->ndescs=2][vq->first_desc=0]
[  328.593777] [2124] 2322: [vq=0000000060d7302b][vq->last_avail_idx=9][vq->avail_idx=10][vq->ndescs=1][vq->first_desc=0]
[  328.723911] [2124] 2322: [vq=0000000060d7302b][vq->last_avail_idx=10][vq->avail_idx=11][vq->ndescs=2][vq->first_desc=0]
[  328.926482] [2124] 2322: [vq=0000000060d7302b][vq->last_avail_idx=11][vq->avail_idx=12][vq->ndescs=2][vq->first_desc=0]
[  329.129059] [2124] 2322: [vq=0000000060d7302b][vq->last_avail_idx=12][vq->avail_idx=13][vq->ndescs=2][vq->first_desc=0]
[  329.295360] [2124] 2322: [vq=0000000060d7302b][vq->last_avail_idx=13][vq->avail_idx=14][vq->ndescs=1][vq->first_desc=0]
[  329.633801] [2124] 2322: [vq=0000000060d7302b][vq->last_avail_idx=14][vq->avail_idx=15][vq->ndescs=1][vq->first_desc=0]
[  329.663784] [2124] 2322: [vq=0000000060d7302b][vq->last_avail_idx=15][vq->avail_idx=16][vq->ndescs=1][vq->first_desc=0]
[  330.053043] [2124] 2322: [vq=0000000060d7302b][vq->last_avail_idx=16][vq->avail_idx=17][vq->ndescs=1][vq->first_desc=0]
[  330.111522] [2124] 2322: [vq=0000000060d7302b][vq->last_avail_idx=17][vq->avail_idx=18][vq->ndescs=1][vq->first_desc=0]
[  330.112176] [2124] 2322: [vq=0000000060d7302b][vq->last_avail_idx=18][vq->avail_idx=19][vq->ndescs=1][vq->first_desc=0]
[  330.112449] [2124] 2322: [vq=0000000060d7302b][vq->last_avail_idx=19][vq->avail_idx=20][vq->ndescs=1][vq->first_desc=0]
[  330.112619] [2124] 2322: [vq=0000000060d7302b][vq->last_avail_idx=20][vq->avail_idx=21][vq->ndescs=1][vq->first_desc=0]
[  330.112793] [2124] 2322: [vq=0000000060d7302b][vq->last_avail_idx=21][vq->avail_idx=22][vq->ndescs=1][vq->first_desc=0]
[  330.113028] [2124] 2322: [vq=0000000060d7302b][vq->last_avail_idx=22][vq->avail_idx=23][vq->ndescs=1][vq->first_desc=0]
[  330.113203] [2124] 2322: [vq=0000000060d7302b][vq->last_avail_idx=23][vq->avail_idx=24][vq->ndescs=1][vq->first_desc=0]
[  330.113366] [2124] 2322: [vq=0000000060d7302b][vq->last_avail_idx=24][vq->avail_idx=25][vq->ndescs=1][vq->first_desc=0]
[  330.113527] [2124] 2322: [vq=0000000060d7302b][vq->last_avail_idx=25][vq->avail_idx=26][vq->ndescs=1][vq->first_desc=0]
[  330.113686] [2124] 2322: [vq=0000000060d7302b][vq->last_avail_idx=26][vq->avail_idx=27][vq->ndescs=1][vq->first_desc=0]
[  330.113895] [2124] 2322: [vq=0000000060d7302b][vq->last_avail_idx=27][vq->avail_idx=28][vq->ndescs=1][vq->first_desc=0]
[  330.114064] [2124] 2322: [vq=0000000060d7302b][vq->last_avail_idx=28][vq->avail_idx=29][vq->ndescs=1][vq->first_desc=0]
[  330.114227] [2124] 2322: [vq=0000000060d7302b][vq->last_avail_idx=29][vq->avail_idx=30][vq->ndescs=1][vq->first_desc=0]
[  330.114389] [2124] 2322: [vq=0000000060d7302b][vq->last_avail_idx=30][vq->avail_idx=31][vq->ndescs=1][vq->first_desc=0]
[  330.114549] [2124] 2322: [vq=0000000060d7302b][vq->last_avail_idx=31][vq->avail_idx=32][vq->ndescs=1][vq->first_desc=0]
[  330.114708] [2124] 2322: [vq=0000000060d7302b][vq->last_avail_idx=32][vq->avail_idx=33][vq->ndescs=1][vq->first_desc=0]
[  330.114869] [2124] 2322: [vq=0000000060d7302b][vq->last_avail_idx=33][vq->avail_idx=34][vq->ndescs=1][vq->first_desc=0]
[  330.115033] [2124] 2322: [vq=0000000060d7302b][vq->last_avail_idx=34][vq->avail_idx=35][vq->ndescs=1][vq->first_desc=0]
[  330.115194] [2124] 2322: [vq=0000000060d7302b][vq->last_avail_idx=35][vq->avail_idx=36][vq->ndescs=1][vq->first_desc=0]
[  330.115355] [2124] 2322: [vq=0000000060d7302b][vq->last_avail_idx=36][vq->avail_idx=37][vq->ndescs=1][vq->first_desc=0]
[  330.115515] [2124] 2322: [vq=0000000060d7302b][vq->last_avail_idx=37][vq->avail_idx=38][vq->ndescs=1][vq->first_desc=0]
[  330.433757] [2124] 2322: [vq=0000000060d7302b][vq->last_avail_idx=38][vq->avail_idx=39][vq->ndescs=1][vq->first_desc=0]
[  330.593759] [2124] 2322: [vq=0000000060d7302b][vq->last_avail_idx=39][vq->avail_idx=40][vq->ndescs=1][vq->first_desc=0]
[  332.511365] [2124] 2322: [vq=0000000060d7302b][vq->last_avail_idx=40][vq->avail_idx=41][vq->ndescs=1][vq->first_desc=0]
[  333.245392] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=65][vq->avail_idx=256][vq->ndescs=1][vq->first_desc=0]
[  333.245393] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=66][vq->avail_idx=256][vq->ndescs=2][vq->first_desc=0]
[  333.245394] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=67][vq->avail_idx=256][vq->ndescs=3][vq->first_desc=0]
[  333.245395] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=68][vq->avail_idx=256][vq->ndescs=4][vq->first_desc=0]
[  333.245396] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=69][vq->avail_idx=256][vq->ndescs=5][vq->first_desc=0]
[  333.245397] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=70][vq->avail_idx=256][vq->ndescs=6][vq->first_desc=0]
[  333.245398] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=71][vq->avail_idx=256][vq->ndescs=7][vq->first_desc=0]
[  333.245398] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=72][vq->avail_idx=256][vq->ndescs=8][vq->first_desc=0]
[  333.245399] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=73][vq->avail_idx=256][vq->ndescs=9][vq->first_desc=0]
[  333.245400] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=74][vq->avail_idx=256][vq->ndescs=10][vq->first_desc=0]
[  333.245401] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=75][vq->avail_idx=256][vq->ndescs=11][vq->first_desc=0]
[  333.245402] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=76][vq->avail_idx=256][vq->ndescs=12][vq->first_desc=0]
[  333.245403] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=77][vq->avail_idx=256][vq->ndescs=13][vq->first_desc=0]
[  333.245404] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=78][vq->avail_idx=256][vq->ndescs=14][vq->first_desc=0]
[  333.245405] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=79][vq->avail_idx=256][vq->ndescs=15][vq->first_desc=0]
[  333.245406] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=80][vq->avail_idx=256][vq->ndescs=16][vq->first_desc=0]
[  333.245407] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=81][vq->avail_idx=256][vq->ndescs=17][vq->first_desc=0]
[  333.245408] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=82][vq->avail_idx=256][vq->ndescs=18][vq->first_desc=0]
[  333.245408] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=83][vq->avail_idx=256][vq->ndescs=19][vq->first_desc=0]
[  333.245409] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=84][vq->avail_idx=256][vq->ndescs=20][vq->first_desc=0]
[  333.245410] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=85][vq->avail_idx=256][vq->ndescs=21][vq->first_desc=0]
[  333.245411] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=86][vq->avail_idx=256][vq->ndescs=22][vq->first_desc=0]
[  333.245412] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=87][vq->avail_idx=256][vq->ndescs=23][vq->first_desc=0]
[  333.245413] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=88][vq->avail_idx=256][vq->ndescs=24][vq->first_desc=0]
[  333.245414] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=89][vq->avail_idx=256][vq->ndescs=25][vq->first_desc=0]
[  333.245414] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=90][vq->avail_idx=256][vq->ndescs=26][vq->first_desc=0]
[  333.245415] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=91][vq->avail_idx=256][vq->ndescs=27][vq->first_desc=0]
[  333.245416] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=92][vq->avail_idx=256][vq->ndescs=28][vq->first_desc=0]
[  333.245417] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=93][vq->avail_idx=256][vq->ndescs=29][vq->first_desc=0]
[  333.245418] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=94][vq->avail_idx=256][vq->ndescs=30][vq->first_desc=0]
[  333.245419] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=95][vq->avail_idx=256][vq->ndescs=31][vq->first_desc=0]
[  333.245420] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=96][vq->avail_idx=256][vq->ndescs=32][vq->first_desc=0]
[  333.245421] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=97][vq->avail_idx=256][vq->ndescs=33][vq->first_desc=0]
[  333.245422] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=98][vq->avail_idx=256][vq->ndescs=34][vq->first_desc=0]
[  333.245422] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=99][vq->avail_idx=256][vq->ndescs=35][vq->first_desc=0]
[  333.245423] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=100][vq->avail_idx=256][vq->ndescs=36][vq->first_desc=0]
[  333.245424] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=101][vq->avail_idx=256][vq->ndescs=37][vq->first_desc=0]
[  333.245425] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=102][vq->avail_idx=256][vq->ndescs=38][vq->first_desc=0]
[  333.245426] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=103][vq->avail_idx=256][vq->ndescs=39][vq->first_desc=0]
[  333.245427] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=104][vq->avail_idx=256][vq->ndescs=40][vq->first_desc=0]
[  333.245427] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=105][vq->avail_idx=256][vq->ndescs=41][vq->first_desc=0]
[  333.245428] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=106][vq->avail_idx=256][vq->ndescs=42][vq->first_desc=0]
[  333.245429] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=107][vq->avail_idx=256][vq->ndescs=43][vq->first_desc=0]
[  333.245430] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=108][vq->avail_idx=256][vq->ndescs=44][vq->first_desc=0]
[  333.245431] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=109][vq->avail_idx=256][vq->ndescs=45][vq->first_desc=0]
[  333.245431] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=110][vq->avail_idx=256][vq->ndescs=46][vq->first_desc=0]
[  333.245432] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=111][vq->avail_idx=256][vq->ndescs=47][vq->first_desc=0]
[  333.245433] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=112][vq->avail_idx=256][vq->ndescs=48][vq->first_desc=0]
[  333.245434] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=113][vq->avail_idx=256][vq->ndescs=49][vq->first_desc=0]
[  333.245435] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=114][vq->avail_idx=256][vq->ndescs=50][vq->first_desc=0]
[  333.245436] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=115][vq->avail_idx=256][vq->ndescs=51][vq->first_desc=0]
[  333.245437] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=116][vq->avail_idx=256][vq->ndescs=52][vq->first_desc=0]
[  333.245438] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=117][vq->avail_idx=256][vq->ndescs=53][vq->first_desc=0]
[  333.245438] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=118][vq->avail_idx=256][vq->ndescs=54][vq->first_desc=0]
[  333.245439] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=119][vq->avail_idx=256][vq->ndescs=55][vq->first_desc=0]
[  333.245440] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=120][vq->avail_idx=256][vq->ndescs=56][vq->first_desc=0]
[  333.245441] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=121][vq->avail_idx=256][vq->ndescs=57][vq->first_desc=0]
[  333.245442] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=122][vq->avail_idx=256][vq->ndescs=58][vq->first_desc=0]
[  333.245443] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=123][vq->avail_idx=256][vq->ndescs=59][vq->first_desc=0]
[  333.245443] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=124][vq->avail_idx=256][vq->ndescs=60][vq->first_desc=0]
[  333.245444] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=125][vq->avail_idx=256][vq->ndescs=61][vq->first_desc=0]
[  333.245445] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=126][vq->avail_idx=256][vq->ndescs=62][vq->first_desc=0]
[  333.245446] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=127][vq->avail_idx=256][vq->ndescs=63][vq->first_desc=0]
[  333.245447] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=128][vq->avail_idx=256][vq->ndescs=64][vq->first_desc=0]
[  333.245448] [2124] 2322: [vq=0000000036fdfcb7][vq->last_avail_idx=129][vq->avail_idx=256][vq->ndescs=65][vq->first_desc=0]
[  336.121638] [2126] 1648: VHOST_SET_VRING_BASE [vq=0000000036fdfcb7][vq->last_avail_idx=0][vq->avail_idx=0]
[  336.121654] [2126] 1648: VHOST_SET_VRING_BASE [vq=0000000060d7302b][vq->last_avail_idx=0][vq->avail_idx=0]
[  336.943877] [2124] 2322: [vq=0000000060d7302b][vq->last_avail_idx=0][vq->avail_idx=1][vq->ndescs=1][vq->first_desc=0]
[  336.957322] [2124] 2322: [vq=0000000060d7302b][vq->last_avail_idx=1][vq->avail_idx=2][vq->ndescs=1][vq->first_desc=0]
[  336.973438] [2124] 2322: [vq=0000000060d7302b][vq->last_avail_idx=2][vq->avail_idx=3][vq->ndescs=1][vq->first_desc=0]
[  337.223453] [2124] 2322: [vq=0000000060d7302b][vq->last_avail_idx=3][vq->avail_idx=4][vq->ndescs=1][vq->first_desc=0]
[  337.383453] [2124] 2322: [vq=0000000060d7302b][vq->last_avail_idx=4][vq->avail_idx=5][vq->ndescs=1][vq->first_desc=0]
[  337.473429] [2124] 2322: [vq=0000000060d7302b][vq->last_avail_idx=5][vq->avail_idx=6][vq->ndescs=1][vq->first_desc=0]
[  337.993434] [2124] 2322: [vq=0000000060d7302b][vq->last_avail_idx=6][vq->avail_idx=7][vq->ndescs=1][vq->first_desc=0]
[  338.393446] [2124] 2322: [vq=0000000060d7302b][vq->last_avail_idx=7][vq->avail_idx=8][vq->ndescs=1][vq->first_desc=0]
[  338.423446] [2124] 2322: [vq=0000000060d7302b][vq->last_avail_idx=8][vq->avail_idx=9][vq->ndescs=1][vq->first_desc=0]
[  338.693357] [2124] 2322: [vq=0000000060d7302b][vq->last_avail_idx=9][vq->avail_idx=10][vq->ndescs=1][vq->first_desc=0]
[  338.843477] [2124] 2322: [vq=0000000060d7302b][vq->last_avail_idx=10][vq->avail_idx=11][vq->ndescs=1][vq->first_desc=0]
[  338.943915] [2124] 2322: [vq=0000000060d7302b][vq->last_avail_idx=11][vq->avail_idx=12][vq->ndescs=1][vq->first_desc=0]
[  339.033436] [2124] 2322: [vq=0000000060d7302b][vq->last_avail_idx=12][vq->avail_idx=13][vq->ndescs=1][vq->first_desc=0]

Re: vhost changes (batched) in linux-next after 12/13 trigger random crashes in KVM guests after reboot

[Eugenio Pérez]

Can we try with this traces?

From b793b4106085ab1970bdedb340e49f37843ed585 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Eugenio=20P=C3=A9rez?= <eperezma@redhat.com>
Date: Thu, 13 Feb 2020 17:27:05 +0100
Subject: [PATCH] vhost: Add debug in ioctl calls

---
 drivers/vhost/net.c   | 20 +++++++++++++++++---
 drivers/vhost/vhost.c | 16 ++++++++++++++--
 2 files changed, 31 insertions(+), 5 deletions(-)

diff --git a/drivers/vhost/net.c b/drivers/vhost/net.c
index e158159671fa..e4d5f843f9c0 100644
--- a/drivers/vhost/net.c
+++ b/drivers/vhost/net.c
@@ -1505,10 +1505,13 @@ static long vhost_net_set_backend(struct vhost_net *n, unsigned index, int fd)
 
 	mutex_lock(&n->dev.mutex);
 	r = vhost_dev_check_owner(&n->dev);
-	if (r)
+	if (r) {
+		pr_debug("vhost_dev_check_owner index=%u fd=%d rc r=%d", index, fd, r);
 		goto err;
+	}
 
 	if (index >= VHOST_NET_VQ_MAX) {
+		pr_debug("vhost_dev_check_owner index=%u fd=%d MAX=%d", index, fd, VHOST_NET_VQ_MAX);
 		r = -ENOBUFS;
 		goto err;
 	}
@@ -1518,22 +1521,26 @@ static long vhost_net_set_backend(struct vhost_net *n, unsigned index, int fd)
 
 	/* Verify that ring has been setup correctly. */
 	if (!vhost_vq_access_ok(vq)) {
+		pr_debug("vhost_net_set_backend index=%u fd=%d !vhost_vq_access_ok", index, fd);
 		r = -EFAULT;
 		goto err_vq;
 	}
 	sock = get_socket(fd);
 	if (IS_ERR(sock)) {
 		r = PTR_ERR(sock);
+		pr_debug("vhost_net_set_backend index=%u fd=%d get_socket err r=%d", index, fd, r);
 		goto err_vq;
 	}
 
 	/* start polling new socket */
 	oldsock = vq->private_data;
 	if (sock != oldsock) {
+		pr_debug("sock=%p != oldsock=%p index=%u fd=%d vq=%p", sock, oldsock, index, fd, vq);
 		ubufs = vhost_net_ubuf_alloc(vq,
 					     sock && vhost_sock_zcopy(sock));
 		if (IS_ERR(ubufs)) {
 			r = PTR_ERR(ubufs);
+			pr_debug("ubufs index=%u fd=%d err r=%d vq=%p", index, fd, r, vq);
 			goto err_ubufs;
 		}
 
@@ -1541,11 +1548,15 @@ static long vhost_net_set_backend(struct vhost_net *n, unsigned index, int fd)
 		vq->private_data = sock;
 		vhost_net_buf_unproduce(nvq);
 		r = vhost_vq_init_access(vq);
-		if (r)
+		if (r) {
+			pr_debug("init_access index=%u fd=%d r=%d vq=%p", index, fd, r, vq);
 			goto err_used;
+		}
 		r = vhost_net_enable_vq(n, vq);
-		if (r)
+		if (r) {
+			pr_debug("enable_vq index=%u fd=%d r=%d vq=%p", index, fd, r, vq);
 			goto err_used;
+		}
 		if (index == VHOST_NET_VQ_RX)
 			nvq->rx_ring = get_tap_ptr_ring(fd);
 
@@ -1559,6 +1570,8 @@ static long vhost_net_set_backend(struct vhost_net *n, unsigned index, int fd)
 
 	mutex_unlock(&vq->mutex);
 
+	pr_debug("sock=%p", sock);
+
 	if (oldubufs) {
 		vhost_net_ubuf_put_wait_and_free(oldubufs);
 		mutex_lock(&vq->mutex);
@@ -1710,6 +1723,7 @@ static long vhost_net_ioctl(struct file *f, unsigned int ioctl,
 
 	switch (ioctl) {
 	case VHOST_NET_SET_BACKEND:
+		pr_debug("VHOST_NET_SET_BACKEND");
 		if (copy_from_user(&backend, argp, sizeof backend))
 			return -EFAULT;
 		return vhost_net_set_backend(n, backend.index, backend.fd);
diff --git a/drivers/vhost/vhost.c b/drivers/vhost/vhost.c
index 021d70bed015..7f4848f9cec3 100644
--- a/drivers/vhost/vhost.c
+++ b/drivers/vhost/vhost.c
@@ -1642,18 +1642,30 @@ long vhost_vring_ioctl(struct vhost_dev *d, unsigned int ioctl, void __user *arg
 			r = -EINVAL;
 			break;
 		}
+
+		if (vq->last_avail_idx || vq->avail_idx) {
+			pr_debug(
+				"strange VHOST_SET_VRING_BASE [vq=%p][s.index=%u][s.num=%u]",
+				vq, s.index, s.num);
+			dump_stack();
+			r = 0;
+			break;
+		}
 		vq->last_avail_idx = s.num;
 		/* Forget the cached index value. */
 		vq->avail_idx = vq->last_avail_idx;
 		pr_debug(
-			"VHOST_SET_VRING_BASE [vq=%p][vq->last_avail_idx=%u][vq->avail_idx=%u]",
-			vq, vq->last_avail_idx, vq->avail_idx);
+			"VHOST_SET_VRING_BASE [vq=%p][vq->last_avail_idx=%u][vq->avail_idx=%u][s.index=%u][s.num=%u]",
+			vq, vq->last_avail_idx, vq->avail_idx, s.index, s.num);
 		break;
 	case VHOST_GET_VRING_BASE:
 		s.index = idx;
 		s.num = vq->last_avail_idx;
 		if (copy_to_user(argp, &s, sizeof s))
 			r = -EFAULT;
+		pr_debug(
+			"VHOST_GET_VRING_BASE [vq=%p][vq->last_avail_idx=%u][vq->avail_idx=%u][s.index=%u][s.num=%u]",
+			vq, vq->last_avail_idx, vq->avail_idx, s.index, s.num);
 		break;
 	case VHOST_SET_VRING_KICK:
 		if (copy_from_user(&f, argp, sizeof f)) {
-- 
2.18.1

Re: vhost changes (batched) in linux-next after 12/13 trigger random crashes in KVM guests after reboot

[Christian Borntraeger]

On 13.02.20 17:29, Eugenio Pérez wrote:
> Can we try with this traces?

Does not apply on eccb852f1fe6bede630e2e4f1a121a81e34354ab, can you double check?

> 
> From b793b4106085ab1970bdedb340e49f37843ed585 Mon Sep 17 00:00:00 2001
> From: =?UTF-8?q?Eugenio=20P=C3=A9rez?= <eperezma@redhat.com>
> Date: Thu, 13 Feb 2020 17:27:05 +0100
> Subject: [PATCH] vhost: Add debug in ioctl calls
> 
> ---
>  drivers/vhost/net.c   | 20 +++++++++++++++++---
>  drivers/vhost/vhost.c | 16 ++++++++++++++--
>  2 files changed, 31 insertions(+), 5 deletions(-)
> 
> diff --git a/drivers/vhost/net.c b/drivers/vhost/net.c
> index e158159671fa..e4d5f843f9c0 100644
> --- a/drivers/vhost/net.c
> +++ b/drivers/vhost/net.c
> @@ -1505,10 +1505,13 @@ static long vhost_net_set_backend(struct vhost_net *n, unsigned index, int fd)
>  
>  	mutex_lock(&n->dev.mutex);
>  	r = vhost_dev_check_owner(&n->dev);
> -	if (r)
> +	if (r) {
> +		pr_debug("vhost_dev_check_owner index=%u fd=%d rc r=%d", index, fd, r);
>  		goto err;
> +	}
>  
>  	if (index >= VHOST_NET_VQ_MAX) {
> +		pr_debug("vhost_dev_check_owner index=%u fd=%d MAX=%d", index, fd, VHOST_NET_VQ_MAX);
>  		r = -ENOBUFS;
>  		goto err;
>  	}
> @@ -1518,22 +1521,26 @@ static long vhost_net_set_backend(struct vhost_net *n, unsigned index, int fd)
>  
>  	/* Verify that ring has been setup correctly. */
>  	if (!vhost_vq_access_ok(vq)) {
> +		pr_debug("vhost_net_set_backend index=%u fd=%d !vhost_vq_access_ok", index, fd);
>  		r = -EFAULT;
>  		goto err_vq;
>  	}
>  	sock = get_socket(fd);
>  	if (IS_ERR(sock)) {
>  		r = PTR_ERR(sock);
> +		pr_debug("vhost_net_set_backend index=%u fd=%d get_socket err r=%d", index, fd, r);
>  		goto err_vq;
>  	}
>  
>  	/* start polling new socket */
>  	oldsock = vq->private_data;
>  	if (sock != oldsock) {
> +		pr_debug("sock=%p != oldsock=%p index=%u fd=%d vq=%p", sock, oldsock, index, fd, vq);
>  		ubufs = vhost_net_ubuf_alloc(vq,
>  					     sock && vhost_sock_zcopy(sock));
>  		if (IS_ERR(ubufs)) {
>  			r = PTR_ERR(ubufs);
> +			pr_debug("ubufs index=%u fd=%d err r=%d vq=%p", index, fd, r, vq);
>  			goto err_ubufs;
>  		}
>  
> @@ -1541,11 +1548,15 @@ static long vhost_net_set_backend(struct vhost_net *n, unsigned index, int fd)
>  		vq->private_data = sock;
>  		vhost_net_buf_unproduce(nvq);
>  		r = vhost_vq_init_access(vq);
> -		if (r)
> +		if (r) {
> +			pr_debug("init_access index=%u fd=%d r=%d vq=%p", index, fd, r, vq);
>  			goto err_used;
> +		}
>  		r = vhost_net_enable_vq(n, vq);
> -		if (r)
> +		if (r) {
> +			pr_debug("enable_vq index=%u fd=%d r=%d vq=%p", index, fd, r, vq);
>  			goto err_used;
> +		}
>  		if (index == VHOST_NET_VQ_RX)
>  			nvq->rx_ring = get_tap_ptr_ring(fd);
>  
> @@ -1559,6 +1570,8 @@ static long vhost_net_set_backend(struct vhost_net *n, unsigned index, int fd)
>  
>  	mutex_unlock(&vq->mutex);
>  
> +	pr_debug("sock=%p", sock);
> +
>  	if (oldubufs) {
>  		vhost_net_ubuf_put_wait_and_free(oldubufs);
>  		mutex_lock(&vq->mutex);
> @@ -1710,6 +1723,7 @@ static long vhost_net_ioctl(struct file *f, unsigned int ioctl,
>  
>  	switch (ioctl) {
>  	case VHOST_NET_SET_BACKEND:
> +		pr_debug("VHOST_NET_SET_BACKEND");
>  		if (copy_from_user(&backend, argp, sizeof backend))
>  			return -EFAULT;
>  		return vhost_net_set_backend(n, backend.index, backend.fd);
> diff --git a/drivers/vhost/vhost.c b/drivers/vhost/vhost.c
> index 021d70bed015..7f4848f9cec3 100644
> --- a/drivers/vhost/vhost.c
> +++ b/drivers/vhost/vhost.c
> @@ -1642,18 +1642,30 @@ long vhost_vring_ioctl(struct vhost_dev *d, unsigned int ioctl, void __user *arg
>  			r = -EINVAL;
>  			break;
>  		}
> +
> +		if (vq->last_avail_idx || vq->avail_idx) {
> +			pr_debug(
> +				"strange VHOST_SET_VRING_BASE [vq=%p][s.index=%u][s.num=%u]",
> +				vq, s.index, s.num);
> +			dump_stack();
> +			r = 0;
> +			break;
> +		}
>  		vq->last_avail_idx = s.num;
>  		/* Forget the cached index value. */
>  		vq->avail_idx = vq->last_avail_idx;
>  		pr_debug(
> -			"VHOST_SET_VRING_BASE [vq=%p][vq->last_avail_idx=%u][vq->avail_idx=%u]",
> -			vq, vq->last_avail_idx, vq->avail_idx);
> +			"VHOST_SET_VRING_BASE [vq=%p][vq->last_avail_idx=%u][vq->avail_idx=%u][s.index=%u][s.num=%u]",
> +			vq, vq->last_avail_idx, vq->avail_idx, s.index, s.num);
>  		break;
>  	case VHOST_GET_VRING_BASE:
>  		s.index = idx;
>  		s.num = vq->last_avail_idx;
>  		if (copy_to_user(argp, &s, sizeof s))
>  			r = -EFAULT;
> +		pr_debug(
> +			"VHOST_GET_VRING_BASE [vq=%p][vq->last_avail_idx=%u][vq->avail_idx=%u][s.index=%u][s.num=%u]",
> +			vq, vq->last_avail_idx, vq->avail_idx, s.index, s.num);
>  		break;
>  	case VHOST_SET_VRING_KICK:
>  		if (copy_from_user(&f, argp, sizeof f)) {
> 

Re: vhost changes (batched) in linux-next after 12/13 trigger random crashes in KVM guests after reboot

[Eugenio Pérez]

Hi Christian.

Sorry, that was meant to be applied over previous debug patch.

Here I inline the one meant to be applied over eccb852f1fe6bede630e2e4f1a121a81e34354ab.

Thanks!

From d978ace99e4844b49b794d768385db3d128a4cc0 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Eugenio=20P=C3=A9rez?= <eperezma@redhat.com>
Date: Fri, 14 Feb 2020 08:02:26 +0100
Subject: [PATCH] vhost: disable all features and trace last_avail_idx and
 ioctl calls

---
 drivers/vhost/net.c   | 20 +++++++++++++++++---
 drivers/vhost/vhost.c | 25 +++++++++++++++++++++++--
 drivers/vhost/vhost.h | 10 +++++-----
 3 files changed, 45 insertions(+), 10 deletions(-)

diff --git a/drivers/vhost/net.c b/drivers/vhost/net.c
index e158159671fa..e4d5f843f9c0 100644
--- a/drivers/vhost/net.c
+++ b/drivers/vhost/net.c
@@ -1505,10 +1505,13 @@ static long vhost_net_set_backend(struct vhost_net *n, unsigned index, int fd)
 
 	mutex_lock(&n->dev.mutex);
 	r = vhost_dev_check_owner(&n->dev);
-	if (r)
+	if (r) {
+		pr_debug("vhost_dev_check_owner index=%u fd=%d rc r=%d", index, fd, r);
 		goto err;
+	}
 
 	if (index >= VHOST_NET_VQ_MAX) {
+		pr_debug("vhost_dev_check_owner index=%u fd=%d MAX=%d", index, fd, VHOST_NET_VQ_MAX);
 		r = -ENOBUFS;
 		goto err;
 	}
@@ -1518,22 +1521,26 @@ static long vhost_net_set_backend(struct vhost_net *n, unsigned index, int fd)
 
 	/* Verify that ring has been setup correctly. */
 	if (!vhost_vq_access_ok(vq)) {
+		pr_debug("vhost_net_set_backend index=%u fd=%d !vhost_vq_access_ok", index, fd);
 		r = -EFAULT;
 		goto err_vq;
 	}
 	sock = get_socket(fd);
 	if (IS_ERR(sock)) {
 		r = PTR_ERR(sock);
+		pr_debug("vhost_net_set_backend index=%u fd=%d get_socket err r=%d", index, fd, r);
 		goto err_vq;
 	}
 
 	/* start polling new socket */
 	oldsock = vq->private_data;
 	if (sock != oldsock) {
+		pr_debug("sock=%p != oldsock=%p index=%u fd=%d vq=%p", sock, oldsock, index, fd, vq);
 		ubufs = vhost_net_ubuf_alloc(vq,
 					     sock && vhost_sock_zcopy(sock));
 		if (IS_ERR(ubufs)) {
 			r = PTR_ERR(ubufs);
+			pr_debug("ubufs index=%u fd=%d err r=%d vq=%p", index, fd, r, vq);
 			goto err_ubufs;
 		}
 
@@ -1541,11 +1548,15 @@ static long vhost_net_set_backend(struct vhost_net *n, unsigned index, int fd)
 		vq->private_data = sock;
 		vhost_net_buf_unproduce(nvq);
 		r = vhost_vq_init_access(vq);
-		if (r)
+		if (r) {
+			pr_debug("init_access index=%u fd=%d r=%d vq=%p", index, fd, r, vq);
 			goto err_used;
+		}
 		r = vhost_net_enable_vq(n, vq);
-		if (r)
+		if (r) {
+			pr_debug("enable_vq index=%u fd=%d r=%d vq=%p", index, fd, r, vq);
 			goto err_used;
+		}
 		if (index == VHOST_NET_VQ_RX)
 			nvq->rx_ring = get_tap_ptr_ring(fd);
 
@@ -1559,6 +1570,8 @@ static long vhost_net_set_backend(struct vhost_net *n, unsigned index, int fd)
 
 	mutex_unlock(&vq->mutex);
 
+	pr_debug("sock=%p", sock);
+
 	if (oldubufs) {
 		vhost_net_ubuf_put_wait_and_free(oldubufs);
 		mutex_lock(&vq->mutex);
@@ -1710,6 +1723,7 @@ static long vhost_net_ioctl(struct file *f, unsigned int ioctl,
 
 	switch (ioctl) {
 	case VHOST_NET_SET_BACKEND:
+		pr_debug("VHOST_NET_SET_BACKEND");
 		if (copy_from_user(&backend, argp, sizeof backend))
 			return -EFAULT;
 		return vhost_net_set_backend(n, backend.index, backend.fd);
diff --git a/drivers/vhost/vhost.c b/drivers/vhost/vhost.c
index b5a51b1f2e79..ec25ba32fe81 100644
--- a/drivers/vhost/vhost.c
+++ b/drivers/vhost/vhost.c
@@ -1642,15 +1642,30 @@ long vhost_vring_ioctl(struct vhost_dev *d, unsigned int ioctl, void __user *arg
 			r = -EINVAL;
 			break;
 		}
+
+		if (vq->last_avail_idx || vq->avail_idx) {
+			pr_debug(
+				"strange VHOST_SET_VRING_BASE [vq=%p][s.index=%u][s.num=%u]",
+				vq, s.index, s.num);
+			dump_stack();
+			r = 0;
+			break;
+		}
 		vq->last_avail_idx = s.num;
 		/* Forget the cached index value. */
 		vq->avail_idx = vq->last_avail_idx;
+		pr_debug(
+			"VHOST_SET_VRING_BASE [vq=%p][vq->last_avail_idx=%u][vq->avail_idx=%u][s.index=%u][s.num=%u]",
+			vq, vq->last_avail_idx, vq->avail_idx, s.index, s.num);
 		break;
 	case VHOST_GET_VRING_BASE:
 		s.index = idx;
 		s.num = vq->last_avail_idx;
 		if (copy_to_user(argp, &s, sizeof s))
 			r = -EFAULT;
+		pr_debug(
+			"VHOST_GET_VRING_BASE [vq=%p][vq->last_avail_idx=%u][vq->avail_idx=%u][s.index=%u][s.num=%u]",
+			vq, vq->last_avail_idx, vq->avail_idx, s.index, s.num);
 		break;
 	case VHOST_SET_VRING_KICK:
 		if (copy_from_user(&f, argp, sizeof f)) {
@@ -2239,8 +2254,8 @@ static int fetch_buf(struct vhost_virtqueue *vq)
 		vq->avail_idx = vhost16_to_cpu(vq, avail_idx);
 
 		if (unlikely((u16)(vq->avail_idx - last_avail_idx) > vq->num)) {
-			vq_err(vq, "Guest moved used index from %u to %u",
-				last_avail_idx, vq->avail_idx);
+			vq_err(vq, "Guest moved vq %p used index from %u to %u",
+				vq, last_avail_idx, vq->avail_idx);
 			return -EFAULT;
 		}
 
@@ -2316,6 +2331,9 @@ static int fetch_buf(struct vhost_virtqueue *vq)
 	BUG_ON(!(vq->used_flags & VRING_USED_F_NO_NOTIFY));
 
 	/* On success, increment avail index. */
+	pr_debug(
+		"[vq=%p][vq->last_avail_idx=%u][vq->avail_idx=%u][vq->ndescs=%d][vq->first_desc=%d]",
+		vq, vq->last_avail_idx, vq->avail_idx, vq->ndescs, vq->first_desc);
 	vq->last_avail_idx++;
 
 	return 0;
@@ -2432,6 +2450,9 @@ EXPORT_SYMBOL_GPL(vhost_get_vq_desc);
 /* Reverse the effect of vhost_get_vq_desc. Useful for error handling. */
 void vhost_discard_vq_desc(struct vhost_virtqueue *vq, int n)
 {
+	pr_debug(
+		"DISCARD [vq=%p][vq->last_avail_idx=%u][vq->avail_idx=%u][n=%d]",
+		vq, vq->last_avail_idx, vq->avail_idx, n);
 	vq->last_avail_idx -= n;
 }
 EXPORT_SYMBOL_GPL(vhost_discard_vq_desc);
diff --git a/drivers/vhost/vhost.h b/drivers/vhost/vhost.h
index 661088ae6dc7..08f6d2ccb697 100644
--- a/drivers/vhost/vhost.h
+++ b/drivers/vhost/vhost.h
@@ -250,11 +250,11 @@ int vhost_init_device_iotlb(struct vhost_dev *d, bool enabled);
 	} while (0)
 
 enum {
-	VHOST_FEATURES = (1ULL << VIRTIO_F_NOTIFY_ON_EMPTY) |
-			 (1ULL << VIRTIO_RING_F_INDIRECT_DESC) |
-			 (1ULL << VIRTIO_RING_F_EVENT_IDX) |
-			 (1ULL << VHOST_F_LOG_ALL) |
-			 (1ULL << VIRTIO_F_ANY_LAYOUT) |
+	VHOST_FEATURES = /* (1ULL << VIRTIO_F_NOTIFY_ON_EMPTY) | */
+			 /* (1ULL << VIRTIO_RING_F_INDIRECT_DESC) | */
+			 /* (1ULL << VIRTIO_RING_F_EVENT_IDX) | */
+			 /* (1ULL << VHOST_F_LOG_ALL) | */
+			 /* (1ULL << VIRTIO_F_ANY_LAYOUT) | */
 			 (1ULL << VIRTIO_F_VERSION_1)
 };
 
-- 
2.18.1

Re: vhost changes (batched) in linux-next after 12/13 trigger random crashes in KVM guests after reboot

[Christian Borntraeger]

I did 
ping -c 20 -f ... ; reboot 
twice

The ping after the first reboot showed .......E 

this was on the host console

[   55.951885] CPU: 34 PID: 1908 Comm: CPU 0/KVM Not tainted 5.5.0+ #21
[   55.951891] Hardware name: IBM 3906 M04 704 (LPAR)
[   55.951892] Call Trace:
[   55.951902]  [<0000001ede114132>] show_stack+0x8a/0xd0 
[   55.951906]  [<0000001edeb0672a>] dump_stack+0x8a/0xb8 
[   55.951915]  [<000003ff803736a6>] vhost_vring_ioctl+0x6fe/0x858 [vhost] 
[   55.951919]  [<000003ff8042a608>] vhost_net_ioctl+0x510/0x570 [vhost_net] 
[   55.951924]  [<0000001ede3c4dd8>] do_vfs_ioctl+0x430/0x6f8 
[   55.951926]  [<0000001ede3c5124>] ksys_ioctl+0x84/0xb0 
[   55.951927]  [<0000001ede3c51ba>] __s390x_sys_ioctl+0x2a/0x38 
[   55.951931]  [<0000001edeb27f72>] system_call+0x2a6/0x2c8 
[   55.951949] CPU: 34 PID: 1908 Comm: CPU 0/KVM Not tainted 5.5.0+ #21
[   55.951950] Hardware name: IBM 3906 M04 704 (LPAR)
[   55.951951] Call Trace:
[   55.951952]  [<0000001ede114132>] show_stack+0x8a/0xd0 
[   55.951954]  [<0000001edeb0672a>] dump_stack+0x8a/0xb8 
[   55.951956]  [<000003ff803736a6>] vhost_vring_ioctl+0x6fe/0x858 [vhost] 
[   55.951958]  [<000003ff8042a608>] vhost_net_ioctl+0x510/0x570 [vhost_net] 
[   55.951959]  [<0000001ede3c4dd8>] do_vfs_ioctl+0x430/0x6f8 
[   55.951961]  [<0000001ede3c5124>] ksys_ioctl+0x84/0xb0 
[   55.951962]  [<0000001ede3c51ba>] __s390x_sys_ioctl+0x2a/0x38 
[   55.951964]  [<0000001edeb27f72>] system_call+0x2a6/0x2c8 
[   55.951997] Guest moved vq 0000000063d896c6 used index from 44 to 0
[   56.609831] unexpected descriptor format for RX: out 0, in 0
[   86.540460] CPU: 6 PID: 1908 Comm: CPU 0/KVM Not tainted 5.5.0+ #21
[   86.540464] Hardware name: IBM 3906 M04 704 (LPAR)
[   86.540466] Call Trace:
[   86.540473]  [<0000001ede114132>] show_stack+0x8a/0xd0 
[   86.540477]  [<0000001edeb0672a>] dump_stack+0x8a/0xb8 
[   86.540486]  [<000003ff803736a6>] vhost_vring_ioctl+0x6fe/0x858 [vhost] 
[   86.540490]  [<000003ff8042a608>] vhost_net_ioctl+0x510/0x570 [vhost_net] 
[   86.540494]  [<0000001ede3c4dd8>] do_vfs_ioctl+0x430/0x6f8 
[   86.540496]  [<0000001ede3c5124>] ksys_ioctl+0x84/0xb0 
[   86.540498]  [<0000001ede3c51ba>] __s390x_sys_ioctl+0x2a/0x38 
[   86.540501]  [<0000001edeb27f72>] system_call+0x2a6/0x2c8 
[   86.540524] CPU: 6 PID: 1908 Comm: CPU 0/KVM Not tainted 5.5.0+ #21
[   86.540525] Hardware name: IBM 3906 M04 704 (LPAR)
[   86.540526] Call Trace:
[   86.540527]  [<0000001ede114132>] show_stack+0x8a/0xd0 
[   86.540528]  [<0000001edeb0672a>] dump_stack+0x8a/0xb8 
[   86.540531]  [<000003ff803736a6>] vhost_vring_ioctl+0x6fe/0x858 [vhost] 
[   86.540532]  [<000003ff8042a608>] vhost_net_ioctl+0x510/0x570 [vhost_net] 
[   86.540534]  [<0000001ede3c4dd8>] do_vfs_ioctl+0x430/0x6f8 
[   86.540536]  [<0000001ede3c5124>] ksys_ioctl+0x84/0xb0 
[   86.540537]  [<0000001ede3c51ba>] __s390x_sys_ioctl+0x2a/0x38 
[   86.540538]  [<0000001edeb27f72>] system_call+0x2a6/0x2c8 
[   86.540570] unexpected descriptor format for RX: out 0, in 0
[   86.540577] Unexpected header len for TX: 0 expected 0


On 14.02.20 08:06, Eugenio Pérez wrote:
> Hi Christian.
> 
> Sorry, that was meant to be applied over previous debug patch.
> 
> Here I inline the one meant to be applied over eccb852f1fe6bede630e2e4f1a121a81e34354ab.
> 
> Thanks!
> 
> From d978ace99e4844b49b794d768385db3d128a4cc0 Mon Sep 17 00:00:00 2001
> From: =?UTF-8?q?Eugenio=20P=C3=A9rez?= <eperezma@redhat.com>
> Date: Fri, 14 Feb 2020 08:02:26 +0100
> Subject: [PATCH] vhost: disable all features and trace last_avail_idx and
>  ioctl calls
> 
> ---
>  drivers/vhost/net.c   | 20 +++++++++++++++++---
>  drivers/vhost/vhost.c | 25 +++++++++++++++++++++++--
>  drivers/vhost/vhost.h | 10 +++++-----
>  3 files changed, 45 insertions(+), 10 deletions(-)
> 
> diff --git a/drivers/vhost/net.c b/drivers/vhost/net.c
> index e158159671fa..e4d5f843f9c0 100644
> --- a/drivers/vhost/net.c
> +++ b/drivers/vhost/net.c
> @@ -1505,10 +1505,13 @@ static long vhost_net_set_backend(struct vhost_net *n, unsigned index, int fd)
>  
>  	mutex_lock(&n->dev.mutex);
>  	r = vhost_dev_check_owner(&n->dev);
> -	if (r)
> +	if (r) {
> +		pr_debug("vhost_dev_check_owner index=%u fd=%d rc r=%d", index, fd, r);
>  		goto err;
> +	}
>  
>  	if (index >= VHOST_NET_VQ_MAX) {
> +		pr_debug("vhost_dev_check_owner index=%u fd=%d MAX=%d", index, fd, VHOST_NET_VQ_MAX);
>  		r = -ENOBUFS;
>  		goto err;
>  	}
> @@ -1518,22 +1521,26 @@ static long vhost_net_set_backend(struct vhost_net *n, unsigned index, int fd)
>  
>  	/* Verify that ring has been setup correctly. */
>  	if (!vhost_vq_access_ok(vq)) {
> +		pr_debug("vhost_net_set_backend index=%u fd=%d !vhost_vq_access_ok", index, fd);
>  		r = -EFAULT;
>  		goto err_vq;
>  	}
>  	sock = get_socket(fd);
>  	if (IS_ERR(sock)) {
>  		r = PTR_ERR(sock);
> +		pr_debug("vhost_net_set_backend index=%u fd=%d get_socket err r=%d", index, fd, r);
>  		goto err_vq;
>  	}
>  
>  	/* start polling new socket */
>  	oldsock = vq->private_data;
>  	if (sock != oldsock) {
> +		pr_debug("sock=%p != oldsock=%p index=%u fd=%d vq=%p", sock, oldsock, index, fd, vq);
>  		ubufs = vhost_net_ubuf_alloc(vq,
>  					     sock && vhost_sock_zcopy(sock));
>  		if (IS_ERR(ubufs)) {
>  			r = PTR_ERR(ubufs);
> +			pr_debug("ubufs index=%u fd=%d err r=%d vq=%p", index, fd, r, vq);
>  			goto err_ubufs;
>  		}
>  
> @@ -1541,11 +1548,15 @@ static long vhost_net_set_backend(struct vhost_net *n, unsigned index, int fd)
>  		vq->private_data = sock;
>  		vhost_net_buf_unproduce(nvq);
>  		r = vhost_vq_init_access(vq);
> -		if (r)
> +		if (r) {
> +			pr_debug("init_access index=%u fd=%d r=%d vq=%p", index, fd, r, vq);
>  			goto err_used;
> +		}
>  		r = vhost_net_enable_vq(n, vq);
> -		if (r)
> +		if (r) {
> +			pr_debug("enable_vq index=%u fd=%d r=%d vq=%p", index, fd, r, vq);
>  			goto err_used;
> +		}
>  		if (index == VHOST_NET_VQ_RX)
>  			nvq->rx_ring = get_tap_ptr_ring(fd);
>  
> @@ -1559,6 +1570,8 @@ static long vhost_net_set_backend(struct vhost_net *n, unsigned index, int fd)
>  
>  	mutex_unlock(&vq->mutex);
>  
> +	pr_debug("sock=%p", sock);
> +
>  	if (oldubufs) {
>  		vhost_net_ubuf_put_wait_and_free(oldubufs);
>  		mutex_lock(&vq->mutex);
> @@ -1710,6 +1723,7 @@ static long vhost_net_ioctl(struct file *f, unsigned int ioctl,
>  
>  	switch (ioctl) {
>  	case VHOST_NET_SET_BACKEND:
> +		pr_debug("VHOST_NET_SET_BACKEND");
>  		if (copy_from_user(&backend, argp, sizeof backend))
>  			return -EFAULT;
>  		return vhost_net_set_backend(n, backend.index, backend.fd);
> diff --git a/drivers/vhost/vhost.c b/drivers/vhost/vhost.c
> index b5a51b1f2e79..ec25ba32fe81 100644
> --- a/drivers/vhost/vhost.c
> +++ b/drivers/vhost/vhost.c
> @@ -1642,15 +1642,30 @@ long vhost_vring_ioctl(struct vhost_dev *d, unsigned int ioctl, void __user *arg
>  			r = -EINVAL;
>  			break;
>  		}
> +
> +		if (vq->last_avail_idx || vq->avail_idx) {
> +			pr_debug(
> +				"strange VHOST_SET_VRING_BASE [vq=%p][s.index=%u][s.num=%u]",
> +				vq, s.index, s.num);
> +			dump_stack();
> +			r = 0;
> +			break;
> +		}
>  		vq->last_avail_idx = s.num;
>  		/* Forget the cached index value. */
>  		vq->avail_idx = vq->last_avail_idx;
> +		pr_debug(
> +			"VHOST_SET_VRING_BASE [vq=%p][vq->last_avail_idx=%u][vq->avail_idx=%u][s.index=%u][s.num=%u]",
> +			vq, vq->last_avail_idx, vq->avail_idx, s.index, s.num);
>  		break;
>  	case VHOST_GET_VRING_BASE:
>  		s.index = idx;
>  		s.num = vq->last_avail_idx;
>  		if (copy_to_user(argp, &s, sizeof s))
>  			r = -EFAULT;
> +		pr_debug(
> +			"VHOST_GET_VRING_BASE [vq=%p][vq->last_avail_idx=%u][vq->avail_idx=%u][s.index=%u][s.num=%u]",
> +			vq, vq->last_avail_idx, vq->avail_idx, s.index, s.num);
>  		break;
>  	case VHOST_SET_VRING_KICK:
>  		if (copy_from_user(&f, argp, sizeof f)) {
> @@ -2239,8 +2254,8 @@ static int fetch_buf(struct vhost_virtqueue *vq)
>  		vq->avail_idx = vhost16_to_cpu(vq, avail_idx);
>  
>  		if (unlikely((u16)(vq->avail_idx - last_avail_idx) > vq->num)) {
> -			vq_err(vq, "Guest moved used index from %u to %u",
> -				last_avail_idx, vq->avail_idx);
> +			vq_err(vq, "Guest moved vq %p used index from %u to %u",
> +				vq, last_avail_idx, vq->avail_idx);
>  			return -EFAULT;
>  		}
>  
> @@ -2316,6 +2331,9 @@ static int fetch_buf(struct vhost_virtqueue *vq)
>  	BUG_ON(!(vq->used_flags & VRING_USED_F_NO_NOTIFY));
>  
>  	/* On success, increment avail index. */
> +	pr_debug(
> +		"[vq=%p][vq->last_avail_idx=%u][vq->avail_idx=%u][vq->ndescs=%d][vq->first_desc=%d]",
> +		vq, vq->last_avail_idx, vq->avail_idx, vq->ndescs, vq->first_desc);
>  	vq->last_avail_idx++;
>  
>  	return 0;
> @@ -2432,6 +2450,9 @@ EXPORT_SYMBOL_GPL(vhost_get_vq_desc);
>  /* Reverse the effect of vhost_get_vq_desc. Useful for error handling. */
>  void vhost_discard_vq_desc(struct vhost_virtqueue *vq, int n)
>  {
> +	pr_debug(
> +		"DISCARD [vq=%p][vq->last_avail_idx=%u][vq->avail_idx=%u][n=%d]",
> +		vq, vq->last_avail_idx, vq->avail_idx, n);
>  	vq->last_avail_idx -= n;
>  }
>  EXPORT_SYMBOL_GPL(vhost_discard_vq_desc);
> diff --git a/drivers/vhost/vhost.h b/drivers/vhost/vhost.h
> index 661088ae6dc7..08f6d2ccb697 100644
> --- a/drivers/vhost/vhost.h
> +++ b/drivers/vhost/vhost.h
> @@ -250,11 +250,11 @@ int vhost_init_device_iotlb(struct vhost_dev *d, bool enabled);
>  	} while (0)
>  
>  enum {
> -	VHOST_FEATURES = (1ULL << VIRTIO_F_NOTIFY_ON_EMPTY) |
> -			 (1ULL << VIRTIO_RING_F_INDIRECT_DESC) |
> -			 (1ULL << VIRTIO_RING_F_EVENT_IDX) |
> -			 (1ULL << VHOST_F_LOG_ALL) |
> -			 (1ULL << VIRTIO_F_ANY_LAYOUT) |
> +	VHOST_FEATURES = /* (1ULL << VIRTIO_F_NOTIFY_ON_EMPTY) | */
> +			 /* (1ULL << VIRTIO_RING_F_INDIRECT_DESC) | */
> +			 /* (1ULL << VIRTIO_RING_F_EVENT_IDX) | */
> +			 /* (1ULL << VHOST_F_LOG_ALL) | */
> +			 /* (1ULL << VIRTIO_F_ANY_LAYOUT) | */
>  			 (1ULL << VIRTIO_F_VERSION_1)
>  };
>  
> 

Re: vhost changes (batched) in linux-next after 12/13 trigger random crashes in KVM guests after reboot

[Eugenio Perez Martin]

Hi.

Were the vhost and vhost_net modules loaded with dyndbg='+plt'? I miss
all the others regular debug traces on that one.

Thanks!

On Fri, Feb 14, 2020 at 8:34 AM Christian Borntraeger
<borntraeger@de.ibm.com> wrote:
>
> I did
> ping -c 20 -f ... ; reboot
> twice
>
> The ping after the first reboot showed .......E
>
> this was on the host console
>
> [   55.951885] CPU: 34 PID: 1908 Comm: CPU 0/KVM Not tainted 5.5.0+ #21
> [   55.951891] Hardware name: IBM 3906 M04 704 (LPAR)
> [   55.951892] Call Trace:
> [   55.951902]  [<0000001ede114132>] show_stack+0x8a/0xd0
> [   55.951906]  [<0000001edeb0672a>] dump_stack+0x8a/0xb8
> [   55.951915]  [<000003ff803736a6>] vhost_vring_ioctl+0x6fe/0x858 [vhost]
> [   55.951919]  [<000003ff8042a608>] vhost_net_ioctl+0x510/0x570 [vhost_net]
> [   55.951924]  [<0000001ede3c4dd8>] do_vfs_ioctl+0x430/0x6f8
> [   55.951926]  [<0000001ede3c5124>] ksys_ioctl+0x84/0xb0
> [   55.951927]  [<0000001ede3c51ba>] __s390x_sys_ioctl+0x2a/0x38
> [   55.951931]  [<0000001edeb27f72>] system_call+0x2a6/0x2c8
> [   55.951949] CPU: 34 PID: 1908 Comm: CPU 0/KVM Not tainted 5.5.0+ #21
> [   55.951950] Hardware name: IBM 3906 M04 704 (LPAR)
> [   55.951951] Call Trace:
> [   55.951952]  [<0000001ede114132>] show_stack+0x8a/0xd0
> [   55.951954]  [<0000001edeb0672a>] dump_stack+0x8a/0xb8
> [   55.951956]  [<000003ff803736a6>] vhost_vring_ioctl+0x6fe/0x858 [vhost]
> [   55.951958]  [<000003ff8042a608>] vhost_net_ioctl+0x510/0x570 [vhost_net]
> [   55.951959]  [<0000001ede3c4dd8>] do_vfs_ioctl+0x430/0x6f8
> [   55.951961]  [<0000001ede3c5124>] ksys_ioctl+0x84/0xb0
> [   55.951962]  [<0000001ede3c51ba>] __s390x_sys_ioctl+0x2a/0x38
> [   55.951964]  [<0000001edeb27f72>] system_call+0x2a6/0x2c8
> [   55.951997] Guest moved vq 0000000063d896c6 used index from 44 to 0
> [   56.609831] unexpected descriptor format for RX: out 0, in 0
> [   86.540460] CPU: 6 PID: 1908 Comm: CPU 0/KVM Not tainted 5.5.0+ #21
> [   86.540464] Hardware name: IBM 3906 M04 704 (LPAR)
> [   86.540466] Call Trace:
> [   86.540473]  [<0000001ede114132>] show_stack+0x8a/0xd0
> [   86.540477]  [<0000001edeb0672a>] dump_stack+0x8a/0xb8
> [   86.540486]  [<000003ff803736a6>] vhost_vring_ioctl+0x6fe/0x858 [vhost]
> [   86.540490]  [<000003ff8042a608>] vhost_net_ioctl+0x510/0x570 [vhost_net]
> [   86.540494]  [<0000001ede3c4dd8>] do_vfs_ioctl+0x430/0x6f8
> [   86.540496]  [<0000001ede3c5124>] ksys_ioctl+0x84/0xb0
> [   86.540498]  [<0000001ede3c51ba>] __s390x_sys_ioctl+0x2a/0x38
> [   86.540501]  [<0000001edeb27f72>] system_call+0x2a6/0x2c8
> [   86.540524] CPU: 6 PID: 1908 Comm: CPU 0/KVM Not tainted 5.5.0+ #21
> [   86.540525] Hardware name: IBM 3906 M04 704 (LPAR)
> [   86.540526] Call Trace:
> [   86.540527]  [<0000001ede114132>] show_stack+0x8a/0xd0
> [   86.540528]  [<0000001edeb0672a>] dump_stack+0x8a/0xb8
> [   86.540531]  [<000003ff803736a6>] vhost_vring_ioctl+0x6fe/0x858 [vhost]
> [   86.540532]  [<000003ff8042a608>] vhost_net_ioctl+0x510/0x570 [vhost_net]
> [   86.540534]  [<0000001ede3c4dd8>] do_vfs_ioctl+0x430/0x6f8
> [   86.540536]  [<0000001ede3c5124>] ksys_ioctl+0x84/0xb0
> [   86.540537]  [<0000001ede3c51ba>] __s390x_sys_ioctl+0x2a/0x38
> [   86.540538]  [<0000001edeb27f72>] system_call+0x2a6/0x2c8
> [   86.540570] unexpected descriptor format for RX: out 0, in 0
> [   86.540577] Unexpected header len for TX: 0 expected 0
>
>
> On 14.02.20 08:06, Eugenio Pérez wrote:
> > Hi Christian.
> >
> > Sorry, that was meant to be applied over previous debug patch.
> >
> > Here I inline the one meant to be applied over eccb852f1fe6bede630e2e4f1a121a81e34354ab.
> >
> > Thanks!
> >
> > From d978ace99e4844b49b794d768385db3d128a4cc0 Mon Sep 17 00:00:00 2001
> > From: =?UTF-8?q?Eugenio=20P=C3=A9rez?= <eperezma@redhat.com>
> > Date: Fri, 14 Feb 2020 08:02:26 +0100
> > Subject: [PATCH] vhost: disable all features and trace last_avail_idx and
> >  ioctl calls
> >
> > ---
> >  drivers/vhost/net.c   | 20 +++++++++++++++++---
> >  drivers/vhost/vhost.c | 25 +++++++++++++++++++++++--
> >  drivers/vhost/vhost.h | 10 +++++-----
> >  3 files changed, 45 insertions(+), 10 deletions(-)
> >
> > diff --git a/drivers/vhost/net.c b/drivers/vhost/net.c
> > index e158159671fa..e4d5f843f9c0 100644
> > --- a/drivers/vhost/net.c
> > +++ b/drivers/vhost/net.c
> > @@ -1505,10 +1505,13 @@ static long vhost_net_set_backend(struct vhost_net *n, unsigned index, int fd)
> >
> >       mutex_lock(&n->dev.mutex);
> >       r = vhost_dev_check_owner(&n->dev);
> > -     if (r)
> > +     if (r) {
> > +             pr_debug("vhost_dev_check_owner index=%u fd=%d rc r=%d", index, fd, r);
> >               goto err;
> > +     }
> >
> >       if (index >= VHOST_NET_VQ_MAX) {
> > +             pr_debug("vhost_dev_check_owner index=%u fd=%d MAX=%d", index, fd, VHOST_NET_VQ_MAX);
> >               r = -ENOBUFS;
> >               goto err;
> >       }
> > @@ -1518,22 +1521,26 @@ static long vhost_net_set_backend(struct vhost_net *n, unsigned index, int fd)
> >
> >       /* Verify that ring has been setup correctly. */
> >       if (!vhost_vq_access_ok(vq)) {
> > +             pr_debug("vhost_net_set_backend index=%u fd=%d !vhost_vq_access_ok", index, fd);
> >               r = -EFAULT;
> >               goto err_vq;
> >       }
> >       sock = get_socket(fd);
> >       if (IS_ERR(sock)) {
> >               r = PTR_ERR(sock);
> > +             pr_debug("vhost_net_set_backend index=%u fd=%d get_socket err r=%d", index, fd, r);
> >               goto err_vq;
> >       }
> >
> >       /* start polling new socket */
> >       oldsock = vq->private_data;
> >       if (sock != oldsock) {
> > +             pr_debug("sock=%p != oldsock=%p index=%u fd=%d vq=%p", sock, oldsock, index, fd, vq);
> >               ubufs = vhost_net_ubuf_alloc(vq,
> >                                            sock && vhost_sock_zcopy(sock));
> >               if (IS_ERR(ubufs)) {
> >                       r = PTR_ERR(ubufs);
> > +                     pr_debug("ubufs index=%u fd=%d err r=%d vq=%p", index, fd, r, vq);
> >                       goto err_ubufs;
> >               }
> >
> > @@ -1541,11 +1548,15 @@ static long vhost_net_set_backend(struct vhost_net *n, unsigned index, int fd)
> >               vq->private_data = sock;
> >               vhost_net_buf_unproduce(nvq);
> >               r = vhost_vq_init_access(vq);
> > -             if (r)
> > +             if (r) {
> > +                     pr_debug("init_access index=%u fd=%d r=%d vq=%p", index, fd, r, vq);
> >                       goto err_used;
> > +             }
> >               r = vhost_net_enable_vq(n, vq);
> > -             if (r)
> > +             if (r) {
> > +                     pr_debug("enable_vq index=%u fd=%d r=%d vq=%p", index, fd, r, vq);
> >                       goto err_used;
> > +             }
> >               if (index == VHOST_NET_VQ_RX)
> >                       nvq->rx_ring = get_tap_ptr_ring(fd);
> >
> > @@ -1559,6 +1570,8 @@ static long vhost_net_set_backend(struct vhost_net *n, unsigned index, int fd)
> >
> >       mutex_unlock(&vq->mutex);
> >
> > +     pr_debug("sock=%p", sock);
> > +
> >       if (oldubufs) {
> >               vhost_net_ubuf_put_wait_and_free(oldubufs);
> >               mutex_lock(&vq->mutex);
> > @@ -1710,6 +1723,7 @@ static long vhost_net_ioctl(struct file *f, unsigned int ioctl,
> >
> >       switch (ioctl) {
> >       case VHOST_NET_SET_BACKEND:
> > +             pr_debug("VHOST_NET_SET_BACKEND");
> >               if (copy_from_user(&backend, argp, sizeof backend))
> >                       return -EFAULT;
> >               return vhost_net_set_backend(n, backend.index, backend.fd);
> > diff --git a/drivers/vhost/vhost.c b/drivers/vhost/vhost.c
> > index b5a51b1f2e79..ec25ba32fe81 100644
> > --- a/drivers/vhost/vhost.c
> > +++ b/drivers/vhost/vhost.c
> > @@ -1642,15 +1642,30 @@ long vhost_vring_ioctl(struct vhost_dev *d, unsigned int ioctl, void __user *arg
> >                       r = -EINVAL;
> >                       break;
> >               }
> > +
> > +             if (vq->last_avail_idx || vq->avail_idx) {
> > +                     pr_debug(
> > +                             "strange VHOST_SET_VRING_BASE [vq=%p][s.index=%u][s.num=%u]",
> > +                             vq, s.index, s.num);
> > +                     dump_stack();
> > +                     r = 0;
> > +                     break;
> > +             }
> >               vq->last_avail_idx = s.num;
> >               /* Forget the cached index value. */
> >               vq->avail_idx = vq->last_avail_idx;
> > +             pr_debug(
> > +                     "VHOST_SET_VRING_BASE [vq=%p][vq->last_avail_idx=%u][vq->avail_idx=%u][s.index=%u][s.num=%u]",
> > +                     vq, vq->last_avail_idx, vq->avail_idx, s.index, s.num);
> >               break;
> >       case VHOST_GET_VRING_BASE:
> >               s.index = idx;
> >               s.num = vq->last_avail_idx;
> >               if (copy_to_user(argp, &s, sizeof s))
> >                       r = -EFAULT;
> > +             pr_debug(
> > +                     "VHOST_GET_VRING_BASE [vq=%p][vq->last_avail_idx=%u][vq->avail_idx=%u][s.index=%u][s.num=%u]",
> > +                     vq, vq->last_avail_idx, vq->avail_idx, s.index, s.num);
> >               break;
> >       case VHOST_SET_VRING_KICK:
> >               if (copy_from_user(&f, argp, sizeof f)) {
> > @@ -2239,8 +2254,8 @@ static int fetch_buf(struct vhost_virtqueue *vq)
> >               vq->avail_idx = vhost16_to_cpu(vq, avail_idx);
> >
> >               if (unlikely((u16)(vq->avail_idx - last_avail_idx) > vq->num)) {
> > -                     vq_err(vq, "Guest moved used index from %u to %u",
> > -                             last_avail_idx, vq->avail_idx);
> > +                     vq_err(vq, "Guest moved vq %p used index from %u to %u",
> > +                             vq, last_avail_idx, vq->avail_idx);
> >                       return -EFAULT;
> >               }
> >
> > @@ -2316,6 +2331,9 @@ static int fetch_buf(struct vhost_virtqueue *vq)
> >       BUG_ON(!(vq->used_flags & VRING_USED_F_NO_NOTIFY));
> >
> >       /* On success, increment avail index. */
> > +     pr_debug(
> > +             "[vq=%p][vq->last_avail_idx=%u][vq->avail_idx=%u][vq->ndescs=%d][vq->first_desc=%d]",
> > +             vq, vq->last_avail_idx, vq->avail_idx, vq->ndescs, vq->first_desc);
> >       vq->last_avail_idx++;
> >
> >       return 0;
> > @@ -2432,6 +2450,9 @@ EXPORT_SYMBOL_GPL(vhost_get_vq_desc);
> >  /* Reverse the effect of vhost_get_vq_desc. Useful for error handling. */
> >  void vhost_discard_vq_desc(struct vhost_virtqueue *vq, int n)
> >  {
> > +     pr_debug(
> > +             "DISCARD [vq=%p][vq->last_avail_idx=%u][vq->avail_idx=%u][n=%d]",
> > +             vq, vq->last_avail_idx, vq->avail_idx, n);
> >       vq->last_avail_idx -= n;
> >  }
> >  EXPORT_SYMBOL_GPL(vhost_discard_vq_desc);
> > diff --git a/drivers/vhost/vhost.h b/drivers/vhost/vhost.h
> > index 661088ae6dc7..08f6d2ccb697 100644
> > --- a/drivers/vhost/vhost.h
> > +++ b/drivers/vhost/vhost.h
> > @@ -250,11 +250,11 @@ int vhost_init_device_iotlb(struct vhost_dev *d, bool enabled);
> >       } while (0)
> >
> >  enum {
> > -     VHOST_FEATURES = (1ULL << VIRTIO_F_NOTIFY_ON_EMPTY) |
> > -                      (1ULL << VIRTIO_RING_F_INDIRECT_DESC) |
> > -                      (1ULL << VIRTIO_RING_F_EVENT_IDX) |
> > -                      (1ULL << VHOST_F_LOG_ALL) |
> > -                      (1ULL << VIRTIO_F_ANY_LAYOUT) |
> > +     VHOST_FEATURES = /* (1ULL << VIRTIO_F_NOTIFY_ON_EMPTY) | */
> > +                      /* (1ULL << VIRTIO_RING_F_INDIRECT_DESC) | */
> > +                      /* (1ULL << VIRTIO_RING_F_EVENT_IDX) | */
> > +                      /* (1ULL << VHOST_F_LOG_ALL) | */
> > +                      /* (1ULL << VIRTIO_F_ANY_LAYOUT) | */
> >                        (1ULL << VIRTIO_F_VERSION_1)
> >  };
> >
> >
>

Re: vhost changes (batched) in linux-next after 12/13 trigger random crashes in KVM guests after reboot

[Christian Borntraeger]

On 14.02.20 08:40, Eugenio Perez Martin wrote:
> Hi.
> 
> Were the vhost and vhost_net modules loaded with dyndbg='+plt'? I miss
> all the others regular debug traces on that one.

I did 

 echo -n 'file drivers/vhost/vhost.c +plt' > control
and
echo -n 'file drivers/vhost/net.c +plt'  > control

but apparently it did not work...me hates dynamic debug.

> 
> Thanks!
> 
> On Fri, Feb 14, 2020 at 8:34 AM Christian Borntraeger
> <borntraeger@de.ibm.com> wrote:
>>
>> I did
>> ping -c 20 -f ... ; reboot
>> twice
>>
>> The ping after the first reboot showed .......E
>>
>> this was on the host console
>>
>> [   55.951885] CPU: 34 PID: 1908 Comm: CPU 0/KVM Not tainted 5.5.0+ #21
>> [   55.951891] Hardware name: IBM 3906 M04 704 (LPAR)
>> [   55.951892] Call Trace:
>> [   55.951902]  [<0000001ede114132>] show_stack+0x8a/0xd0
>> [   55.951906]  [<0000001edeb0672a>] dump_stack+0x8a/0xb8
>> [   55.951915]  [<000003ff803736a6>] vhost_vring_ioctl+0x6fe/0x858 [vhost]
>> [   55.951919]  [<000003ff8042a608>] vhost_net_ioctl+0x510/0x570 [vhost_net]
>> [   55.951924]  [<0000001ede3c4dd8>] do_vfs_ioctl+0x430/0x6f8
>> [   55.951926]  [<0000001ede3c5124>] ksys_ioctl+0x84/0xb0
>> [   55.951927]  [<0000001ede3c51ba>] __s390x_sys_ioctl+0x2a/0x38
>> [   55.951931]  [<0000001edeb27f72>] system_call+0x2a6/0x2c8
>> [   55.951949] CPU: 34 PID: 1908 Comm: CPU 0/KVM Not tainted 5.5.0+ #21
>> [   55.951950] Hardware name: IBM 3906 M04 704 (LPAR)
>> [   55.951951] Call Trace:
>> [   55.951952]  [<0000001ede114132>] show_stack+0x8a/0xd0
>> [   55.951954]  [<0000001edeb0672a>] dump_stack+0x8a/0xb8
>> [   55.951956]  [<000003ff803736a6>] vhost_vring_ioctl+0x6fe/0x858 [vhost]
>> [   55.951958]  [<000003ff8042a608>] vhost_net_ioctl+0x510/0x570 [vhost_net]
>> [   55.951959]  [<0000001ede3c4dd8>] do_vfs_ioctl+0x430/0x6f8
>> [   55.951961]  [<0000001ede3c5124>] ksys_ioctl+0x84/0xb0
>> [   55.951962]  [<0000001ede3c51ba>] __s390x_sys_ioctl+0x2a/0x38
>> [   55.951964]  [<0000001edeb27f72>] system_call+0x2a6/0x2c8
>> [   55.951997] Guest moved vq 0000000063d896c6 used index from 44 to 0
>> [   56.609831] unexpected descriptor format for RX: out 0, in 0
>> [   86.540460] CPU: 6 PID: 1908 Comm: CPU 0/KVM Not tainted 5.5.0+ #21
>> [   86.540464] Hardware name: IBM 3906 M04 704 (LPAR)
>> [   86.540466] Call Trace:
>> [   86.540473]  [<0000001ede114132>] show_stack+0x8a/0xd0
>> [   86.540477]  [<0000001edeb0672a>] dump_stack+0x8a/0xb8
>> [   86.540486]  [<000003ff803736a6>] vhost_vring_ioctl+0x6fe/0x858 [vhost]
>> [   86.540490]  [<000003ff8042a608>] vhost_net_ioctl+0x510/0x570 [vhost_net]
>> [   86.540494]  [<0000001ede3c4dd8>] do_vfs_ioctl+0x430/0x6f8
>> [   86.540496]  [<0000001ede3c5124>] ksys_ioctl+0x84/0xb0
>> [   86.540498]  [<0000001ede3c51ba>] __s390x_sys_ioctl+0x2a/0x38
>> [   86.540501]  [<0000001edeb27f72>] system_call+0x2a6/0x2c8
>> [   86.540524] CPU: 6 PID: 1908 Comm: CPU 0/KVM Not tainted 5.5.0+ #21
>> [   86.540525] Hardware name: IBM 3906 M04 704 (LPAR)
>> [   86.540526] Call Trace:
>> [   86.540527]  [<0000001ede114132>] show_stack+0x8a/0xd0
>> [   86.540528]  [<0000001edeb0672a>] dump_stack+0x8a/0xb8
>> [   86.540531]  [<000003ff803736a6>] vhost_vring_ioctl+0x6fe/0x858 [vhost]
>> [   86.540532]  [<000003ff8042a608>] vhost_net_ioctl+0x510/0x570 [vhost_net]
>> [   86.540534]  [<0000001ede3c4dd8>] do_vfs_ioctl+0x430/0x6f8
>> [   86.540536]  [<0000001ede3c5124>] ksys_ioctl+0x84/0xb0
>> [   86.540537]  [<0000001ede3c51ba>] __s390x_sys_ioctl+0x2a/0x38
>> [   86.540538]  [<0000001edeb27f72>] system_call+0x2a6/0x2c8
>> [   86.540570] unexpected descriptor format for RX: out 0, in 0
>> [   86.540577] Unexpected header len for TX: 0 expected 0
>>
>>
>> On 14.02.20 08:06, Eugenio Pérez wrote:
>>> Hi Christian.
>>>
>>> Sorry, that was meant to be applied over previous debug patch.
>>>
>>> Here I inline the one meant to be applied over eccb852f1fe6bede630e2e4f1a121a81e34354ab.
>>>
>>> Thanks!
>>>
>>> From d978ace99e4844b49b794d768385db3d128a4cc0 Mon Sep 17 00:00:00 2001
>>> From: =?UTF-8?q?Eugenio=20P=C3=A9rez?= <eperezma@redhat.com>
>>> Date: Fri, 14 Feb 2020 08:02:26 +0100
>>> Subject: [PATCH] vhost: disable all features and trace last_avail_idx and
>>>  ioctl calls
>>>
>>> ---
>>>  drivers/vhost/net.c   | 20 +++++++++++++++++---
>>>  drivers/vhost/vhost.c | 25 +++++++++++++++++++++++--
>>>  drivers/vhost/vhost.h | 10 +++++-----
>>>  3 files changed, 45 insertions(+), 10 deletions(-)
>>>
>>> diff --git a/drivers/vhost/net.c b/drivers/vhost/net.c
>>> index e158159671fa..e4d5f843f9c0 100644
>>> --- a/drivers/vhost/net.c
>>> +++ b/drivers/vhost/net.c
>>> @@ -1505,10 +1505,13 @@ static long vhost_net_set_backend(struct vhost_net *n, unsigned index, int fd)
>>>
>>>       mutex_lock(&n->dev.mutex);
>>>       r = vhost_dev_check_owner(&n->dev);
>>> -     if (r)
>>> +     if (r) {
>>> +             pr_debug("vhost_dev_check_owner index=%u fd=%d rc r=%d", index, fd, r);
>>>               goto err;
>>> +     }
>>>
>>>       if (index >= VHOST_NET_VQ_MAX) {
>>> +             pr_debug("vhost_dev_check_owner index=%u fd=%d MAX=%d", index, fd, VHOST_NET_VQ_MAX);
>>>               r = -ENOBUFS;
>>>               goto err;
>>>       }
>>> @@ -1518,22 +1521,26 @@ static long vhost_net_set_backend(struct vhost_net *n, unsigned index, int fd)
>>>
>>>       /* Verify that ring has been setup correctly. */
>>>       if (!vhost_vq_access_ok(vq)) {
>>> +             pr_debug("vhost_net_set_backend index=%u fd=%d !vhost_vq_access_ok", index, fd);
>>>               r = -EFAULT;
>>>               goto err_vq;
>>>       }
>>>       sock = get_socket(fd);
>>>       if (IS_ERR(sock)) {
>>>               r = PTR_ERR(sock);
>>> +             pr_debug("vhost_net_set_backend index=%u fd=%d get_socket err r=%d", index, fd, r);
>>>               goto err_vq;
>>>       }
>>>
>>>       /* start polling new socket */
>>>       oldsock = vq->private_data;
>>>       if (sock != oldsock) {
>>> +             pr_debug("sock=%p != oldsock=%p index=%u fd=%d vq=%p", sock, oldsock, index, fd, vq);
>>>               ubufs = vhost_net_ubuf_alloc(vq,
>>>                                            sock && vhost_sock_zcopy(sock));
>>>               if (IS_ERR(ubufs)) {
>>>                       r = PTR_ERR(ubufs);
>>> +                     pr_debug("ubufs index=%u fd=%d err r=%d vq=%p", index, fd, r, vq);
>>>                       goto err_ubufs;
>>>               }
>>>
>>> @@ -1541,11 +1548,15 @@ static long vhost_net_set_backend(struct vhost_net *n, unsigned index, int fd)
>>>               vq->private_data = sock;
>>>               vhost_net_buf_unproduce(nvq);
>>>               r = vhost_vq_init_access(vq);
>>> -             if (r)
>>> +             if (r) {
>>> +                     pr_debug("init_access index=%u fd=%d r=%d vq=%p", index, fd, r, vq);
>>>                       goto err_used;
>>> +             }
>>>               r = vhost_net_enable_vq(n, vq);
>>> -             if (r)
>>> +             if (r) {
>>> +                     pr_debug("enable_vq index=%u fd=%d r=%d vq=%p", index, fd, r, vq);
>>>                       goto err_used;
>>> +             }
>>>               if (index == VHOST_NET_VQ_RX)
>>>                       nvq->rx_ring = get_tap_ptr_ring(fd);
>>>
>>> @@ -1559,6 +1570,8 @@ static long vhost_net_set_backend(struct vhost_net *n, unsigned index, int fd)
>>>
>>>       mutex_unlock(&vq->mutex);
>>>
>>> +     pr_debug("sock=%p", sock);
>>> +
>>>       if (oldubufs) {
>>>               vhost_net_ubuf_put_wait_and_free(oldubufs);
>>>               mutex_lock(&vq->mutex);
>>> @@ -1710,6 +1723,7 @@ static long vhost_net_ioctl(struct file *f, unsigned int ioctl,
>>>
>>>       switch (ioctl) {
>>>       case VHOST_NET_SET_BACKEND:
>>> +             pr_debug("VHOST_NET_SET_BACKEND");
>>>               if (copy_from_user(&backend, argp, sizeof backend))
>>>                       return -EFAULT;
>>>               return vhost_net_set_backend(n, backend.index, backend.fd);
>>> diff --git a/drivers/vhost/vhost.c b/drivers/vhost/vhost.c
>>> index b5a51b1f2e79..ec25ba32fe81 100644
>>> --- a/drivers/vhost/vhost.c
>>> +++ b/drivers/vhost/vhost.c
>>> @@ -1642,15 +1642,30 @@ long vhost_vring_ioctl(struct vhost_dev *d, unsigned int ioctl, void __user *arg
>>>                       r = -EINVAL;
>>>                       break;
>>>               }
>>> +
>>> +             if (vq->last_avail_idx || vq->avail_idx) {
>>> +                     pr_debug(
>>> +                             "strange VHOST_SET_VRING_BASE [vq=%p][s.index=%u][s.num=%u]",
>>> +                             vq, s.index, s.num);
>>> +                     dump_stack();
>>> +                     r = 0;
>>> +                     break;
>>> +             }
>>>               vq->last_avail_idx = s.num;
>>>               /* Forget the cached index value. */
>>>               vq->avail_idx = vq->last_avail_idx;
>>> +             pr_debug(
>>> +                     "VHOST_SET_VRING_BASE [vq=%p][vq->last_avail_idx=%u][vq->avail_idx=%u][s.index=%u][s.num=%u]",
>>> +                     vq, vq->last_avail_idx, vq->avail_idx, s.index, s.num);
>>>               break;
>>>       case VHOST_GET_VRING_BASE:
>>>               s.index = idx;
>>>               s.num = vq->last_avail_idx;
>>>               if (copy_to_user(argp, &s, sizeof s))
>>>                       r = -EFAULT;
>>> +             pr_debug(
>>> +                     "VHOST_GET_VRING_BASE [vq=%p][vq->last_avail_idx=%u][vq->avail_idx=%u][s.index=%u][s.num=%u]",
>>> +                     vq, vq->last_avail_idx, vq->avail_idx, s.index, s.num);
>>>               break;
>>>       case VHOST_SET_VRING_KICK:
>>>               if (copy_from_user(&f, argp, sizeof f)) {
>>> @@ -2239,8 +2254,8 @@ static int fetch_buf(struct vhost_virtqueue *vq)
>>>               vq->avail_idx = vhost16_to_cpu(vq, avail_idx);
>>>
>>>               if (unlikely((u16)(vq->avail_idx - last_avail_idx) > vq->num)) {
>>> -                     vq_err(vq, "Guest moved used index from %u to %u",
>>> -                             last_avail_idx, vq->avail_idx);
>>> +                     vq_err(vq, "Guest moved vq %p used index from %u to %u",
>>> +                             vq, last_avail_idx, vq->avail_idx);
>>>                       return -EFAULT;
>>>               }
>>>
>>> @@ -2316,6 +2331,9 @@ static int fetch_buf(struct vhost_virtqueue *vq)
>>>       BUG_ON(!(vq->used_flags & VRING_USED_F_NO_NOTIFY));
>>>
>>>       /* On success, increment avail index. */
>>> +     pr_debug(
>>> +             "[vq=%p][vq->last_avail_idx=%u][vq->avail_idx=%u][vq->ndescs=%d][vq->first_desc=%d]",
>>> +             vq, vq->last_avail_idx, vq->avail_idx, vq->ndescs, vq->first_desc);
>>>       vq->last_avail_idx++;
>>>
>>>       return 0;
>>> @@ -2432,6 +2450,9 @@ EXPORT_SYMBOL_GPL(vhost_get_vq_desc);
>>>  /* Reverse the effect of vhost_get_vq_desc. Useful for error handling. */
>>>  void vhost_discard_vq_desc(struct vhost_virtqueue *vq, int n)
>>>  {
>>> +     pr_debug(
>>> +             "DISCARD [vq=%p][vq->last_avail_idx=%u][vq->avail_idx=%u][n=%d]",
>>> +             vq, vq->last_avail_idx, vq->avail_idx, n);
>>>       vq->last_avail_idx -= n;
>>>  }
>>>  EXPORT_SYMBOL_GPL(vhost_discard_vq_desc);
>>> diff --git a/drivers/vhost/vhost.h b/drivers/vhost/vhost.h
>>> index 661088ae6dc7..08f6d2ccb697 100644
>>> --- a/drivers/vhost/vhost.h
>>> +++ b/drivers/vhost/vhost.h
>>> @@ -250,11 +250,11 @@ int vhost_init_device_iotlb(struct vhost_dev *d, bool enabled);
>>>       } while (0)
>>>
>>>  enum {
>>> -     VHOST_FEATURES = (1ULL << VIRTIO_F_NOTIFY_ON_EMPTY) |
>>> -                      (1ULL << VIRTIO_RING_F_INDIRECT_DESC) |
>>> -                      (1ULL << VIRTIO_RING_F_EVENT_IDX) |
>>> -                      (1ULL << VHOST_F_LOG_ALL) |
>>> -                      (1ULL << VIRTIO_F_ANY_LAYOUT) |
>>> +     VHOST_FEATURES = /* (1ULL << VIRTIO_F_NOTIFY_ON_EMPTY) | */
>>> +                      /* (1ULL << VIRTIO_RING_F_INDIRECT_DESC) | */
>>> +                      /* (1ULL << VIRTIO_RING_F_EVENT_IDX) | */
>>> +                      /* (1ULL << VHOST_F_LOG_ALL) | */
>>> +                      /* (1ULL << VIRTIO_F_ANY_LAYOUT) | */
>>>                        (1ULL << VIRTIO_F_VERSION_1)
>>>  };
>>>
>>>
>>
> 

Re: vhost changes (batched) in linux-next after 12/13 trigger random crashes in KVM guests after reboot

[Christian Borntraeger]

[-- Attachment #1: Type: text/plain, Size: 12556 bytes --]

repro


On 14.02.20 08:43, Christian Borntraeger wrote:
> 
> 
> On 14.02.20 08:40, Eugenio Perez Martin wrote:
>> Hi.
>>
>> Were the vhost and vhost_net modules loaded with dyndbg='+plt'? I miss
>> all the others regular debug traces on that one.
> 
> I did 
> 
>  echo -n 'file drivers/vhost/vhost.c +plt' > control
> and
> echo -n 'file drivers/vhost/net.c +plt'  > control
> 
> but apparently it did not work...me hates dynamic debug.
> 
>>
>> Thanks!
>>
>> On Fri, Feb 14, 2020 at 8:34 AM Christian Borntraeger
>> <borntraeger@de.ibm.com> wrote:
>>>
>>> I did
>>> ping -c 20 -f ... ; reboot
>>> twice
>>>
>>> The ping after the first reboot showed .......E
>>>
>>> this was on the host console
>>>
>>> [   55.951885] CPU: 34 PID: 1908 Comm: CPU 0/KVM Not tainted 5.5.0+ #21
>>> [   55.951891] Hardware name: IBM 3906 M04 704 (LPAR)
>>> [   55.951892] Call Trace:
>>> [   55.951902]  [<0000001ede114132>] show_stack+0x8a/0xd0
>>> [   55.951906]  [<0000001edeb0672a>] dump_stack+0x8a/0xb8
>>> [   55.951915]  [<000003ff803736a6>] vhost_vring_ioctl+0x6fe/0x858 [vhost]
>>> [   55.951919]  [<000003ff8042a608>] vhost_net_ioctl+0x510/0x570 [vhost_net]
>>> [   55.951924]  [<0000001ede3c4dd8>] do_vfs_ioctl+0x430/0x6f8
>>> [   55.951926]  [<0000001ede3c5124>] ksys_ioctl+0x84/0xb0
>>> [   55.951927]  [<0000001ede3c51ba>] __s390x_sys_ioctl+0x2a/0x38
>>> [   55.951931]  [<0000001edeb27f72>] system_call+0x2a6/0x2c8
>>> [   55.951949] CPU: 34 PID: 1908 Comm: CPU 0/KVM Not tainted 5.5.0+ #21
>>> [   55.951950] Hardware name: IBM 3906 M04 704 (LPAR)
>>> [   55.951951] Call Trace:
>>> [   55.951952]  [<0000001ede114132>] show_stack+0x8a/0xd0
>>> [   55.951954]  [<0000001edeb0672a>] dump_stack+0x8a/0xb8
>>> [   55.951956]  [<000003ff803736a6>] vhost_vring_ioctl+0x6fe/0x858 [vhost]
>>> [   55.951958]  [<000003ff8042a608>] vhost_net_ioctl+0x510/0x570 [vhost_net]
>>> [   55.951959]  [<0000001ede3c4dd8>] do_vfs_ioctl+0x430/0x6f8
>>> [   55.951961]  [<0000001ede3c5124>] ksys_ioctl+0x84/0xb0
>>> [   55.951962]  [<0000001ede3c51ba>] __s390x_sys_ioctl+0x2a/0x38
>>> [   55.951964]  [<0000001edeb27f72>] system_call+0x2a6/0x2c8
>>> [   55.951997] Guest moved vq 0000000063d896c6 used index from 44 to 0
>>> [   56.609831] unexpected descriptor format for RX: out 0, in 0
>>> [   86.540460] CPU: 6 PID: 1908 Comm: CPU 0/KVM Not tainted 5.5.0+ #21
>>> [   86.540464] Hardware name: IBM 3906 M04 704 (LPAR)
>>> [   86.540466] Call Trace:
>>> [   86.540473]  [<0000001ede114132>] show_stack+0x8a/0xd0
>>> [   86.540477]  [<0000001edeb0672a>] dump_stack+0x8a/0xb8
>>> [   86.540486]  [<000003ff803736a6>] vhost_vring_ioctl+0x6fe/0x858 [vhost]
>>> [   86.540490]  [<000003ff8042a608>] vhost_net_ioctl+0x510/0x570 [vhost_net]
>>> [   86.540494]  [<0000001ede3c4dd8>] do_vfs_ioctl+0x430/0x6f8
>>> [   86.540496]  [<0000001ede3c5124>] ksys_ioctl+0x84/0xb0
>>> [   86.540498]  [<0000001ede3c51ba>] __s390x_sys_ioctl+0x2a/0x38
>>> [   86.540501]  [<0000001edeb27f72>] system_call+0x2a6/0x2c8
>>> [   86.540524] CPU: 6 PID: 1908 Comm: CPU 0/KVM Not tainted 5.5.0+ #21
>>> [   86.540525] Hardware name: IBM 3906 M04 704 (LPAR)
>>> [   86.540526] Call Trace:
>>> [   86.540527]  [<0000001ede114132>] show_stack+0x8a/0xd0
>>> [   86.540528]  [<0000001edeb0672a>] dump_stack+0x8a/0xb8
>>> [   86.540531]  [<000003ff803736a6>] vhost_vring_ioctl+0x6fe/0x858 [vhost]
>>> [   86.540532]  [<000003ff8042a608>] vhost_net_ioctl+0x510/0x570 [vhost_net]
>>> [   86.540534]  [<0000001ede3c4dd8>] do_vfs_ioctl+0x430/0x6f8
>>> [   86.540536]  [<0000001ede3c5124>] ksys_ioctl+0x84/0xb0
>>> [   86.540537]  [<0000001ede3c51ba>] __s390x_sys_ioctl+0x2a/0x38
>>> [   86.540538]  [<0000001edeb27f72>] system_call+0x2a6/0x2c8
>>> [   86.540570] unexpected descriptor format for RX: out 0, in 0
>>> [   86.540577] Unexpected header len for TX: 0 expected 0
>>>
>>>
>>> On 14.02.20 08:06, Eugenio Pérez wrote:
>>>> Hi Christian.
>>>>
>>>> Sorry, that was meant to be applied over previous debug patch.
>>>>
>>>> Here I inline the one meant to be applied over eccb852f1fe6bede630e2e4f1a121a81e34354ab.
>>>>
>>>> Thanks!
>>>>
>>>> From d978ace99e4844b49b794d768385db3d128a4cc0 Mon Sep 17 00:00:00 2001
>>>> From: =?UTF-8?q?Eugenio=20P=C3=A9rez?= <eperezma@redhat.com>
>>>> Date: Fri, 14 Feb 2020 08:02:26 +0100
>>>> Subject: [PATCH] vhost: disable all features and trace last_avail_idx and
>>>>  ioctl calls
>>>>
>>>> ---
>>>>  drivers/vhost/net.c   | 20 +++++++++++++++++---
>>>>  drivers/vhost/vhost.c | 25 +++++++++++++++++++++++--
>>>>  drivers/vhost/vhost.h | 10 +++++-----
>>>>  3 files changed, 45 insertions(+), 10 deletions(-)
>>>>
>>>> diff --git a/drivers/vhost/net.c b/drivers/vhost/net.c
>>>> index e158159671fa..e4d5f843f9c0 100644
>>>> --- a/drivers/vhost/net.c
>>>> +++ b/drivers/vhost/net.c
>>>> @@ -1505,10 +1505,13 @@ static long vhost_net_set_backend(struct vhost_net *n, unsigned index, int fd)
>>>>
>>>>       mutex_lock(&n->dev.mutex);
>>>>       r = vhost_dev_check_owner(&n->dev);
>>>> -     if (r)
>>>> +     if (r) {
>>>> +             pr_debug("vhost_dev_check_owner index=%u fd=%d rc r=%d", index, fd, r);
>>>>               goto err;
>>>> +     }
>>>>
>>>>       if (index >= VHOST_NET_VQ_MAX) {
>>>> +             pr_debug("vhost_dev_check_owner index=%u fd=%d MAX=%d", index, fd, VHOST_NET_VQ_MAX);
>>>>               r = -ENOBUFS;
>>>>               goto err;
>>>>       }
>>>> @@ -1518,22 +1521,26 @@ static long vhost_net_set_backend(struct vhost_net *n, unsigned index, int fd)
>>>>
>>>>       /* Verify that ring has been setup correctly. */
>>>>       if (!vhost_vq_access_ok(vq)) {
>>>> +             pr_debug("vhost_net_set_backend index=%u fd=%d !vhost_vq_access_ok", index, fd);
>>>>               r = -EFAULT;
>>>>               goto err_vq;
>>>>       }
>>>>       sock = get_socket(fd);
>>>>       if (IS_ERR(sock)) {
>>>>               r = PTR_ERR(sock);
>>>> +             pr_debug("vhost_net_set_backend index=%u fd=%d get_socket err r=%d", index, fd, r);
>>>>               goto err_vq;
>>>>       }
>>>>
>>>>       /* start polling new socket */
>>>>       oldsock = vq->private_data;
>>>>       if (sock != oldsock) {
>>>> +             pr_debug("sock=%p != oldsock=%p index=%u fd=%d vq=%p", sock, oldsock, index, fd, vq);
>>>>               ubufs = vhost_net_ubuf_alloc(vq,
>>>>                                            sock && vhost_sock_zcopy(sock));
>>>>               if (IS_ERR(ubufs)) {
>>>>                       r = PTR_ERR(ubufs);
>>>> +                     pr_debug("ubufs index=%u fd=%d err r=%d vq=%p", index, fd, r, vq);
>>>>                       goto err_ubufs;
>>>>               }
>>>>
>>>> @@ -1541,11 +1548,15 @@ static long vhost_net_set_backend(struct vhost_net *n, unsigned index, int fd)
>>>>               vq->private_data = sock;
>>>>               vhost_net_buf_unproduce(nvq);
>>>>               r = vhost_vq_init_access(vq);
>>>> -             if (r)
>>>> +             if (r) {
>>>> +                     pr_debug("init_access index=%u fd=%d r=%d vq=%p", index, fd, r, vq);
>>>>                       goto err_used;
>>>> +             }
>>>>               r = vhost_net_enable_vq(n, vq);
>>>> -             if (r)
>>>> +             if (r) {
>>>> +                     pr_debug("enable_vq index=%u fd=%d r=%d vq=%p", index, fd, r, vq);
>>>>                       goto err_used;
>>>> +             }
>>>>               if (index == VHOST_NET_VQ_RX)
>>>>                       nvq->rx_ring = get_tap_ptr_ring(fd);
>>>>
>>>> @@ -1559,6 +1570,8 @@ static long vhost_net_set_backend(struct vhost_net *n, unsigned index, int fd)
>>>>
>>>>       mutex_unlock(&vq->mutex);
>>>>
>>>> +     pr_debug("sock=%p", sock);
>>>> +
>>>>       if (oldubufs) {
>>>>               vhost_net_ubuf_put_wait_and_free(oldubufs);
>>>>               mutex_lock(&vq->mutex);
>>>> @@ -1710,6 +1723,7 @@ static long vhost_net_ioctl(struct file *f, unsigned int ioctl,
>>>>
>>>>       switch (ioctl) {
>>>>       case VHOST_NET_SET_BACKEND:
>>>> +             pr_debug("VHOST_NET_SET_BACKEND");
>>>>               if (copy_from_user(&backend, argp, sizeof backend))
>>>>                       return -EFAULT;
>>>>               return vhost_net_set_backend(n, backend.index, backend.fd);
>>>> diff --git a/drivers/vhost/vhost.c b/drivers/vhost/vhost.c
>>>> index b5a51b1f2e79..ec25ba32fe81 100644
>>>> --- a/drivers/vhost/vhost.c
>>>> +++ b/drivers/vhost/vhost.c
>>>> @@ -1642,15 +1642,30 @@ long vhost_vring_ioctl(struct vhost_dev *d, unsigned int ioctl, void __user *arg
>>>>                       r = -EINVAL;
>>>>                       break;
>>>>               }
>>>> +
>>>> +             if (vq->last_avail_idx || vq->avail_idx) {
>>>> +                     pr_debug(
>>>> +                             "strange VHOST_SET_VRING_BASE [vq=%p][s.index=%u][s.num=%u]",
>>>> +                             vq, s.index, s.num);
>>>> +                     dump_stack();
>>>> +                     r = 0;
>>>> +                     break;
>>>> +             }
>>>>               vq->last_avail_idx = s.num;
>>>>               /* Forget the cached index value. */
>>>>               vq->avail_idx = vq->last_avail_idx;
>>>> +             pr_debug(
>>>> +                     "VHOST_SET_VRING_BASE [vq=%p][vq->last_avail_idx=%u][vq->avail_idx=%u][s.index=%u][s.num=%u]",
>>>> +                     vq, vq->last_avail_idx, vq->avail_idx, s.index, s.num);
>>>>               break;
>>>>       case VHOST_GET_VRING_BASE:
>>>>               s.index = idx;
>>>>               s.num = vq->last_avail_idx;
>>>>               if (copy_to_user(argp, &s, sizeof s))
>>>>                       r = -EFAULT;
>>>> +             pr_debug(
>>>> +                     "VHOST_GET_VRING_BASE [vq=%p][vq->last_avail_idx=%u][vq->avail_idx=%u][s.index=%u][s.num=%u]",
>>>> +                     vq, vq->last_avail_idx, vq->avail_idx, s.index, s.num);
>>>>               break;
>>>>       case VHOST_SET_VRING_KICK:
>>>>               if (copy_from_user(&f, argp, sizeof f)) {
>>>> @@ -2239,8 +2254,8 @@ static int fetch_buf(struct vhost_virtqueue *vq)
>>>>               vq->avail_idx = vhost16_to_cpu(vq, avail_idx);
>>>>
>>>>               if (unlikely((u16)(vq->avail_idx - last_avail_idx) > vq->num)) {
>>>> -                     vq_err(vq, "Guest moved used index from %u to %u",
>>>> -                             last_avail_idx, vq->avail_idx);
>>>> +                     vq_err(vq, "Guest moved vq %p used index from %u to %u",
>>>> +                             vq, last_avail_idx, vq->avail_idx);
>>>>                       return -EFAULT;
>>>>               }
>>>>
>>>> @@ -2316,6 +2331,9 @@ static int fetch_buf(struct vhost_virtqueue *vq)
>>>>       BUG_ON(!(vq->used_flags & VRING_USED_F_NO_NOTIFY));
>>>>
>>>>       /* On success, increment avail index. */
>>>> +     pr_debug(
>>>> +             "[vq=%p][vq->last_avail_idx=%u][vq->avail_idx=%u][vq->ndescs=%d][vq->first_desc=%d]",
>>>> +             vq, vq->last_avail_idx, vq->avail_idx, vq->ndescs, vq->first_desc);
>>>>       vq->last_avail_idx++;
>>>>
>>>>       return 0;
>>>> @@ -2432,6 +2450,9 @@ EXPORT_SYMBOL_GPL(vhost_get_vq_desc);
>>>>  /* Reverse the effect of vhost_get_vq_desc. Useful for error handling. */
>>>>  void vhost_discard_vq_desc(struct vhost_virtqueue *vq, int n)
>>>>  {
>>>> +     pr_debug(
>>>> +             "DISCARD [vq=%p][vq->last_avail_idx=%u][vq->avail_idx=%u][n=%d]",
>>>> +             vq, vq->last_avail_idx, vq->avail_idx, n);
>>>>       vq->last_avail_idx -= n;
>>>>  }
>>>>  EXPORT_SYMBOL_GPL(vhost_discard_vq_desc);
>>>> diff --git a/drivers/vhost/vhost.h b/drivers/vhost/vhost.h
>>>> index 661088ae6dc7..08f6d2ccb697 100644
>>>> --- a/drivers/vhost/vhost.h
>>>> +++ b/drivers/vhost/vhost.h
>>>> @@ -250,11 +250,11 @@ int vhost_init_device_iotlb(struct vhost_dev *d, bool enabled);
>>>>       } while (0)
>>>>
>>>>  enum {
>>>> -     VHOST_FEATURES = (1ULL << VIRTIO_F_NOTIFY_ON_EMPTY) |
>>>> -                      (1ULL << VIRTIO_RING_F_INDIRECT_DESC) |
>>>> -                      (1ULL << VIRTIO_RING_F_EVENT_IDX) |
>>>> -                      (1ULL << VHOST_F_LOG_ALL) |
>>>> -                      (1ULL << VIRTIO_F_ANY_LAYOUT) |
>>>> +     VHOST_FEATURES = /* (1ULL << VIRTIO_F_NOTIFY_ON_EMPTY) | */
>>>> +                      /* (1ULL << VIRTIO_RING_F_INDIRECT_DESC) | */
>>>> +                      /* (1ULL << VIRTIO_RING_F_EVENT_IDX) | */
>>>> +                      /* (1ULL << VHOST_F_LOG_ALL) | */
>>>> +                      /* (1ULL << VIRTIO_F_ANY_LAYOUT) | */
>>>>                        (1ULL << VIRTIO_F_VERSION_1)
>>>>  };
>>>>
>>>>
>>>
>>

[-- Attachment #2: log --]
[-- Type: text/plain, Size: 33477 bytes --]

[  924.337135] [1901] 1666: VHOST_GET_VRING_BASE [vq=0000000063d896c6][vq->last_avail_idx=109][vq->avail_idx=0][s.index=1][s.num=109]
[  937.802750] [2237] 1657: VHOST_SET_VRING_BASE [vq=00000000742665e9][vq->last_avail_idx=0][vq->avail_idx=0][s.index=0][s.num=0]
[  937.802757] [2237] 1657: VHOST_SET_VRING_BASE [vq=000000007d36f807][vq->last_avail_idx=0][vq->avail_idx=0][s.index=1][s.num=0]
[  937.802763] [2237] 1726: VHOST_NET_SET_BACKEND
[  937.802766] [2237] 1538: sock=00000000de4d87a3 != oldsock=0000000012e3bc88 index=0 fd=39 vq=00000000742665e9
[  937.802770] [2237] 1573: sock=00000000de4d87a3
[  937.802771] [2237] 1726: VHOST_NET_SET_BACKEND
[  937.802772] [2237] 1538: sock=00000000de4d87a3 != oldsock=0000000012e3bc88 index=1 fd=39 vq=000000007d36f807
[  937.802773] [2237] 1573: sock=00000000de4d87a3
[  937.802775] [2235] 2453: DISCARD [vq=00000000742665e9][vq->last_avail_idx=0][vq->avail_idx=0][n=0]
[  938.475541] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=0][vq->avail_idx=256][vq->ndescs=1][vq->first_desc=0]
[  938.475543] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=1][vq->avail_idx=256][vq->ndescs=2][vq->first_desc=0]
[  938.475544] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=2][vq->avail_idx=256][vq->ndescs=3][vq->first_desc=0]
[  938.475545] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=3][vq->avail_idx=256][vq->ndescs=4][vq->first_desc=0]
[  938.475546] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=4][vq->avail_idx=256][vq->ndescs=5][vq->first_desc=0]
[  938.475547] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=5][vq->avail_idx=256][vq->ndescs=6][vq->first_desc=0]
[  938.475548] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=6][vq->avail_idx=256][vq->ndescs=7][vq->first_desc=0]
[  938.475549] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=7][vq->avail_idx=256][vq->ndescs=8][vq->first_desc=0]
[  938.475550] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=8][vq->avail_idx=256][vq->ndescs=9][vq->first_desc=0]
[  938.475551] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=9][vq->avail_idx=256][vq->ndescs=10][vq->first_desc=0]
[  938.475552] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=10][vq->avail_idx=256][vq->ndescs=11][vq->first_desc=0]
[  938.475553] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=11][vq->avail_idx=256][vq->ndescs=12][vq->first_desc=0]
[  938.475554] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=12][vq->avail_idx=256][vq->ndescs=13][vq->first_desc=0]
[  938.475555] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=13][vq->avail_idx=256][vq->ndescs=14][vq->first_desc=0]
[  938.475556] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=14][vq->avail_idx=256][vq->ndescs=15][vq->first_desc=0]
[  938.475557] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=15][vq->avail_idx=256][vq->ndescs=16][vq->first_desc=0]
[  938.475558] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=16][vq->avail_idx=256][vq->ndescs=17][vq->first_desc=0]
[  938.475559] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=17][vq->avail_idx=256][vq->ndescs=18][vq->first_desc=0]
[  938.475560] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=18][vq->avail_idx=256][vq->ndescs=19][vq->first_desc=0]
[  938.475561] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=19][vq->avail_idx=256][vq->ndescs=20][vq->first_desc=0]
[  938.475562] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=20][vq->avail_idx=256][vq->ndescs=21][vq->first_desc=0]
[  938.475563] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=21][vq->avail_idx=256][vq->ndescs=22][vq->first_desc=0]
[  938.475564] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=22][vq->avail_idx=256][vq->ndescs=23][vq->first_desc=0]
[  938.475564] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=23][vq->avail_idx=256][vq->ndescs=24][vq->first_desc=0]
[  938.475565] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=24][vq->avail_idx=256][vq->ndescs=25][vq->first_desc=0]
[  938.475566] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=25][vq->avail_idx=256][vq->ndescs=26][vq->first_desc=0]
[  938.475567] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=26][vq->avail_idx=256][vq->ndescs=27][vq->first_desc=0]
[  938.475568] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=27][vq->avail_idx=256][vq->ndescs=28][vq->first_desc=0]
[  938.475569] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=28][vq->avail_idx=256][vq->ndescs=29][vq->first_desc=0]
[  938.475570] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=29][vq->avail_idx=256][vq->ndescs=30][vq->first_desc=0]
[  938.475571] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=30][vq->avail_idx=256][vq->ndescs=31][vq->first_desc=0]
[  938.475572] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=31][vq->avail_idx=256][vq->ndescs=32][vq->first_desc=0]
[  938.475573] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=32][vq->avail_idx=256][vq->ndescs=33][vq->first_desc=0]
[  938.475574] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=33][vq->avail_idx=256][vq->ndescs=34][vq->first_desc=0]
[  938.475575] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=34][vq->avail_idx=256][vq->ndescs=35][vq->first_desc=0]
[  938.475576] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=35][vq->avail_idx=256][vq->ndescs=36][vq->first_desc=0]
[  938.475577] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=36][vq->avail_idx=256][vq->ndescs=37][vq->first_desc=0]
[  938.475577] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=37][vq->avail_idx=256][vq->ndescs=38][vq->first_desc=0]
[  938.475578] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=38][vq->avail_idx=256][vq->ndescs=39][vq->first_desc=0]
[  938.475579] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=39][vq->avail_idx=256][vq->ndescs=40][vq->first_desc=0]
[  938.475580] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=40][vq->avail_idx=256][vq->ndescs=41][vq->first_desc=0]
[  938.475581] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=41][vq->avail_idx=256][vq->ndescs=42][vq->first_desc=0]
[  938.475582] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=42][vq->avail_idx=256][vq->ndescs=43][vq->first_desc=0]
[  938.475583] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=43][vq->avail_idx=256][vq->ndescs=44][vq->first_desc=0]
[  938.475584] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=44][vq->avail_idx=256][vq->ndescs=45][vq->first_desc=0]
[  938.475585] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=45][vq->avail_idx=256][vq->ndescs=46][vq->first_desc=0]
[  938.475586] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=46][vq->avail_idx=256][vq->ndescs=47][vq->first_desc=0]
[  938.475587] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=47][vq->avail_idx=256][vq->ndescs=48][vq->first_desc=0]
[  938.475588] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=48][vq->avail_idx=256][vq->ndescs=49][vq->first_desc=0]
[  938.475589] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=49][vq->avail_idx=256][vq->ndescs=50][vq->first_desc=0]
[  938.475589] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=50][vq->avail_idx=256][vq->ndescs=51][vq->first_desc=0]
[  938.475590] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=51][vq->avail_idx=256][vq->ndescs=52][vq->first_desc=0]
[  938.475591] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=52][vq->avail_idx=256][vq->ndescs=53][vq->first_desc=0]
[  938.475592] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=53][vq->avail_idx=256][vq->ndescs=54][vq->first_desc=0]
[  938.475593] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=54][vq->avail_idx=256][vq->ndescs=55][vq->first_desc=0]
[  938.475594] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=55][vq->avail_idx=256][vq->ndescs=56][vq->first_desc=0]
[  938.475595] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=56][vq->avail_idx=256][vq->ndescs=57][vq->first_desc=0]
[  938.475596] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=57][vq->avail_idx=256][vq->ndescs=58][vq->first_desc=0]
[  938.475597] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=58][vq->avail_idx=256][vq->ndescs=59][vq->first_desc=0]
[  938.475598] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=59][vq->avail_idx=256][vq->ndescs=60][vq->first_desc=0]
[  938.475599] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=60][vq->avail_idx=256][vq->ndescs=61][vq->first_desc=0]
[  938.475600] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=61][vq->avail_idx=256][vq->ndescs=62][vq->first_desc=0]
[  938.475601] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=62][vq->avail_idx=256][vq->ndescs=63][vq->first_desc=0]
[  938.475601] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=63][vq->avail_idx=256][vq->ndescs=64][vq->first_desc=0]
[  938.475603] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=64][vq->avail_idx=256][vq->ndescs=65][vq->first_desc=0]
[  938.591320] [2235] 2334: [vq=000000007d36f807][vq->last_avail_idx=0][vq->avail_idx=1][vq->ndescs=1][vq->first_desc=0]
[  938.607139] [2235] 2334: [vq=000000007d36f807][vq->last_avail_idx=1][vq->avail_idx=2][vq->ndescs=1][vq->first_desc=0]
[  938.607538] [2235] 2334: [vq=000000007d36f807][vq->last_avail_idx=2][vq->avail_idx=4][vq->ndescs=1][vq->first_desc=0]
[  938.607540] [2235] 2334: [vq=000000007d36f807][vq->last_avail_idx=3][vq->avail_idx=4][vq->ndescs=2][vq->first_desc=0]
[  938.616010] [2235] 2334: [vq=000000007d36f807][vq->last_avail_idx=4][vq->avail_idx=5][vq->ndescs=1][vq->first_desc=0]
[  938.796034] [2235] 2334: [vq=000000007d36f807][vq->last_avail_idx=5][vq->avail_idx=6][vq->ndescs=1][vq->first_desc=0]
[  938.811243] [2235] 2334: [vq=000000007d36f807][vq->last_avail_idx=6][vq->avail_idx=7][vq->ndescs=2][vq->first_desc=0]
[  938.905985] [2235] 2334: [vq=000000007d36f807][vq->last_avail_idx=7][vq->avail_idx=8][vq->ndescs=1][vq->first_desc=0]
[  939.011405] [2235] 2334: [vq=000000007d36f807][vq->last_avail_idx=8][vq->avail_idx=9][vq->ndescs=2][vq->first_desc=0]
[  939.212298] [2235] 2334: [vq=000000007d36f807][vq->last_avail_idx=9][vq->avail_idx=10][vq->ndescs=2][vq->first_desc=0]
[  939.413649] [2235] 2334: [vq=000000007d36f807][vq->last_avail_idx=10][vq->avail_idx=11][vq->ndescs=2][vq->first_desc=0]
[  939.596036] [2235] 2334: [vq=000000007d36f807][vq->last_avail_idx=11][vq->avail_idx=12][vq->ndescs=1][vq->first_desc=0]
[  939.996031] [2235] 2334: [vq=000000007d36f807][vq->last_avail_idx=12][vq->avail_idx=13][vq->ndescs=1][vq->first_desc=0]
[  940.591371] [2235] 2334: [vq=000000007d36f807][vq->last_avail_idx=13][vq->avail_idx=14][vq->ndescs=1][vq->first_desc=0]
[  940.636040] [2235] 2334: [vq=000000007d36f807][vq->last_avail_idx=14][vq->avail_idx=15][vq->ndescs=1][vq->first_desc=0]
[  940.665984] [2235] 2334: [vq=000000007d36f807][vq->last_avail_idx=15][vq->avail_idx=16][vq->ndescs=1][vq->first_desc=0]
[  940.865048] [2235] 2334: [vq=000000007d36f807][vq->last_avail_idx=16][vq->avail_idx=17][vq->ndescs=2][vq->first_desc=0]
[  941.067594] [2235] 2334: [vq=000000007d36f807][vq->last_avail_idx=17][vq->avail_idx=18][vq->ndescs=2][vq->first_desc=0]
[  941.126013] [2235] 2334: [vq=000000007d36f807][vq->last_avail_idx=18][vq->avail_idx=19][vq->ndescs=1][vq->first_desc=0]
[  941.273297] [2235] 2334: [vq=000000007d36f807][vq->last_avail_idx=19][vq->avail_idx=20][vq->ndescs=2][vq->first_desc=0]
[  941.294416] [2235] 2334: [vq=000000007d36f807][vq->last_avail_idx=20][vq->avail_idx=21][vq->ndescs=1][vq->first_desc=0]
[  941.476486] [2235] 2334: [vq=000000007d36f807][vq->last_avail_idx=21][vq->avail_idx=22][vq->ndescs=2][vq->first_desc=0]
[  941.516034] [2235] 2334: [vq=000000007d36f807][vq->last_avail_idx=22][vq->avail_idx=23][vq->ndescs=1][vq->first_desc=0]
[  941.651361] [2235] 2334: [vq=000000007d36f807][vq->last_avail_idx=23][vq->avail_idx=24][vq->ndescs=1][vq->first_desc=0]
[  941.651942] [2235] 2334: [vq=000000007d36f807][vq->last_avail_idx=24][vq->avail_idx=25][vq->ndescs=1][vq->first_desc=0]
[  941.652190] [2235] 2334: [vq=000000007d36f807][vq->last_avail_idx=25][vq->avail_idx=26][vq->ndescs=1][vq->first_desc=0]
[  941.652363] [2235] 2334: [vq=000000007d36f807][vq->last_avail_idx=26][vq->avail_idx=27][vq->ndescs=1][vq->first_desc=0]
[  941.652542] [2235] 2334: [vq=000000007d36f807][vq->last_avail_idx=27][vq->avail_idx=28][vq->ndescs=1][vq->first_desc=0]
[  941.652715] [2235] 2334: [vq=000000007d36f807][vq->last_avail_idx=28][vq->avail_idx=29][vq->ndescs=1][vq->first_desc=0]
[  941.652882] [2235] 2334: [vq=000000007d36f807][vq->last_avail_idx=29][vq->avail_idx=30][vq->ndescs=1][vq->first_desc=0]
[  941.653050] [2235] 2334: [vq=000000007d36f807][vq->last_avail_idx=30][vq->avail_idx=31][vq->ndescs=1][vq->first_desc=0]
[  941.653218] [2235] 2334: [vq=000000007d36f807][vq->last_avail_idx=31][vq->avail_idx=32][vq->ndescs=1][vq->first_desc=0]
[  941.653388] [2235] 2334: [vq=000000007d36f807][vq->last_avail_idx=32][vq->avail_idx=33][vq->ndescs=1][vq->first_desc=0]
[  941.653558] [2235] 2334: [vq=000000007d36f807][vq->last_avail_idx=33][vq->avail_idx=34][vq->ndescs=1][vq->first_desc=0]
[  941.653725] [2235] 2334: [vq=000000007d36f807][vq->last_avail_idx=34][vq->avail_idx=35][vq->ndescs=1][vq->first_desc=0]
[  941.653896] [2235] 2334: [vq=000000007d36f807][vq->last_avail_idx=35][vq->avail_idx=36][vq->ndescs=1][vq->first_desc=0]
[  941.654063] [2235] 2334: [vq=000000007d36f807][vq->last_avail_idx=36][vq->avail_idx=37][vq->ndescs=1][vq->first_desc=0]
[  941.654229] [2235] 2334: [vq=000000007d36f807][vq->last_avail_idx=37][vq->avail_idx=38][vq->ndescs=1][vq->first_desc=0]
[  941.654398] [2235] 2334: [vq=000000007d36f807][vq->last_avail_idx=38][vq->avail_idx=39][vq->ndescs=1][vq->first_desc=0]
[  941.654567] [2235] 2334: [vq=000000007d36f807][vq->last_avail_idx=39][vq->avail_idx=40][vq->ndescs=1][vq->first_desc=0]
[  941.654738] [2235] 2334: [vq=000000007d36f807][vq->last_avail_idx=40][vq->avail_idx=41][vq->ndescs=1][vq->first_desc=0]
[  941.654906] [2235] 2334: [vq=000000007d36f807][vq->last_avail_idx=41][vq->avail_idx=42][vq->ndescs=1][vq->first_desc=0]
[  941.655070] [2235] 2334: [vq=000000007d36f807][vq->last_avail_idx=42][vq->avail_idx=43][vq->ndescs=1][vq->first_desc=0]
[  941.655240] [2235] 2334: [vq=000000007d36f807][vq->last_avail_idx=43][vq->avail_idx=44][vq->ndescs=1][vq->first_desc=0]
[  943.627121] [2235] 2334: [vq=000000007d36f807][vq->last_avail_idx=44][vq->avail_idx=45][vq->ndescs=1][vq->first_desc=0]
[  945.158600] [2228] 1726: VHOST_NET_SET_BACKEND
[  945.158603] [2228] 1538: sock=0000000012e3bc88 != oldsock=00000000de4d87a3 index=0 fd=-1 vq=00000000742665e9
[  945.158604] [2228] 1573: sock=0000000012e3bc88
[  945.158622] [2228] 1726: VHOST_NET_SET_BACKEND
[  945.158623] [2228] 1538: sock=0000000012e3bc88 != oldsock=00000000de4d87a3 index=1 fd=-1 vq=000000007d36f807
[  945.158624] [2228] 1573: sock=0000000012e3bc88
[  945.158646] [2228] 1666: VHOST_GET_VRING_BASE [vq=00000000742665e9][vq->last_avail_idx=65][vq->avail_idx=256][s.index=0][s.num=65]
[  945.158648] [2228] 1666: VHOST_GET_VRING_BASE [vq=000000007d36f807][vq->last_avail_idx=45][vq->avail_idx=45][s.index=1][s.num=45]
[  947.678597] [2237] 1647: strange VHOST_SET_VRING_BASE [vq=00000000742665e9][s.index=0][s.num=0]
[  947.678600] CPU: 22 PID: 2237 Comm: CPU 0/KVM Not tainted 5.5.0+ #21
[  947.678604] Hardware name: IBM 3906 M04 704 (LPAR)
[  947.678606] Call Trace:
[  947.678612]  [<0000001ede114132>] show_stack+0x8a/0xd0 
[  947.678615]  [<0000001edeb0672a>] dump_stack+0x8a/0xb8 
[  947.678625]  [<000003ff803736a6>] vhost_vring_ioctl+0x6fe/0x858 [vhost] 
[  947.678628]  [<000003ff8042a608>] vhost_net_ioctl+0x510/0x570 [vhost_net] 
[  947.678632]  [<0000001ede3c4dd8>] do_vfs_ioctl+0x430/0x6f8 
[  947.678634]  [<0000001ede3c5124>] ksys_ioctl+0x84/0xb0 
[  947.678635]  [<0000001ede3c51ba>] __s390x_sys_ioctl+0x2a/0x38 
[  947.678638]  [<0000001edeb27f72>] system_call+0x2a6/0x2c8 
[  947.678660] [2237] 1647: strange VHOST_SET_VRING_BASE [vq=000000007d36f807][s.index=1][s.num=0]
[  947.678661] CPU: 22 PID: 2237 Comm: CPU 0/KVM Not tainted 5.5.0+ #21
[  947.678662] Hardware name: IBM 3906 M04 704 (LPAR)
[  947.678662] Call Trace:
[  947.678664]  [<0000001ede114132>] show_stack+0x8a/0xd0 
[  947.678665]  [<0000001edeb0672a>] dump_stack+0x8a/0xb8 
[  947.678667]  [<000003ff803736a6>] vhost_vring_ioctl+0x6fe/0x858 [vhost] 
[  947.678669]  [<000003ff8042a608>] vhost_net_ioctl+0x510/0x570 [vhost_net] 
[  947.678671]  [<0000001ede3c4dd8>] do_vfs_ioctl+0x430/0x6f8 
[  947.678672]  [<0000001ede3c5124>] ksys_ioctl+0x84/0xb0 
[  947.678673]  [<0000001ede3c51ba>] __s390x_sys_ioctl+0x2a/0x38 
[  947.678675]  [<0000001edeb27f72>] system_call+0x2a6/0x2c8 
[  947.678694] [2237] 1726: VHOST_NET_SET_BACKEND
[  947.678697] [2237] 1538: sock=00000000de4d87a3 != oldsock=0000000012e3bc88 index=0 fd=39 vq=00000000742665e9
[  947.678702] [2237] 1573: sock=00000000de4d87a3
[  947.678703] [2237] 1726: VHOST_NET_SET_BACKEND
[  947.678704] [2237] 1538: sock=00000000de4d87a3 != oldsock=0000000012e3bc88 index=1 fd=39 vq=000000007d36f807
[  947.678705] [2237] 1573: sock=00000000de4d87a3
[  947.678717] Guest moved vq 000000007d36f807 used index from 45 to 0
[  950.562666] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=65][vq->avail_idx=256][vq->ndescs=1][vq->first_desc=0]
[  950.562671] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=66][vq->avail_idx=256][vq->ndescs=2][vq->first_desc=0]
[  950.562672] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=67][vq->avail_idx=256][vq->ndescs=3][vq->first_desc=0]
[  950.562673] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=68][vq->avail_idx=256][vq->ndescs=4][vq->first_desc=0]
[  950.562674] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=69][vq->avail_idx=256][vq->ndescs=5][vq->first_desc=0]
[  950.562675] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=70][vq->avail_idx=256][vq->ndescs=6][vq->first_desc=0]
[  950.562676] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=71][vq->avail_idx=256][vq->ndescs=7][vq->first_desc=0]
[  950.562677] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=72][vq->avail_idx=256][vq->ndescs=8][vq->first_desc=0]
[  950.562678] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=73][vq->avail_idx=256][vq->ndescs=9][vq->first_desc=0]
[  950.562679] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=74][vq->avail_idx=256][vq->ndescs=10][vq->first_desc=0]
[  950.562679] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=75][vq->avail_idx=256][vq->ndescs=11][vq->first_desc=0]
[  950.562680] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=76][vq->avail_idx=256][vq->ndescs=12][vq->first_desc=0]
[  950.562681] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=77][vq->avail_idx=256][vq->ndescs=13][vq->first_desc=0]
[  950.562682] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=78][vq->avail_idx=256][vq->ndescs=14][vq->first_desc=0]
[  950.562683] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=79][vq->avail_idx=256][vq->ndescs=15][vq->first_desc=0]
[  950.562684] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=80][vq->avail_idx=256][vq->ndescs=16][vq->first_desc=0]
[  950.562685] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=81][vq->avail_idx=256][vq->ndescs=17][vq->first_desc=0]
[  950.562686] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=82][vq->avail_idx=256][vq->ndescs=18][vq->first_desc=0]
[  950.562687] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=83][vq->avail_idx=256][vq->ndescs=19][vq->first_desc=0]
[  950.562688] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=84][vq->avail_idx=256][vq->ndescs=20][vq->first_desc=0]
[  950.562689] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=85][vq->avail_idx=256][vq->ndescs=21][vq->first_desc=0]
[  950.562690] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=86][vq->avail_idx=256][vq->ndescs=22][vq->first_desc=0]
[  950.562690] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=87][vq->avail_idx=256][vq->ndescs=23][vq->first_desc=0]
[  950.562691] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=88][vq->avail_idx=256][vq->ndescs=24][vq->first_desc=0]
[  950.562692] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=89][vq->avail_idx=256][vq->ndescs=25][vq->first_desc=0]
[  950.562693] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=90][vq->avail_idx=256][vq->ndescs=26][vq->first_desc=0]
[  950.562694] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=91][vq->avail_idx=256][vq->ndescs=27][vq->first_desc=0]
[  950.562695] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=92][vq->avail_idx=256][vq->ndescs=28][vq->first_desc=0]
[  950.562696] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=93][vq->avail_idx=256][vq->ndescs=29][vq->first_desc=0]
[  950.562697] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=94][vq->avail_idx=256][vq->ndescs=30][vq->first_desc=0]
[  950.562698] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=95][vq->avail_idx=256][vq->ndescs=31][vq->first_desc=0]
[  950.562699] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=96][vq->avail_idx=256][vq->ndescs=32][vq->first_desc=0]
[  950.562700] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=97][vq->avail_idx=256][vq->ndescs=33][vq->first_desc=0]
[  950.562700] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=98][vq->avail_idx=256][vq->ndescs=34][vq->first_desc=0]
[  950.562701] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=99][vq->avail_idx=256][vq->ndescs=35][vq->first_desc=0]
[  950.562702] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=100][vq->avail_idx=256][vq->ndescs=36][vq->first_desc=0]
[  950.562703] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=101][vq->avail_idx=256][vq->ndescs=37][vq->first_desc=0]
[  950.562704] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=102][vq->avail_idx=256][vq->ndescs=38][vq->first_desc=0]
[  950.562705] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=103][vq->avail_idx=256][vq->ndescs=39][vq->first_desc=0]
[  950.562706] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=104][vq->avail_idx=256][vq->ndescs=40][vq->first_desc=0]
[  950.562707] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=105][vq->avail_idx=256][vq->ndescs=41][vq->first_desc=0]
[  950.562707] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=106][vq->avail_idx=256][vq->ndescs=42][vq->first_desc=0]
[  950.562708] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=107][vq->avail_idx=256][vq->ndescs=43][vq->first_desc=0]
[  950.562709] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=108][vq->avail_idx=256][vq->ndescs=44][vq->first_desc=0]
[  950.562710] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=109][vq->avail_idx=256][vq->ndescs=45][vq->first_desc=0]
[  950.562711] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=110][vq->avail_idx=256][vq->ndescs=46][vq->first_desc=0]
[  950.562712] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=111][vq->avail_idx=256][vq->ndescs=47][vq->first_desc=0]
[  950.562713] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=112][vq->avail_idx=256][vq->ndescs=48][vq->first_desc=0]
[  950.562714] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=113][vq->avail_idx=256][vq->ndescs=49][vq->first_desc=0]
[  950.562715] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=114][vq->avail_idx=256][vq->ndescs=50][vq->first_desc=0]
[  950.562715] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=115][vq->avail_idx=256][vq->ndescs=51][vq->first_desc=0]
[  950.562716] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=116][vq->avail_idx=256][vq->ndescs=52][vq->first_desc=0]
[  950.562717] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=117][vq->avail_idx=256][vq->ndescs=53][vq->first_desc=0]
[  950.562718] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=118][vq->avail_idx=256][vq->ndescs=54][vq->first_desc=0]
[  950.562719] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=119][vq->avail_idx=256][vq->ndescs=55][vq->first_desc=0]
[  950.562720] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=120][vq->avail_idx=256][vq->ndescs=56][vq->first_desc=0]
[  950.562721] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=121][vq->avail_idx=256][vq->ndescs=57][vq->first_desc=0]
[  950.562722] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=122][vq->avail_idx=256][vq->ndescs=58][vq->first_desc=0]
[  950.562723] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=123][vq->avail_idx=256][vq->ndescs=59][vq->first_desc=0]
[  950.562724] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=124][vq->avail_idx=256][vq->ndescs=60][vq->first_desc=0]
[  950.562725] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=125][vq->avail_idx=256][vq->ndescs=61][vq->first_desc=0]
[  950.562726] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=126][vq->avail_idx=256][vq->ndescs=62][vq->first_desc=0]
[  950.562727] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=127][vq->avail_idx=256][vq->ndescs=63][vq->first_desc=0]
[  950.562728] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=128][vq->avail_idx=256][vq->ndescs=64][vq->first_desc=0]
[  950.562730] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=129][vq->avail_idx=256][vq->ndescs=65][vq->first_desc=0]
[  968.267262] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=130][vq->avail_idx=256][vq->ndescs=1][vq->first_desc=0]
[  968.267264] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=131][vq->avail_idx=256][vq->ndescs=2][vq->first_desc=0]
[  968.267265] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=132][vq->avail_idx=256][vq->ndescs=3][vq->first_desc=0]
[  968.267266] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=133][vq->avail_idx=256][vq->ndescs=4][vq->first_desc=0]
[  968.267267] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=134][vq->avail_idx=256][vq->ndescs=5][vq->first_desc=0]
[  968.267268] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=135][vq->avail_idx=256][vq->ndescs=6][vq->first_desc=0]
[  968.267269] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=136][vq->avail_idx=256][vq->ndescs=7][vq->first_desc=0]
[  968.267270] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=137][vq->avail_idx=256][vq->ndescs=8][vq->first_desc=0]
[  968.267271] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=138][vq->avail_idx=256][vq->ndescs=9][vq->first_desc=0]
[  968.267271] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=139][vq->avail_idx=256][vq->ndescs=10][vq->first_desc=0]
[  968.267272] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=140][vq->avail_idx=256][vq->ndescs=11][vq->first_desc=0]
[  968.267273] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=141][vq->avail_idx=256][vq->ndescs=12][vq->first_desc=0]
[  968.267274] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=142][vq->avail_idx=256][vq->ndescs=13][vq->first_desc=0]
[  968.267275] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=143][vq->avail_idx=256][vq->ndescs=14][vq->first_desc=0]
[  968.267276] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=144][vq->avail_idx=256][vq->ndescs=15][vq->first_desc=0]
[  968.267277] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=145][vq->avail_idx=256][vq->ndescs=16][vq->first_desc=0]
[  968.267278] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=146][vq->avail_idx=256][vq->ndescs=17][vq->first_desc=0]
[  968.267278] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=147][vq->avail_idx=256][vq->ndescs=18][vq->first_desc=0]
[  968.267279] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=148][vq->avail_idx=256][vq->ndescs=19][vq->first_desc=0]
[  968.267280] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=149][vq->avail_idx=256][vq->ndescs=20][vq->first_desc=0]
[  968.267281] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=150][vq->avail_idx=256][vq->ndescs=21][vq->first_desc=0]
[  968.267282] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=151][vq->avail_idx=256][vq->ndescs=22][vq->first_desc=0]
[  968.267283] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=152][vq->avail_idx=256][vq->ndescs=23][vq->first_desc=0]
[  968.267284] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=153][vq->avail_idx=256][vq->ndescs=24][vq->first_desc=0]
[  968.267284] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=154][vq->avail_idx=256][vq->ndescs=25][vq->first_desc=0]
[  968.267285] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=155][vq->avail_idx=256][vq->ndescs=26][vq->first_desc=0]
[  968.267286] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=156][vq->avail_idx=256][vq->ndescs=27][vq->first_desc=0]
[  968.267287] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=157][vq->avail_idx=256][vq->ndescs=28][vq->first_desc=0]
[  968.267288] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=158][vq->avail_idx=256][vq->ndescs=29][vq->first_desc=0]
[  968.267289] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=159][vq->avail_idx=256][vq->ndescs=30][vq->first_desc=0]
[  968.267290] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=160][vq->avail_idx=256][vq->ndescs=31][vq->first_desc=0]
[  968.267291] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=161][vq->avail_idx=256][vq->ndescs=32][vq->first_desc=0]
[  968.267292] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=162][vq->avail_idx=256][vq->ndescs=33][vq->first_desc=0]
[  968.267292] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=163][vq->avail_idx=256][vq->ndescs=34][vq->first_desc=0]
[  968.267293] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=164][vq->avail_idx=256][vq->ndescs=35][vq->first_desc=0]
[  968.267294] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=165][vq->avail_idx=256][vq->ndescs=36][vq->first_desc=0]
[  968.267295] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=166][vq->avail_idx=256][vq->ndescs=37][vq->first_desc=0]
[  968.267296] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=167][vq->avail_idx=256][vq->ndescs=38][vq->first_desc=0]
[  968.267297] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=168][vq->avail_idx=256][vq->ndescs=39][vq->first_desc=0]
[  968.267298] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=169][vq->avail_idx=256][vq->ndescs=40][vq->first_desc=0]
[  968.267298] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=170][vq->avail_idx=256][vq->ndescs=41][vq->first_desc=0]
[  968.267299] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=171][vq->avail_idx=256][vq->ndescs=42][vq->first_desc=0]
[  968.267300] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=172][vq->avail_idx=256][vq->ndescs=43][vq->first_desc=0]
[  968.267301] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=173][vq->avail_idx=256][vq->ndescs=44][vq->first_desc=0]
[  968.267302] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=174][vq->avail_idx=256][vq->ndescs=45][vq->first_desc=0]
[  968.267303] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=175][vq->avail_idx=256][vq->ndescs=46][vq->first_desc=0]
[  968.267304] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=176][vq->avail_idx=256][vq->ndescs=47][vq->first_desc=0]
[  968.267304] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=177][vq->avail_idx=256][vq->ndescs=48][vq->first_desc=0]
[  968.267305] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=178][vq->avail_idx=256][vq->ndescs=49][vq->first_desc=0]
[  968.267306] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=179][vq->avail_idx=256][vq->ndescs=50][vq->first_desc=0]
[  968.267307] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=180][vq->avail_idx=256][vq->ndescs=51][vq->first_desc=0]
[  968.267308] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=181][vq->avail_idx=256][vq->ndescs=52][vq->first_desc=0]
[  968.267309] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=182][vq->avail_idx=256][vq->ndescs=53][vq->first_desc=0]
[  968.267310] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=183][vq->avail_idx=256][vq->ndescs=54][vq->first_desc=0]
[  968.267310] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=184][vq->avail_idx=256][vq->ndescs=55][vq->first_desc=0]
[  968.267311] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=185][vq->avail_idx=256][vq->ndescs=56][vq->first_desc=0]
[  968.267312] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=186][vq->avail_idx=256][vq->ndescs=57][vq->first_desc=0]
[  968.267313] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=187][vq->avail_idx=256][vq->ndescs=58][vq->first_desc=0]
[  968.267314] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=188][vq->avail_idx=256][vq->ndescs=59][vq->first_desc=0]
[  968.267315] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=189][vq->avail_idx=256][vq->ndescs=60][vq->first_desc=0]
[  968.267316] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=190][vq->avail_idx=256][vq->ndescs=61][vq->first_desc=0]
[  968.267317] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=191][vq->avail_idx=256][vq->ndescs=62][vq->first_desc=0]
[  968.267318] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=192][vq->avail_idx=256][vq->ndescs=63][vq->first_desc=0]
[  968.267319] [2235] 2334: [vq=00000000742665e9][vq->last_avail_idx=193][vq->avail_idx=256][vq->ndescs=64][vq->first_desc=0]

Re: vhost changes (batched) in linux-next after 12/13 trigger random crashes in KVM guests after reboot

[Eugenio Pérez]

On Fri, 2020-02-14 at 08:47 +0100, Christian Borntraeger wrote:
> repro
> 
> 
> On 14.02.20 08:43, Christian Borntraeger wrote:
> > 
> > On 14.02.20 08:40, Eugenio Perez Martin wrote:
> > > Hi.
> > > 
> > > Were the vhost and vhost_net modules loaded with dyndbg='+plt'? I miss
> > > all the others regular debug traces on that one.
> > 
> > I did 
> > 
> >  echo -n 'file drivers/vhost/vhost.c +plt' > control
> > and
> > echo -n 'file drivers/vhost/net.c +plt'  > control
> > 
> > but apparently it did not work...me hates dynamic debug.

Sorry about use dyndbg, but it is the easiest way to obtain accurate line numbers, threads id...

I usually load the module with that option (modprobe vhost dyndbg=+pfmlt && modprobe vhost_net dyndbg=+pfmlt), so it is
available from the very beginning.

> > 
> > > Thanks!
> > > 
> > > On Fri, Feb 14, 2020 at 8:34 AM Christian Borntraeger
> > > <borntraeger@de.ibm.com> wrote:
> > > > I did
> > > > ping -c 20 -f ... ; reboot
> > > > twice
> > > > 
> > > > The ping after the first reboot showed .......E
> > > > 
> > > > this was on the host console
> > > > 
> > > > [   55.951885] CPU: 34 PID: 1908 Comm: CPU 0/KVM Not tainted 5.5.0+ #21
> > > > [   55.951891] Hardware name: IBM 3906 M04 704 (LPAR)
> > > > [   55.951892] Call Trace:
> > > > [   55.951902]  [<0000001ede114132>] show_stack+0x8a/0xd0
> > > > [   55.951906]  [<0000001edeb0672a>] dump_stack+0x8a/0xb8
> > > > [   55.951915]  [<000003ff803736a6>] vhost_vring_ioctl+0x6fe/0x858 [vhost]
> > > > [   55.951919]  [<000003ff8042a608>] vhost_net_ioctl+0x510/0x570 [vhost_net]
> > > > [   55.951924]  [<0000001ede3c4dd8>] do_vfs_ioctl+0x430/0x6f8
> > > > [   55.951926]  [<0000001ede3c5124>] ksys_ioctl+0x84/0xb0
> > > > [   55.951927]  [<0000001ede3c51ba>] __s390x_sys_ioctl+0x2a/0x38
> > > > [   55.951931]  [<0000001edeb27f72>] system_call+0x2a6/0x2c8
> > > > [   55.951949] CPU: 34 PID: 1908 Comm: CPU 0/KVM Not tainted 5.5.0+ #21
> > > > [   55.951950] Hardware name: IBM 3906 M04 704 (LPAR)
> > > > [   55.951951] Call Trace:
> > > > [   55.951952]  [<0000001ede114132>] show_stack+0x8a/0xd0
> > > > [   55.951954]  [<0000001edeb0672a>] dump_stack+0x8a/0xb8
> > > > [   55.951956]  [<000003ff803736a6>] vhost_vring_ioctl+0x6fe/0x858 [vhost]
> > > > [   55.951958]  [<000003ff8042a608>] vhost_net_ioctl+0x510/0x570 [vhost_net]
> > > > [   55.951959]  [<0000001ede3c4dd8>] do_vfs_ioctl+0x430/0x6f8
> > > > [   55.951961]  [<0000001ede3c5124>] ksys_ioctl+0x84/0xb0
> > > > [   55.951962]  [<0000001ede3c51ba>] __s390x_sys_ioctl+0x2a/0x38
> > > > [   55.951964]  [<0000001edeb27f72>] system_call+0x2a6/0x2c8
> > > > [   55.951997] Guest moved vq 0000000063d896c6 used index from 44 to 0
> > > > [   56.609831] unexpected descriptor format for RX: out 0, in 0
> > > > [   86.540460] CPU: 6 PID: 1908 Comm: CPU 0/KVM Not tainted 5.5.0+ #21
> > > > [   86.540464] Hardware name: IBM 3906 M04 704 (LPAR)
> > > > [   86.540466] Call Trace:
> > > > [   86.540473]  [<0000001ede114132>] show_stack+0x8a/0xd0
> > > > [   86.540477]  [<0000001edeb0672a>] dump_stack+0x8a/0xb8
> > > > [   86.540486]  [<000003ff803736a6>] vhost_vring_ioctl+0x6fe/0x858 [vhost]
> > > > [   86.540490]  [<000003ff8042a608>] vhost_net_ioctl+0x510/0x570 [vhost_net]
> > > > [   86.540494]  [<0000001ede3c4dd8>] do_vfs_ioctl+0x430/0x6f8
> > > > [   86.540496]  [<0000001ede3c5124>] ksys_ioctl+0x84/0xb0
> > > > [   86.540498]  [<0000001ede3c51ba>] __s390x_sys_ioctl+0x2a/0x38
> > > > [   86.540501]  [<0000001edeb27f72>] system_call+0x2a6/0x2c8
> > > > [   86.540524] CPU: 6 PID: 1908 Comm: CPU 0/KVM Not tainted 5.5.0+ #21
> > > > [   86.540525] Hardware name: IBM 3906 M04 704 (LPAR)
> > > > [   86.540526] Call Trace:
> > > > [   86.540527]  [<0000001ede114132>] show_stack+0x8a/0xd0
> > > > [   86.540528]  [<0000001edeb0672a>] dump_stack+0x8a/0xb8
> > > > [   86.540531]  [<000003ff803736a6>] vhost_vring_ioctl+0x6fe/0x858 [vhost]
> > > > [   86.540532]  [<000003ff8042a608>] vhost_net_ioctl+0x510/0x570 [vhost_net]
> > > > [   86.540534]  [<0000001ede3c4dd8>] do_vfs_ioctl+0x430/0x6f8
> > > > [   86.540536]  [<0000001ede3c5124>] ksys_ioctl+0x84/0xb0
> > > > [   86.540537]  [<0000001ede3c51ba>] __s390x_sys_ioctl+0x2a/0x38
> > > > [   86.540538]  [<0000001edeb27f72>] system_call+0x2a6/0x2c8
> > > > [   86.540570] unexpected descriptor format for RX: out 0, in 0
> > > > [   86.540577] Unexpected header len for TX: 0 expected 0
> > > > 
> > > > 
> > > > On 14.02.20 08:06, Eugenio Pérez wrote:
> > > > > Hi Christian.
> > > > > 
> > > > > Sorry, that was meant to be applied over previous debug patch.
> > > > > 
> > > > > Here I inline the one meant to be applied over eccb852f1fe6bede630e2e4f1a121a81e34354ab.
> > > > > 
> > > > > Thanks!
> > > > > 
> > > > > From d978ace99e4844b49b794d768385db3d128a4cc0 Mon Sep 17 00:00:00 2001
> > > > > From: =?UTF-8?q?Eugenio=20P=C3=A9rez?= <eperezma@redhat.com>
> > > > > Date: Fri, 14 Feb 2020 08:02:26 +0100
> > > > > Subject: [PATCH] vhost: disable all features and trace last_avail_idx and
> > > > >  ioctl calls
> > > > > 
> > > > > ---
> > > > >  drivers/vhost/net.c   | 20 +++++++++++++++++---
> > > > >  drivers/vhost/vhost.c | 25 +++++++++++++++++++++++--
> > > > >  drivers/vhost/vhost.h | 10 +++++-----
> > > > >  3 files changed, 45 insertions(+), 10 deletions(-)
> > > > > 
> > > > > diff --git a/drivers/vhost/net.c b/drivers/vhost/net.c
> > > > > index e158159671fa..e4d5f843f9c0 100644
> > > > > --- a/drivers/vhost/net.c
> > > > > +++ b/drivers/vhost/net.c
> > > > > @@ -1505,10 +1505,13 @@ static long vhost_net_set_backend(struct vhost_net *n, unsigned index, int fd)
> > > > > 
> > > > >       mutex_lock(&n->dev.mutex);
> > > > >       r = vhost_dev_check_owner(&n->dev);
> > > > > -     if (r)
> > > > > +     if (r) {
> > > > > +             pr_debug("vhost_dev_check_owner index=%u fd=%d rc r=%d", index, fd, r);
> > > > >               goto err;
> > > > > +     }
> > > > > 
> > > > >       if (index >= VHOST_NET_VQ_MAX) {
> > > > > +             pr_debug("vhost_dev_check_owner index=%u fd=%d MAX=%d", index, fd, VHOST_NET_VQ_MAX);
> > > > >               r = -ENOBUFS;
> > > > >               goto err;
> > > > >       }
> > > > > @@ -1518,22 +1521,26 @@ static long vhost_net_set_backend(struct vhost_net *n, unsigned index, int fd)
> > > > > 
> > > > >       /* Verify that ring has been setup correctly. */
> > > > >       if (!vhost_vq_access_ok(vq)) {
> > > > > +             pr_debug("vhost_net_set_backend index=%u fd=%d !vhost_vq_access_ok", index, fd);
> > > > >               r = -EFAULT;
> > > > >               goto err_vq;
> > > > >       }
> > > > >       sock = get_socket(fd);
> > > > >       if (IS_ERR(sock)) {
> > > > >               r = PTR_ERR(sock);
> > > > > +             pr_debug("vhost_net_set_backend index=%u fd=%d get_socket err r=%d", index, fd, r);
> > > > >               goto err_vq;
> > > > >       }
> > > > > 
> > > > >       /* start polling new socket */
> > > > >       oldsock = vq->private_data;
> > > > >       if (sock != oldsock) {
> > > > > +             pr_debug("sock=%p != oldsock=%p index=%u fd=%d vq=%p", sock, oldsock, index, fd, vq);
> > > > >               ubufs = vhost_net_ubuf_alloc(vq,
> > > > >                                            sock && vhost_sock_zcopy(sock));
> > > > >               if (IS_ERR(ubufs)) {
> > > > >                       r = PTR_ERR(ubufs);
> > > > > +                     pr_debug("ubufs index=%u fd=%d err r=%d vq=%p", index, fd, r, vq);
> > > > >                       goto err_ubufs;
> > > > >               }
> > > > > 
> > > > > @@ -1541,11 +1548,15 @@ static long vhost_net_set_backend(struct vhost_net *n, unsigned index, int fd)
> > > > >               vq->private_data = sock;
> > > > >               vhost_net_buf_unproduce(nvq);
> > > > >               r = vhost_vq_init_access(vq);
> > > > > -             if (r)
> > > > > +             if (r) {
> > > > > +                     pr_debug("init_access index=%u fd=%d r=%d vq=%p", index, fd, r, vq);
> > > > >                       goto err_used;
> > > > > +             }
> > > > >               r = vhost_net_enable_vq(n, vq);
> > > > > -             if (r)
> > > > > +             if (r) {
> > > > > +                     pr_debug("enable_vq index=%u fd=%d r=%d vq=%p", index, fd, r, vq);
> > > > >                       goto err_used;
> > > > > +             }
> > > > >               if (index == VHOST_NET_VQ_RX)
> > > > >                       nvq->rx_ring = get_tap_ptr_ring(fd);
> > > > > 
> > > > > @@ -1559,6 +1570,8 @@ static long vhost_net_set_backend(struct vhost_net *n, unsigned index, int fd)
> > > > > 
> > > > >       mutex_unlock(&vq->mutex);
> > > > > 
> > > > > +     pr_debug("sock=%p", sock);
> > > > > +
> > > > >       if (oldubufs) {
> > > > >               vhost_net_ubuf_put_wait_and_free(oldubufs);
> > > > >               mutex_lock(&vq->mutex);
> > > > > @@ -1710,6 +1723,7 @@ static long vhost_net_ioctl(struct file *f, unsigned int ioctl,
> > > > > 
> > > > >       switch (ioctl) {
> > > > >       case VHOST_NET_SET_BACKEND:
> > > > > +             pr_debug("VHOST_NET_SET_BACKEND");
> > > > >               if (copy_from_user(&backend, argp, sizeof backend))
> > > > >                       return -EFAULT;
> > > > >               return vhost_net_set_backend(n, backend.index, backend.fd);
> > > > > diff --git a/drivers/vhost/vhost.c b/drivers/vhost/vhost.c
> > > > > index b5a51b1f2e79..ec25ba32fe81 100644
> > > > > --- a/drivers/vhost/vhost.c
> > > > > +++ b/drivers/vhost/vhost.c
> > > > > @@ -1642,15 +1642,30 @@ long vhost_vring_ioctl(struct vhost_dev *d, unsigned int ioctl, void __user *arg
> > > > >                       r = -EINVAL;
> > > > >                       break;
> > > > >               }
> > > > > +
> > > > > +             if (vq->last_avail_idx || vq->avail_idx) {
> > > > > +                     pr_debug(
> > > > > +                             "strange VHOST_SET_VRING_BASE [vq=%p][s.index=%u][s.num=%u]",
> > > > > +                             vq, s.index, s.num);
> > > > > +                     dump_stack();
> > > > > +                     r = 0;
> > > > > +                     break;
> > > > > +             }
> > > > >               vq->last_avail_idx = s.num;
> > > > >               /* Forget the cached index value. */
> > > > >               vq->avail_idx = vq->last_avail_idx;
> > > > > +             pr_debug(
> > > > > +                     "VHOST_SET_VRING_BASE [vq=%p][vq->last_avail_idx=%u][vq-
> > > > > >avail_idx=%u][s.index=%u][s.num=%u]",
> > > > > +                     vq, vq->last_avail_idx, vq->avail_idx, s.index, s.num);
> > > > >               break;
> > > > >       case VHOST_GET_VRING_BASE:
> > > > >               s.index = idx;
> > > > >               s.num = vq->last_avail_idx;
> > > > >               if (copy_to_user(argp, &s, sizeof s))
> > > > >                       r = -EFAULT;
> > > > > +             pr_debug(
> > > > > +                     "VHOST_GET_VRING_BASE [vq=%p][vq->last_avail_idx=%u][vq-
> > > > > >avail_idx=%u][s.index=%u][s.num=%u]",
> > > > > +                     vq, vq->last_avail_idx, vq->avail_idx, s.index, s.num);
> > > > >               break;
> > > > >       case VHOST_SET_VRING_KICK:
> > > > >               if (copy_from_user(&f, argp, sizeof f)) {
> > > > > @@ -2239,8 +2254,8 @@ static int fetch_buf(struct vhost_virtqueue *vq)
> > > > >               vq->avail_idx = vhost16_to_cpu(vq, avail_idx);
> > > > > 
> > > > >               if (unlikely((u16)(vq->avail_idx - last_avail_idx) > vq->num)) {
> > > > > -                     vq_err(vq, "Guest moved used index from %u to %u",
> > > > > -                             last_avail_idx, vq->avail_idx);
> > > > > +                     vq_err(vq, "Guest moved vq %p used index from %u to %u",
> > > > > +                             vq, last_avail_idx, vq->avail_idx);
> > > > >                       return -EFAULT;
> > > > >               }
> > > > > 
> > > > > @@ -2316,6 +2331,9 @@ static int fetch_buf(struct vhost_virtqueue *vq)
> > > > >       BUG_ON(!(vq->used_flags & VRING_USED_F_NO_NOTIFY));
> > > > > 
> > > > >       /* On success, increment avail index. */
> > > > > +     pr_debug(
> > > > > +             "[vq=%p][vq->last_avail_idx=%u][vq->avail_idx=%u][vq->ndescs=%d][vq->first_desc=%d]",
> > > > > +             vq, vq->last_avail_idx, vq->avail_idx, vq->ndescs, vq->first_desc);
> > > > >       vq->last_avail_idx++;
> > > > > 
> > > > >       return 0;
> > > > > @@ -2432,6 +2450,9 @@ EXPORT_SYMBOL_GPL(vhost_get_vq_desc);
> > > > >  /* Reverse the effect of vhost_get_vq_desc. Useful for error handling. */
> > > > >  void vhost_discard_vq_desc(struct vhost_virtqueue *vq, int n)
> > > > >  {
> > > > > +     pr_debug(
> > > > > +             "DISCARD [vq=%p][vq->last_avail_idx=%u][vq->avail_idx=%u][n=%d]",
> > > > > +             vq, vq->last_avail_idx, vq->avail_idx, n);
> > > > >       vq->last_avail_idx -= n;
> > > > >  }
> > > > >  EXPORT_SYMBOL_GPL(vhost_discard_vq_desc);
> > > > > diff --git a/drivers/vhost/vhost.h b/drivers/vhost/vhost.h
> > > > > index 661088ae6dc7..08f6d2ccb697 100644
> > > > > --- a/drivers/vhost/vhost.h
> > > > > +++ b/drivers/vhost/vhost.h
> > > > > @@ -250,11 +250,11 @@ int vhost_init_device_iotlb(struct vhost_dev *d, bool enabled);
> > > > >       } while (0)
> > > > > 
> > > > >  enum {
> > > > > -     VHOST_FEATURES = (1ULL << VIRTIO_F_NOTIFY_ON_EMPTY) |
> > > > > -                      (1ULL << VIRTIO_RING_F_INDIRECT_DESC) |
> > > > > -                      (1ULL << VIRTIO_RING_F_EVENT_IDX) |
> > > > > -                      (1ULL << VHOST_F_LOG_ALL) |
> > > > > -                      (1ULL << VIRTIO_F_ANY_LAYOUT) |
> > > > > +     VHOST_FEATURES = /* (1ULL << VIRTIO_F_NOTIFY_ON_EMPTY) | */
> > > > > +                      /* (1ULL << VIRTIO_RING_F_INDIRECT_DESC) | */
> > > > > +                      /* (1ULL << VIRTIO_RING_F_EVENT_IDX) | */
> > > > > +                      /* (1ULL << VHOST_F_LOG_ALL) | */
> > > > > +                      /* (1ULL << VIRTIO_F_ANY_LAYOUT) | */
> > > > >                        (1ULL << VIRTIO_F_VERSION_1)
> > > > >  };
> > > > > 
> > > > > 

Can you try the inlined patch over 52c36ce7f334 ("vhost: use batched version by default")? My intention is to check if
"strange VHOST_SET_VRING_BASE" line appears. In previous tests, it appears very fast, but maybe it takes some time for
it to appear, or it does not appear anymore.

Thanks!

From 756a96e489688b2c04230580770aac17c8a46265 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Eugenio=20P=C3=A9rez?= <eperezma@redhat.com>
Date: Fri, 14 Feb 2020 08:02:26 +0100
Subject: [PATCH] vhost: disable all features and trace last_avail_idx and
 ioctl calls

---
 drivers/vhost/net.c   | 20 +++++++++++++++++---
 drivers/vhost/vhost.c | 25 +++++++++++++++++++++++--
 drivers/vhost/vhost.h | 10 +++++-----
 3 files changed, 45 insertions(+), 10 deletions(-)

diff --git a/drivers/vhost/net.c b/drivers/vhost/net.c
index e158159671fa..e4d5f843f9c0 100644
--- a/drivers/vhost/net.c
+++ b/drivers/vhost/net.c
@@ -1505,10 +1505,13 @@ static long vhost_net_set_backend(struct vhost_net *n, unsigned index, int fd)
 
 	mutex_lock(&n->dev.mutex);
 	r = vhost_dev_check_owner(&n->dev);
-	if (r)
+	if (r) {
+		pr_debug("vhost_dev_check_owner index=%u fd=%d rc r=%d", index, fd, r);
 		goto err;
+	}
 
 	if (index >= VHOST_NET_VQ_MAX) {
+		pr_debug("vhost_dev_check_owner index=%u fd=%d MAX=%d", index, fd, VHOST_NET_VQ_MAX);
 		r = -ENOBUFS;
 		goto err;
 	}
@@ -1518,22 +1521,26 @@ static long vhost_net_set_backend(struct vhost_net *n, unsigned index, int fd)
 
 	/* Verify that ring has been setup correctly. */
 	if (!vhost_vq_access_ok(vq)) {
+		pr_debug("vhost_net_set_backend index=%u fd=%d !vhost_vq_access_ok", index, fd);
 		r = -EFAULT;
 		goto err_vq;
 	}
 	sock = get_socket(fd);
 	if (IS_ERR(sock)) {
 		r = PTR_ERR(sock);
+		pr_debug("vhost_net_set_backend index=%u fd=%d get_socket err r=%d", index, fd, r);
 		goto err_vq;
 	}
 
 	/* start polling new socket */
 	oldsock = vq->private_data;
 	if (sock != oldsock) {
+		pr_debug("sock=%p != oldsock=%p index=%u fd=%d vq=%p", sock, oldsock, index, fd, vq);
 		ubufs = vhost_net_ubuf_alloc(vq,
 					     sock && vhost_sock_zcopy(sock));
 		if (IS_ERR(ubufs)) {
 			r = PTR_ERR(ubufs);
+			pr_debug("ubufs index=%u fd=%d err r=%d vq=%p", index, fd, r, vq);
 			goto err_ubufs;
 		}
 
@@ -1541,11 +1548,15 @@ static long vhost_net_set_backend(struct vhost_net *n, unsigned index, int fd)
 		vq->private_data = sock;
 		vhost_net_buf_unproduce(nvq);
 		r = vhost_vq_init_access(vq);
-		if (r)
+		if (r) {
+			pr_debug("init_access index=%u fd=%d r=%d vq=%p", index, fd, r, vq);
 			goto err_used;
+		}
 		r = vhost_net_enable_vq(n, vq);
-		if (r)
+		if (r) {
+			pr_debug("enable_vq index=%u fd=%d r=%d vq=%p", index, fd, r, vq);
 			goto err_used;
+		}
 		if (index == VHOST_NET_VQ_RX)
 			nvq->rx_ring = get_tap_ptr_ring(fd);
 
@@ -1559,6 +1570,8 @@ static long vhost_net_set_backend(struct vhost_net *n, unsigned index, int fd)
 
 	mutex_unlock(&vq->mutex);
 
+	pr_debug("sock=%p", sock);
+
 	if (oldubufs) {
 		vhost_net_ubuf_put_wait_and_free(oldubufs);
 		mutex_lock(&vq->mutex);
@@ -1710,6 +1723,7 @@ static long vhost_net_ioctl(struct file *f, unsigned int ioctl,
 
 	switch (ioctl) {
 	case VHOST_NET_SET_BACKEND:
+		pr_debug("VHOST_NET_SET_BACKEND");
 		if (copy_from_user(&backend, argp, sizeof backend))
 			return -EFAULT;
 		return vhost_net_set_backend(n, backend.index, backend.fd);
diff --git a/drivers/vhost/vhost.c b/drivers/vhost/vhost.c
index 56c5253056ee..babc48c8a8c4 100644
--- a/drivers/vhost/vhost.c
+++ b/drivers/vhost/vhost.c
@@ -1640,15 +1640,30 @@ long vhost_vring_ioctl(struct vhost_dev *d, unsigned int ioctl, void __user *arg
 			r = -EINVAL;
 			break;
 		}
+
+		if (vq->last_avail_idx || vq->avail_idx) {
+			pr_debug(
+				"strange VHOST_SET_VRING_BASE [vq=%p][s.index=%u][s.num=%u]",
+				vq, s.index, s.num);
+			dump_stack();
+			r = 0;
+			break;
+		}
 		vq->last_avail_idx = s.num;
 		/* Forget the cached index value. */
 		vq->avail_idx = vq->last_avail_idx;
+		pr_debug(
+			"VHOST_SET_VRING_BASE [vq=%p][vq->last_avail_idx=%u][vq->avail_idx=%u][s.index=%u][s.num=%u]",
+			vq, vq->last_avail_idx, vq->avail_idx, s.index, s.num);
 		break;
 	case VHOST_GET_VRING_BASE:
 		s.index = idx;
 		s.num = vq->last_avail_idx;
 		if (copy_to_user(argp, &s, sizeof s))
 			r = -EFAULT;
+		pr_debug(
+			"VHOST_GET_VRING_BASE [vq=%p][vq->last_avail_idx=%u][vq->avail_idx=%u][s.index=%u][s.num=%u]",
+			vq, vq->last_avail_idx, vq->avail_idx, s.index, s.num);
 		break;
 	case VHOST_SET_VRING_KICK:
 		if (copy_from_user(&f, argp, sizeof f)) {
@@ -2233,8 +2248,8 @@ static int fetch_descs(struct vhost_virtqueue *vq)
 		vq->avail_idx = vhost16_to_cpu(vq, avail_idx);
 
 		if (unlikely((u16)(vq->avail_idx - last_avail_idx) > vq->num)) {
-			vq_err(vq, "Guest moved used index from %u to %u",
-				last_avail_idx, vq->avail_idx);
+			vq_err(vq, "Guest moved vq %p used index from %u to %u",
+				vq, last_avail_idx, vq->avail_idx);
 			return -EFAULT;
 		}
 
@@ -2310,6 +2325,9 @@ static int fetch_descs(struct vhost_virtqueue *vq)
 	BUG_ON(!(vq->used_flags & VRING_USED_F_NO_NOTIFY));
 
 	/* On success, increment avail index. */
+	pr_debug(
+		"[vq=%p][vq->last_avail_idx=%u][vq->avail_idx=%u][vq->ndescs=%d][vq->first_desc=%d]",
+		vq, vq->last_avail_idx, vq->avail_idx, vq->ndescs, vq->first_desc);
 	vq->last_avail_idx++;
 
 	return 0;
@@ -2403,6 +2421,9 @@ EXPORT_SYMBOL_GPL(vhost_get_vq_desc);
 /* Reverse the effect of vhost_get_vq_desc. Useful for error handling. */
 void vhost_discard_vq_desc(struct vhost_virtqueue *vq, int n)
 {
+	pr_debug(
+		"DISCARD [vq=%p][vq->last_avail_idx=%u][vq->avail_idx=%u][n=%d]",
+		vq, vq->last_avail_idx, vq->avail_idx, n);
 	vq->last_avail_idx -= n;
 }
 EXPORT_SYMBOL_GPL(vhost_discard_vq_desc);
diff --git a/drivers/vhost/vhost.h b/drivers/vhost/vhost.h
index a0bcf8bffa43..2ce2d3a97c31 100644
--- a/drivers/vhost/vhost.h
+++ b/drivers/vhost/vhost.h
@@ -248,11 +248,11 @@ int vhost_init_device_iotlb(struct vhost_dev *d, bool enabled);
 	} while (0)
 
 enum {
-	VHOST_FEATURES = (1ULL << VIRTIO_F_NOTIFY_ON_EMPTY) |
-			 (1ULL << VIRTIO_RING_F_INDIRECT_DESC) |
-			 (1ULL << VIRTIO_RING_F_EVENT_IDX) |
-			 (1ULL << VHOST_F_LOG_ALL) |
-			 (1ULL << VIRTIO_F_ANY_LAYOUT) |
+	VHOST_FEATURES = /* (1ULL << VIRTIO_F_NOTIFY_ON_EMPTY) | */
+			 /* (1ULL << VIRTIO_RING_F_INDIRECT_DESC) | */
+			 /* (1ULL << VIRTIO_RING_F_EVENT_IDX) | */
+			 /* (1ULL << VHOST_F_LOG_ALL) | */
+			 /* (1ULL << VIRTIO_F_ANY_LAYOUT) | */
 			 (1ULL << VIRTIO_F_VERSION_1)
 };
 
-- 
2.18.1

Re: vhost changes (batched) in linux-next after 12/13 trigger random crashes in KVM guests after reboot

[Christian Borntraeger]

On 14.02.20 13:17, Eugenio Pérez wrote:
> Can you try the inlined patch over 52c36ce7f334 ("vhost: use batched version by default")? My intention is to check if
> "strange VHOST_SET_VRING_BASE" line appears. In previous tests, it appears very fast, but maybe it takes some time for
> it to appear, or it does not appear anymore.

  LD [M]  drivers/vhost/vhost_vsock.o
  CC [M]  drivers/vhost/vhost.o
In file included from ./include/linux/printk.h:331,
                 from ./include/linux/kernel.h:15,
                 from ./include/linux/list.h:9,
                 from ./include/linux/wait.h:7,
                 from ./include/linux/eventfd.h:13,
                 from drivers/vhost/vhost.c:13:
drivers/vhost/vhost.c: In function ‘fetch_descs’:
drivers/vhost/vhost.c:2330:56: error: ‘struct vhost_virtqueue’ has no member named ‘first_desc’
 2330 |   vq, vq->last_avail_idx, vq->avail_idx, vq->ndescs, vq->first_desc);
      |                                                        ^~
./include/linux/dynamic_debug.h:125:15: note: in definition of macro ‘__dynamic_func_call’
  125 |   func(&id, ##__VA_ARGS__);  \
      |               ^~~~~~~~~~~
./include/linux/dynamic_debug.h:153:2: note: in expansion of macro ‘_dynamic_func_call’
  153 |  _dynamic_func_call(fmt, __dynamic_pr_debug,  \
      |  ^~~~~~~~~~~~~~~~~~
./include/linux/printk.h:335:2: note: in expansion of macro ‘dynamic_pr_debug’
  335 |  dynamic_pr_debug(fmt, ##__VA_ARGS__)
      |  ^~~~~~~~~~~~~~~~
drivers/vhost/vhost.c:2328:2: note: in expansion of macro ‘pr_debug’
 2328 |  pr_debug(
      |  ^~~~~~~~
make[2]: *** [scripts/Makefile.build:266: drivers/vhost/vhost.o] Error 1
make[1]: *** [scripts/Makefile.build:503: drivers/vhost] Error 2

Re: vhost changes (batched) in linux-next after 12/13 trigger random crashes in KVM guests after reboot

[Eugenio Pérez]

On Fri, 2020-02-14 at 13:22 +0100, Christian Borntraeger wrote:
> 
> On 14.02.20 13:17, Eugenio Pérez wrote:
> > Can you try the inlined patch over 52c36ce7f334 ("vhost: use batched version by default")? My intention is to check
> > if
> > "strange VHOST_SET_VRING_BASE" line appears. In previous tests, it appears very fast, but maybe it takes some time
> > for
> > it to appear, or it does not appear anymore.
> 
>   LD [M]  drivers/vhost/vhost_vsock.o
>   CC [M]  drivers/vhost/vhost.o
> In file included from ./include/linux/printk.h:331,
>                  from ./include/linux/kernel.h:15,
>                  from ./include/linux/list.h:9,
>                  from ./include/linux/wait.h:7,
>                  from ./include/linux/eventfd.h:13,
>                  from drivers/vhost/vhost.c:13:
> drivers/vhost/vhost.c: In function ‘fetch_descs’:
> drivers/vhost/vhost.c:2330:56: error: ‘struct vhost_virtqueue’ has no member named ‘first_desc’
>  2330 |   vq, vq->last_avail_idx, vq->avail_idx, vq->ndescs, vq->first_desc);
>       |                                                        ^~
> ./include/linux/dynamic_debug.h:125:15: note: in definition of macro ‘__dynamic_func_call’
>   125 |   func(&id, ##__VA_ARGS__);  \
>       |               ^~~~~~~~~~~
> ./include/linux/dynamic_debug.h:153:2: note: in expansion of macro ‘_dynamic_func_call’
>   153 |  _dynamic_func_call(fmt, __dynamic_pr_debug,  \
>       |  ^~~~~~~~~~~~~~~~~~
> ./include/linux/printk.h:335:2: note: in expansion of macro ‘dynamic_pr_debug’
>   335 |  dynamic_pr_debug(fmt, ##__VA_ARGS__)
>       |  ^~~~~~~~~~~~~~~~
> drivers/vhost/vhost.c:2328:2: note: in expansion of macro ‘pr_debug’
>  2328 |  pr_debug(
>       |  ^~~~~~~~
> make[2]: *** [scripts/Makefile.build:266: drivers/vhost/vhost.o] Error 1
> make[1]: *** [scripts/Makefile.build:503: drivers/vhost] Error 2
> 

Sorry about that. Here is the right patch.

From 5d7b5304c163910936382d46561fc43eb770aad2 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Eugenio=20P=C3=A9rez?= <eperezma@redhat.com>
Date: Fri, 14 Feb 2020 08:02:26 +0100
Subject: [PATCH] vhost: disable all features and trace last_avail_idx and
 ioctl calls

---
 drivers/vhost/net.c   | 20 +++++++++++++++++---
 drivers/vhost/vhost.c | 25 +++++++++++++++++++++++--
 drivers/vhost/vhost.h | 10 +++++-----
 3 files changed, 45 insertions(+), 10 deletions(-)

diff --git a/drivers/vhost/net.c b/drivers/vhost/net.c
index e158159671fa..e4d5f843f9c0 100644
--- a/drivers/vhost/net.c
+++ b/drivers/vhost/net.c
@@ -1505,10 +1505,13 @@ static long vhost_net_set_backend(struct vhost_net *n, unsigned index, int fd)
 
 	mutex_lock(&n->dev.mutex);
 	r = vhost_dev_check_owner(&n->dev);
-	if (r)
+	if (r) {
+		pr_debug("vhost_dev_check_owner index=%u fd=%d rc r=%d", index, fd, r);
 		goto err;
+	}
 
 	if (index >= VHOST_NET_VQ_MAX) {
+		pr_debug("vhost_dev_check_owner index=%u fd=%d MAX=%d", index, fd, VHOST_NET_VQ_MAX);
 		r = -ENOBUFS;
 		goto err;
 	}
@@ -1518,22 +1521,26 @@ static long vhost_net_set_backend(struct vhost_net *n, unsigned index, int fd)
 
 	/* Verify that ring has been setup correctly. */
 	if (!vhost_vq_access_ok(vq)) {
+		pr_debug("vhost_net_set_backend index=%u fd=%d !vhost_vq_access_ok", index, fd);
 		r = -EFAULT;
 		goto err_vq;
 	}
 	sock = get_socket(fd);
 	if (IS_ERR(sock)) {
 		r = PTR_ERR(sock);
+		pr_debug("vhost_net_set_backend index=%u fd=%d get_socket err r=%d", index, fd, r);
 		goto err_vq;
 	}
 
 	/* start polling new socket */
 	oldsock = vq->private_data;
 	if (sock != oldsock) {
+		pr_debug("sock=%p != oldsock=%p index=%u fd=%d vq=%p", sock, oldsock, index, fd, vq);
 		ubufs = vhost_net_ubuf_alloc(vq,
 					     sock && vhost_sock_zcopy(sock));
 		if (IS_ERR(ubufs)) {
 			r = PTR_ERR(ubufs);
+			pr_debug("ubufs index=%u fd=%d err r=%d vq=%p", index, fd, r, vq);
 			goto err_ubufs;
 		}
 
@@ -1541,11 +1548,15 @@ static long vhost_net_set_backend(struct vhost_net *n, unsigned index, int fd)
 		vq->private_data = sock;
 		vhost_net_buf_unproduce(nvq);
 		r = vhost_vq_init_access(vq);
-		if (r)
+		if (r) {
+			pr_debug("init_access index=%u fd=%d r=%d vq=%p", index, fd, r, vq);
 			goto err_used;
+		}
 		r = vhost_net_enable_vq(n, vq);
-		if (r)
+		if (r) {
+			pr_debug("enable_vq index=%u fd=%d r=%d vq=%p", index, fd, r, vq);
 			goto err_used;
+		}
 		if (index == VHOST_NET_VQ_RX)
 			nvq->rx_ring = get_tap_ptr_ring(fd);
 
@@ -1559,6 +1570,8 @@ static long vhost_net_set_backend(struct vhost_net *n, unsigned index, int fd)
 
 	mutex_unlock(&vq->mutex);
 
+	pr_debug("sock=%p", sock);
+
 	if (oldubufs) {
 		vhost_net_ubuf_put_wait_and_free(oldubufs);
 		mutex_lock(&vq->mutex);
@@ -1710,6 +1723,7 @@ static long vhost_net_ioctl(struct file *f, unsigned int ioctl,
 
 	switch (ioctl) {
 	case VHOST_NET_SET_BACKEND:
+		pr_debug("VHOST_NET_SET_BACKEND");
 		if (copy_from_user(&backend, argp, sizeof backend))
 			return -EFAULT;
 		return vhost_net_set_backend(n, backend.index, backend.fd);
diff --git a/drivers/vhost/vhost.c b/drivers/vhost/vhost.c
index 56c5253056ee..2e72bbeffac7 100644
--- a/drivers/vhost/vhost.c
+++ b/drivers/vhost/vhost.c
@@ -1640,15 +1640,30 @@ long vhost_vring_ioctl(struct vhost_dev *d, unsigned int ioctl, void __user *arg
 			r = -EINVAL;
 			break;
 		}
+
+		if (vq->last_avail_idx || vq->avail_idx) {
+			pr_debug(
+				"strange VHOST_SET_VRING_BASE [vq=%p][s.index=%u][s.num=%u]",
+				vq, s.index, s.num);
+			dump_stack();
+			r = 0;
+			break;
+		}
 		vq->last_avail_idx = s.num;
 		/* Forget the cached index value. */
 		vq->avail_idx = vq->last_avail_idx;
+		pr_debug(
+			"VHOST_SET_VRING_BASE [vq=%p][vq->last_avail_idx=%u][vq->avail_idx=%u][s.index=%u][s.num=%u]",
+			vq, vq->last_avail_idx, vq->avail_idx, s.index, s.num);
 		break;
 	case VHOST_GET_VRING_BASE:
 		s.index = idx;
 		s.num = vq->last_avail_idx;
 		if (copy_to_user(argp, &s, sizeof s))
 			r = -EFAULT;
+		pr_debug(
+			"VHOST_GET_VRING_BASE [vq=%p][vq->last_avail_idx=%u][vq->avail_idx=%u][s.index=%u][s.num=%u]",
+			vq, vq->last_avail_idx, vq->avail_idx, s.index, s.num);
 		break;
 	case VHOST_SET_VRING_KICK:
 		if (copy_from_user(&f, argp, sizeof f)) {
@@ -2233,8 +2248,8 @@ static int fetch_descs(struct vhost_virtqueue *vq)
 		vq->avail_idx = vhost16_to_cpu(vq, avail_idx);
 
 		if (unlikely((u16)(vq->avail_idx - last_avail_idx) > vq->num)) {
-			vq_err(vq, "Guest moved used index from %u to %u",
-				last_avail_idx, vq->avail_idx);
+			vq_err(vq, "Guest moved vq %p used index from %u to %u",
+				vq, last_avail_idx, vq->avail_idx);
 			return -EFAULT;
 		}
 
@@ -2310,6 +2325,9 @@ static int fetch_descs(struct vhost_virtqueue *vq)
 	BUG_ON(!(vq->used_flags & VRING_USED_F_NO_NOTIFY));
 
 	/* On success, increment avail index. */
+	pr_debug(
+		"[vq=%p][vq->last_avail_idx=%u][vq->avail_idx=%u][vq->ndescs=%d]",
+		vq, vq->last_avail_idx, vq->avail_idx, vq->ndescs);
 	vq->last_avail_idx++;
 
 	return 0;
@@ -2403,6 +2421,9 @@ EXPORT_SYMBOL_GPL(vhost_get_vq_desc);
 /* Reverse the effect of vhost_get_vq_desc. Useful for error handling. */
 void vhost_discard_vq_desc(struct vhost_virtqueue *vq, int n)
 {
+	pr_debug(
+		"DISCARD [vq=%p][vq->last_avail_idx=%u][vq->avail_idx=%u][n=%d]",
+		vq, vq->last_avail_idx, vq->avail_idx, n);
 	vq->last_avail_idx -= n;
 }
 EXPORT_SYMBOL_GPL(vhost_discard_vq_desc);
diff --git a/drivers/vhost/vhost.h b/drivers/vhost/vhost.h
index a0bcf8bffa43..2ce2d3a97c31 100644
--- a/drivers/vhost/vhost.h
+++ b/drivers/vhost/vhost.h
@@ -248,11 +248,11 @@ int vhost_init_device_iotlb(struct vhost_dev *d, bool enabled);
 	} while (0)
 
 enum {
-	VHOST_FEATURES = (1ULL << VIRTIO_F_NOTIFY_ON_EMPTY) |
-			 (1ULL << VIRTIO_RING_F_INDIRECT_DESC) |
-			 (1ULL << VIRTIO_RING_F_EVENT_IDX) |
-			 (1ULL << VHOST_F_LOG_ALL) |
-			 (1ULL << VIRTIO_F_ANY_LAYOUT) |
+	VHOST_FEATURES = /* (1ULL << VIRTIO_F_NOTIFY_ON_EMPTY) | */
+			 /* (1ULL << VIRTIO_RING_F_INDIRECT_DESC) | */
+			 /* (1ULL << VIRTIO_RING_F_EVENT_IDX) | */
+			 /* (1ULL << VHOST_F_LOG_ALL) | */
+			 /* (1ULL << VIRTIO_F_ANY_LAYOUT) | */
 			 (1ULL << VIRTIO_F_VERSION_1)
 };
 
-- 
2.18.1

Re: vhost changes (batched) in linux-next after 12/13 trigger random crashes in KVM guests after reboot

[Christian Borntraeger]

On 14.02.20 13:26, Eugenio Pérez wrote:
> On Fri, 2020-02-14 at 13:22 +0100, Christian Borntraeger wrote:
>>
>> On 14.02.20 13:17, Eugenio Pérez wrote:
>>> Can you try the inlined patch over 52c36ce7f334 ("vhost: use batched version by default")? My intention is to check
>>> if
>>> "strange VHOST_SET_VRING_BASE" line appears. In previous tests, it appears very fast, but maybe it takes some time
>>> for
>>> it to appear, or it does not appear anymore.


yep it does:

[   67.801012] [1917] vhost:vhost_vring_ioctl:1655: VHOST_SET_VRING_BASE [vq=0000000088199421][vq->last_avail_idx=0][vq->avail_idx=0][s.index=0][s.num=0]
[   67.801018] [1917] vhost:vhost_vring_ioctl:1655: VHOST_SET_VRING_BASE [vq=00000000175f11ec][vq->last_avail_idx=0][vq->avail_idx=0][s.index=1][s.num=0]
[   67.801026] [1917] vhost_net:vhost_net_ioctl:1726: VHOST_NET_SET_BACKEND
[   67.801028] [1917] vhost_net:vhost_net_set_backend:1538: sock=0000000082d8d291 != oldsock=000000001ae027fd index=0 fd=39 vq=0000000088199421
[   67.801032] [1917] vhost_net:vhost_net_set_backend:1573: sock=0000000082d8d291
[   67.801033] [1917] vhost_net:vhost_net_ioctl:1726: VHOST_NET_SET_BACKEND
[   67.801034] [1917] vhost_net:vhost_net_set_backend:1538: sock=0000000082d8d291 != oldsock=000000001ae027fd index=1 fd=39 vq=00000000175f11ec
[   67.801035] [1917] vhost_net:vhost_net_set_backend:1573: sock=0000000082d8d291
[   67.801037] [1915] vhost:vhost_discard_vq_desc:2424: DISCARD [vq=0000000088199421][vq->last_avail_idx=0][vq->avail_idx=0][n=0]
[   68.648803] [1915] vhost:fetch_descs:2328: [vq=0000000088199421][vq->last_avail_idx=0][vq->avail_idx=256][vq->ndescs=1]
[   68.648810] [1915] vhost:fetch_descs:2328: [vq=0000000088199421][vq->last_avail_idx=1][vq->avail_idx=256][vq->ndescs=1]
[   68.648812] [1915] vhost:fetch_descs:2328: [vq=0000000088199421][vq->last_avail_idx=2][vq->avail_idx=256][vq->ndescs=1]
[   68.648815] [1915] vhost:fetch_descs:2328: [vq=0000000088199421][vq->last_avail_idx=3][vq->avail_idx=256][vq->ndescs=1]
[   68.648817] [1915] vhost:fetch_descs:2328: [vq=0000000088199421][vq->last_avail_idx=4][vq->avail_idx=256][vq->ndescs=1]
[   68.648818] [1915] vhost:fetch_descs:2328: [vq=0000000088199421][vq->last_avail_idx=5][vq->avail_idx=256][vq->ndescs=1]
[   68.648820] [1915] vhost:fetch_descs:2328: [vq=0000000088199421][vq->last_avail_idx=6][vq->avail_idx=256][vq->ndescs=1]
[   68.648822] [1915] vhost:fetch_descs:2328: [vq=0000000088199421][vq->last_avail_idx=7][vq->avail_idx=256][vq->ndescs=1]
[   68.648824] [1915] vhost:fetch_descs:2328: [vq=0000000088199421][vq->last_avail_idx=8][vq->avail_idx=256][vq->ndescs=1]
[   68.648826] [1915] vhost:fetch_descs:2328: [vq=0000000088199421][vq->last_avail_idx=9][vq->avail_idx=256][vq->ndescs=1]
[   68.648828] [1915] vhost:fetch_descs:2328: [vq=0000000088199421][vq->last_avail_idx=10][vq->avail_idx=256][vq->ndescs=1]
[   68.648829] [1915] vhost:fetch_descs:2328: [vq=0000000088199421][vq->last_avail_idx=11][vq->avail_idx=256][vq->ndescs=1]
[   68.648831] [1915] vhost:fetch_descs:2328: [vq=0000000088199421][vq->last_avail_idx=12][vq->avail_idx=256][vq->ndescs=1]
[   68.648832] [1915] vhost:fetch_descs:2328: [vq=0000000088199421][vq->last_avail_idx=13][vq->avail_idx=256][vq->ndescs=1]
[   68.648833] [1915] vhost:fetch_descs:2328: [vq=0000000088199421][vq->last_avail_idx=14][vq->avail_idx=256][vq->ndescs=1]
[   68.670292] [1915] vhost:fetch_descs:2328: [vq=00000000175f11ec][vq->last_avail_idx=0][vq->avail_idx=1][vq->ndescs=1]
[   68.687187] [1915] vhost:fetch_descs:2328: [vq=00000000175f11ec][vq->last_avail_idx=1][vq->avail_idx=2][vq->ndescs=1]
[   68.687623] [1915] vhost:fetch_descs:2328: [vq=0000000088199421][vq->last_avail_idx=15][vq->avail_idx=256][vq->ndescs=1]
[   68.687641] [1915] vhost:fetch_descs:2328: [vq=00000000175f11ec][vq->last_avail_idx=2][vq->avail_idx=4][vq->ndescs=1]
[   68.687642] [1915] vhost:fetch_descs:2328: [vq=00000000175f11ec][vq->last_avail_idx=3][vq->avail_idx=4][vq->ndescs=1]
[   68.690274] [1915] vhost:fetch_descs:2328: [vq=0000000088199421][vq->last_avail_idx=16][vq->avail_idx=256][vq->ndescs=1]
[   68.690539] [1915] vhost:fetch_descs:2328: [vq=00000000175f11ec][vq->last_avail_idx=4][vq->avail_idx=5][vq->ndescs=1]
[   68.715379] [1915] vhost:fetch_descs:2328: [vq=0000000088199421][vq->last_avail_idx=17][vq->avail_idx=256][vq->ndescs=1]
[   68.800525] [1915] vhost:fetch_descs:2328: [vq=00000000175f11ec][vq->last_avail_idx=5][vq->avail_idx=6][vq->ndescs=1]
[   68.890537] [1915] vhost:fetch_descs:2328: [vq=00000000175f11ec][vq->last_avail_idx=6][vq->avail_idx=7][vq->ndescs=1]
[   68.900587] [1915] vhost:fetch_descs:2328: [vq=00000000175f11ec][vq->last_avail_idx=7][vq->avail_idx=8][vq->ndescs=1]
[   68.916837] [1915] vhost:fetch_descs:2328: [vq=00000000175f11ec][vq->last_avail_idx=8][vq->avail_idx=9][vq->ndescs=2]
[   68.928828] [1915] vhost:fetch_descs:2328: [vq=0000000088199421][vq->last_avail_idx=18][vq->avail_idx=256][vq->ndescs=1]
[   69.090540] [1915] vhost:fetch_descs:2328: [vq=00000000175f11ec][vq->last_avail_idx=9][vq->avail_idx=10][vq->ndescs=1]
[   69.119651] [1915] vhost:fetch_descs:2328: [vq=00000000175f11ec][vq->last_avail_idx=10][vq->avail_idx=11][vq->ndescs=2]
[   69.132325] [1915] vhost:fetch_descs:2328: [vq=0000000088199421][vq->last_avail_idx=19][vq->avail_idx=256][vq->ndescs=1]
[   69.323473] [1915] vhost:fetch_descs:2328: [vq=00000000175f11ec][vq->last_avail_idx=11][vq->avail_idx=12][vq->ndescs=2]
[   69.354557] [1915] vhost:fetch_descs:2328: [vq=0000000088199421][vq->last_avail_idx=20][vq->avail_idx=256][vq->ndescs=1]
[   69.442550] [1915] vhost:fetch_descs:2328: [vq=0000000088199421][vq->last_avail_idx=21][vq->avail_idx=256][vq->ndescs=1]
[   69.523593] [1915] vhost:fetch_descs:2328: [vq=00000000175f11ec][vq->last_avail_idx=12][vq->avail_idx=13][vq->ndescs=2]
[   69.557360] [1915] vhost:fetch_descs:2328: [vq=0000000088199421][vq->last_avail_idx=22][vq->avail_idx=256][vq->ndescs=1]
[   69.980634] [1915] vhost:fetch_descs:2328: [vq=00000000175f11ec][vq->last_avail_idx=13][vq->avail_idx=14][vq->ndescs=1]
[   69.981364] [1915] vhost:fetch_descs:2328: [vq=00000000175f11ec][vq->last_avail_idx=14][vq->avail_idx=15][vq->ndescs=1]
[   70.010545] [1915] vhost:fetch_descs:2328: [vq=00000000175f11ec][vq->last_avail_idx=15][vq->avail_idx=16][vq->ndescs=1]
[   70.161316] [1915] vhost:fetch_descs:2328: [vq=0000000088199421][vq->last_avail_idx=23][vq->avail_idx=256][vq->ndescs=1]
[   70.177640] [1915] vhost:fetch_descs:2328: [vq=0000000088199421][vq->last_avail_idx=24][vq->avail_idx=256][vq->ndescs=1]
[   70.280564] [1915] vhost:fetch_descs:2328: [vq=00000000175f11ec][vq->last_avail_idx=16][vq->avail_idx=17][vq->ndescs=1]
[   70.670327] [1915] vhost:fetch_descs:2328: [vq=00000000175f11ec][vq->last_avail_idx=17][vq->avail_idx=18][vq->ndescs=1]
[   70.932887] [1915] vhost:fetch_descs:2328: [vq=00000000175f11ec][vq->last_avail_idx=18][vq->avail_idx=19][vq->ndescs=2]
[   70.940587] [1915] vhost:fetch_descs:2328: [vq=00000000175f11ec][vq->last_avail_idx=19][vq->avail_idx=20][vq->ndescs=1]
[   70.947598] [1915] vhost:fetch_descs:2328: [vq=0000000088199421][vq->last_avail_idx=25][vq->avail_idx=256][vq->ndescs=1]
[   71.070388] [1915] vhost:fetch_descs:2328: [vq=00000000175f11ec][vq->last_avail_idx=20][vq->avail_idx=21][vq->ndescs=1]
[   71.070770] [1915] vhost:fetch_descs:2328: [vq=0000000088199421][vq->last_avail_idx=26][vq->avail_idx=256][vq->ndescs=1]
[   71.070805] [1915] vhost:fetch_descs:2328: [vq=00000000175f11ec][vq->last_avail_idx=21][vq->avail_idx=22][vq->ndescs=1]
[   71.070977] [1915] vhost:fetch_descs:2328: [vq=0000000088199421][vq->last_avail_idx=27][vq->avail_idx=256][vq->ndescs=1]
[   71.071049] [1915] vhost:fetch_descs:2328: [vq=00000000175f11ec][vq->last_avail_idx=22][vq->avail_idx=23][vq->ndescs=1]
[   71.071195] [1915] vhost:fetch_descs:2328: [vq=0000000088199421][vq->last_avail_idx=28][vq->avail_idx=256][vq->ndescs=1]
[   71.071243] [1915] vhost:fetch_descs:2328: [vq=00000000175f11ec][vq->last_avail_idx=23][vq->avail_idx=24][vq->ndescs=1]
[   71.071386] [1915] vhost:fetch_descs:2328: [vq=0000000088199421][vq->last_avail_idx=29][vq->avail_idx=256][vq->ndescs=1]
[   71.071433] [1915] vhost:fetch_descs:2328: [vq=00000000175f11ec][vq->last_avail_idx=24][vq->avail_idx=25][vq->ndescs=1]
[   71.071575] [1915] vhost:fetch_descs:2328: [vq=0000000088199421][vq->last_avail_idx=30][vq->avail_idx=256][vq->ndescs=1]
[   71.071611] [1915] vhost:fetch_descs:2328: [vq=00000000175f11ec][vq->last_avail_idx=25][vq->avail_idx=26][vq->ndescs=1]
[   71.071747] [1915] vhost:fetch_descs:2328: [vq=0000000088199421][vq->last_avail_idx=31][vq->avail_idx=256][vq->ndescs=1]
[   71.071789] [1915] vhost:fetch_descs:2328: [vq=00000000175f11ec][vq->last_avail_idx=26][vq->avail_idx=27][vq->ndescs=1]
[   71.071923] [1915] vhost:fetch_descs:2328: [vq=0000000088199421][vq->last_avail_idx=32][vq->avail_idx=256][vq->ndescs=1]
[   71.071960] [1915] vhost:fetch_descs:2328: [vq=00000000175f11ec][vq->last_avail_idx=27][vq->avail_idx=28][vq->ndescs=1]
[   71.072096] [1915] vhost:fetch_descs:2328: [vq=0000000088199421][vq->last_avail_idx=33][vq->avail_idx=256][vq->ndescs=1]
[   71.072128] [1915] vhost:fetch_descs:2328: [vq=00000000175f11ec][vq->last_avail_idx=28][vq->avail_idx=29][vq->ndescs=1]
[   71.072267] [1915] vhost:fetch_descs:2328: [vq=0000000088199421][vq->last_avail_idx=34][vq->avail_idx=256][vq->ndescs=1]
[   71.072300] [1915] vhost:fetch_descs:2328: [vq=00000000175f11ec][vq->last_avail_idx=29][vq->avail_idx=30][vq->ndescs=1]
[   71.072432] [1915] vhost:fetch_descs:2328: [vq=0000000088199421][vq->last_avail_idx=35][vq->avail_idx=256][vq->ndescs=1]
[   71.072463] [1915] vhost:fetch_descs:2328: [vq=00000000175f11ec][vq->last_avail_idx=30][vq->avail_idx=31][vq->ndescs=1]
[   71.072596] [1915] vhost:fetch_descs:2328: [vq=0000000088199421][vq->last_avail_idx=36][vq->avail_idx=256][vq->ndescs=1]
[   71.072630] [1915] vhost:fetch_descs:2328: [vq=00000000175f11ec][vq->last_avail_idx=31][vq->avail_idx=32][vq->ndescs=1]
[   71.072759] [1915] vhost:fetch_descs:2328: [vq=0000000088199421][vq->last_avail_idx=37][vq->avail_idx=256][vq->ndescs=1]
[   71.072791] [1915] vhost:fetch_descs:2328: [vq=00000000175f11ec][vq->last_avail_idx=32][vq->avail_idx=33][vq->ndescs=1]
[   71.072933] [1915] vhost:fetch_descs:2328: [vq=0000000088199421][vq->last_avail_idx=38][vq->avail_idx=256][vq->ndescs=1]
[   71.073054] [1915] vhost:fetch_descs:2328: [vq=00000000175f11ec][vq->last_avail_idx=33][vq->avail_idx=34][vq->ndescs=1]
[   71.073193] [1915] vhost:fetch_descs:2328: [vq=0000000088199421][vq->last_avail_idx=39][vq->avail_idx=256][vq->ndescs=1]
[   71.073247] [1915] vhost:fetch_descs:2328: [vq=00000000175f11ec][vq->last_avail_idx=34][vq->avail_idx=35][vq->ndescs=1]
[   71.073383] [1915] vhost:fetch_descs:2328: [vq=0000000088199421][vq->last_avail_idx=40][vq->avail_idx=256][vq->ndescs=1]
[   71.073434] [1915] vhost:fetch_descs:2328: [vq=00000000175f11ec][vq->last_avail_idx=35][vq->avail_idx=36][vq->ndescs=1]
[   71.073571] [1915] vhost:fetch_descs:2328: [vq=0000000088199421][vq->last_avail_idx=41][vq->avail_idx=256][vq->ndescs=1]
[   71.073627] [1915] vhost:fetch_descs:2328: [vq=00000000175f11ec][vq->last_avail_idx=36][vq->avail_idx=37][vq->ndescs=1]
[   71.073762] [1915] vhost:fetch_descs:2328: [vq=0000000088199421][vq->last_avail_idx=42][vq->avail_idx=256][vq->ndescs=1]
[   71.073813] [1915] vhost:fetch_descs:2328: [vq=00000000175f11ec][vq->last_avail_idx=37][vq->avail_idx=38][vq->ndescs=1]
[   71.073948] [1915] vhost:fetch_descs:2328: [vq=0000000088199421][vq->last_avail_idx=43][vq->avail_idx=256][vq->ndescs=1]
[   71.073998] [1915] vhost:fetch_descs:2328: [vq=00000000175f11ec][vq->last_avail_idx=38][vq->avail_idx=39][vq->ndescs=1]
[   71.074136] [1915] vhost:fetch_descs:2328: [vq=0000000088199421][vq->last_avail_idx=44][vq->avail_idx=256][vq->ndescs=1]
[   71.074186] [1915] vhost:fetch_descs:2328: [vq=00000000175f11ec][vq->last_avail_idx=39][vq->avail_idx=40][vq->ndescs=1]
[   71.074320] [1915] vhost:fetch_descs:2328: [vq=0000000088199421][vq->last_avail_idx=45][vq->avail_idx=256][vq->ndescs=1]
[   71.074370] [1915] vhost:fetch_descs:2328: [vq=00000000175f11ec][vq->last_avail_idx=40][vq->avail_idx=41][vq->ndescs=1]
[   71.074503] [1915] vhost:fetch_descs:2328: [vq=0000000088199421][vq->last_avail_idx=46][vq->avail_idx=256][vq->ndescs=1]
[   72.344493] [1915] vhost:fetch_descs:2328: [vq=0000000088199421][vq->last_avail_idx=47][vq->avail_idx=256][vq->ndescs=1]
[   72.553413] [1915] vhost:fetch_descs:2328: [vq=0000000088199421][vq->last_avail_idx=48][vq->avail_idx=256][vq->ndescs=1]
[   73.522174] [1915] vhost:fetch_descs:2328: [vq=0000000088199421][vq->last_avail_idx=49][vq->avail_idx=256][vq->ndescs=1]
[   73.705202] [1915] vhost:fetch_descs:2328: [vq=0000000088199421][vq->last_avail_idx=50][vq->avail_idx=256][vq->ndescs=1]
[   73.705239] [1915] vhost:fetch_descs:2328: [vq=00000000175f11ec][vq->last_avail_idx=41][vq->avail_idx=42][vq->ndescs=1]
[   73.994388] [1915] vhost:fetch_descs:2328: [vq=0000000088199421][vq->last_avail_idx=51][vq->avail_idx=256][vq->ndescs=1]
[   74.208443] [1915] vhost:fetch_descs:2328: [vq=0000000088199421][vq->last_avail_idx=52][vq->avail_idx=256][vq->ndescs=1]
[   74.433345] [1915] vhost:fetch_descs:2328: [vq=0000000088199421][vq->last_avail_idx=53][vq->avail_idx=256][vq->ndescs=1]
[   74.594756] [1910] vhost_net:vhost_net_ioctl:1726: VHOST_NET_SET_BACKEND
[   74.594761] [1910] vhost_net:vhost_net_set_backend:1538: sock=000000001ae027fd != oldsock=0000000082d8d291 index=0 fd=-1 vq=0000000088199421
[   74.594762] [1910] vhost_net:vhost_net_set_backend:1573: sock=000000001ae027fd
[   74.594803] [1910] vhost_net:vhost_net_ioctl:1726: VHOST_NET_SET_BACKEND
[   74.594804] [1910] vhost_net:vhost_net_set_backend:1538: sock=000000001ae027fd != oldsock=0000000082d8d291 index=1 fd=-1 vq=00000000175f11ec
[   74.594805] [1910] vhost_net:vhost_net_set_backend:1573: sock=000000001ae027fd
[   74.594847] [1910] vhost:vhost_vring_ioctl:1664: VHOST_GET_VRING_BASE [vq=0000000088199421][vq->last_avail_idx=54][vq->avail_idx=256][s.index=0][s.num=54]
[   74.594850] [1910] vhost:vhost_vring_ioctl:1664: VHOST_GET_VRING_BASE [vq=00000000175f11ec][vq->last_avail_idx=42][vq->avail_idx=42][s.index=1][s.num=42]
[   77.003191] [1918] vhost:vhost_vring_ioctl:1645: strange VHOST_SET_VRING_BASE [vq=0000000088199421][s.index=0][s.num=0]
[   77.003194] CPU: 62 PID: 1918 Comm: CPU 1/KVM Not tainted 5.5.0+ #22
[   77.003197] Hardware name: IBM 3906 M04 704 (LPAR)
[   77.003198] Call Trace:
[   77.003207]  [<0000000b8d93c132>] show_stack+0x8a/0xd0 
[   77.003211]  [<0000000b8e32e72a>] dump_stack+0x8a/0xb8 
[   77.003224]  [<000003ff803567ae>] vhost_vring_ioctl+0x6fe/0x858 [vhost] 
[   77.003228]  [<000003ff8036c670>] vhost_net_ioctl+0x510/0x570 [vhost_net] 
[   77.003234]  [<0000000b8dbecdd8>] do_vfs_ioctl+0x430/0x6f8 
[   77.003235]  [<0000000b8dbed124>] ksys_ioctl+0x84/0xb0 
[   77.003237]  [<0000000b8dbed1ba>] __s390x_sys_ioctl+0x2a/0x38 
[   77.003240]  [<0000000b8e34ff72>] system_call+0x2a6/0x2c8 
[   77.003261] [1918] vhost:vhost_vring_ioctl:1645: strange VHOST_SET_VRING_BASE [vq=00000000175f11ec][s.index=1][s.num=0]
[   77.003262] CPU: 62 PID: 1918 Comm: CPU 1/KVM Not tainted 5.5.0+ #22
[   77.003263] Hardware name: IBM 3906 M04 704 (LPAR)
[   77.003264] Call Trace:
[   77.003266]  [<0000000b8d93c132>] show_stack+0x8a/0xd0 
[   77.003267]  [<0000000b8e32e72a>] dump_stack+0x8a/0xb8 
[   77.003270]  [<000003ff803567ae>] vhost_vring_ioctl+0x6fe/0x858 [vhost] 
[   77.003271]  [<000003ff8036c670>] vhost_net_ioctl+0x510/0x570 [vhost_net] 
[   77.003273]  [<0000000b8dbecdd8>] do_vfs_ioctl+0x430/0x6f8 
[   77.003274]  [<0000000b8dbed124>] ksys_ioctl+0x84/0xb0 
[   77.003276]  [<0000000b8dbed1ba>] __s390x_sys_ioctl+0x2a/0x38 
[   77.003277]  [<0000000b8e34ff72>] system_call+0x2a6/0x2c8 
[   77.003297] [1918] vhost_net:vhost_net_ioctl:1726: VHOST_NET_SET_BACKEND
[   77.003300] [1918] vhost_net:vhost_net_set_backend:1538: sock=0000000082d8d291 != oldsock=000000001ae027fd index=0 fd=39 vq=0000000088199421
[   77.003304] [1918] vhost_net:vhost_net_set_backend:1573: sock=0000000082d8d291
[   77.003305] [1918] vhost_net:vhost_net_ioctl:1726: VHOST_NET_SET_BACKEND
[   77.003306] [1918] vhost_net:vhost_net_set_backend:1538: sock=0000000082d8d291 != oldsock=000000001ae027fd index=1 fd=39 vq=00000000175f11ec
[   77.003308] [1915] vhost:fetch_descs:2328: [vq=0000000088199421][vq->last_avail_idx=54][vq->avail_idx=256][vq->ndescs=1]
[   77.003308] [1918] vhost_net:vhost_net_set_backend:1573: sock=0000000082d8d291
[   77.003310] [1915] vhost_net:get_rx_bufs:1061: unexpected descriptor format for RX: out 0, in 0
[   77.003312] [1915] vhost:vhost_discard_vq_desc:2424: DISCARD [vq=0000000088199421][vq->last_avail_idx=55][vq->avail_idx=256][n=0]

Re: vhost changes (batched) in linux-next after 12/13 trigger random crashes in KVM guests after reboot

[Eugenio Pérez]

Hi Christian.

Sorry for the late response. Could we try this one over eccb852f1fe6bede630e2e4f1a121a81e34354ab, and see if you still
can reproduce the bug?

Apart from that, could you print me the backtrace when qemu calls vhost_kernel_set_vring_base and
vhost_kernel_get_vring_base functions?

Thank you very much!

diff --git a/drivers/vhost/net.c b/drivers/vhost/net.c
index e158159671fa..a1a4239512bb 100644
--- a/drivers/vhost/net.c
+++ b/drivers/vhost/net.c
@@ -1505,10 +1505,13 @@ static long vhost_net_set_backend(struct vhost_net *n, unsigned index, int fd)
 
 	mutex_lock(&n->dev.mutex);
 	r = vhost_dev_check_owner(&n->dev);
-	if (r)
+	if (r) {
+		pr_debug("vhost_dev_check_owner index=%u fd=%d rc r=%d", index, fd, r);
 		goto err;
+	}
 
 	if (index >= VHOST_NET_VQ_MAX) {
+		pr_debug("vhost_dev_check_owner index=%u fd=%d MAX=%d", index, fd, VHOST_NET_VQ_MAX);
 		r = -ENOBUFS;
 		goto err;
 	}
@@ -1518,22 +1521,26 @@ static long vhost_net_set_backend(struct vhost_net *n, unsigned index, int fd)
 
 	/* Verify that ring has been setup correctly. */
 	if (!vhost_vq_access_ok(vq)) {
+		pr_debug("vhost_net_set_backend index=%u fd=%d !vhost_vq_access_ok", index, fd);
 		r = -EFAULT;
 		goto err_vq;
 	}
 	sock = get_socket(fd);
 	if (IS_ERR(sock)) {
 		r = PTR_ERR(sock);
+		pr_debug("vhost_net_set_backend index=%u fd=%d get_socket err r=%d", index, fd, r);
 		goto err_vq;
 	}
 
 	/* start polling new socket */
 	oldsock = vq->private_data;
 	if (sock != oldsock) {
+		pr_debug("sock=%p != oldsock=%p index=%u fd=%d vq=%p", sock, oldsock, index, fd, vq);
 		ubufs = vhost_net_ubuf_alloc(vq,
 					     sock && vhost_sock_zcopy(sock));
 		if (IS_ERR(ubufs)) {
 			r = PTR_ERR(ubufs);
+			pr_debug("ubufs index=%u fd=%d err r=%d vq=%p", index, fd, r, vq);
 			goto err_ubufs;
 		}
 
@@ -1541,11 +1548,15 @@ static long vhost_net_set_backend(struct vhost_net *n, unsigned index, int fd)
 		vq->private_data = sock;
 		vhost_net_buf_unproduce(nvq);
 		r = vhost_vq_init_access(vq);
-		if (r)
+		if (r) {
+			pr_debug("init_access index=%u fd=%d r=%d vq=%p", index, fd, r, vq);
 			goto err_used;
+		}
 		r = vhost_net_enable_vq(n, vq);
-		if (r)
+		if (r) {
+			pr_debug("enable_vq index=%u fd=%d r=%d vq=%p", index, fd, r, vq);
 			goto err_used;
+		}
 		if (index == VHOST_NET_VQ_RX)
 			nvq->rx_ring = get_tap_ptr_ring(fd);
 
@@ -1559,6 +1570,8 @@ static long vhost_net_set_backend(struct vhost_net *n, unsigned index, int fd)
 
 	mutex_unlock(&vq->mutex);
 
+	pr_debug("sock=%p", sock);
+
 	if (oldubufs) {
 		vhost_net_ubuf_put_wait_and_free(oldubufs);
 		mutex_lock(&vq->mutex);
@@ -1712,6 +1725,9 @@ static long vhost_net_ioctl(struct file *f, unsigned int ioctl,
 	case VHOST_NET_SET_BACKEND:
 		if (copy_from_user(&backend, argp, sizeof backend))
 			return -EFAULT;
+		pr_debug("VHOST_NET_SET_BACKEND [b.index=%u][b.fd=%d]",
+			 backend.index, backend.fd);
+		dump_stack();
 		return vhost_net_set_backend(n, backend.index, backend.fd);
 	case VHOST_GET_FEATURES:
 		features = VHOST_NET_FEATURES;
diff --git a/drivers/vhost/vhost.c b/drivers/vhost/vhost.c
index b5a51b1f2e79..9dd0bcae0b22 100644
--- a/drivers/vhost/vhost.c
+++ b/drivers/vhost/vhost.c
@@ -372,6 +372,11 @@ static int vhost_worker(void *data)
 	return 0;
 }
 
+static int vhost_vq_num_batch_descs(struct vhost_virtqueue *vq)
+{
+	return vq->max_descs - UIO_MAXIOV;
+}
+
 static void vhost_vq_free_iovecs(struct vhost_virtqueue *vq)
 {
 	kfree(vq->descs);
@@ -394,7 +399,9 @@ static long vhost_dev_alloc_iovecs(struct vhost_dev *dev)
 	for (i = 0; i < dev->nvqs; ++i) {
 		vq = dev->vqs[i];
 		vq->max_descs = dev->iov_limit;
-		vq->batch_descs = dev->iov_limit - UIO_MAXIOV;
+		if (vhost_vq_num_batch_descs(vq) < 0) {
+			return -EINVAL;
+		}
 		vq->descs = kmalloc_array(vq->max_descs,
 					  sizeof(*vq->descs),
 					  GFP_KERNEL);
@@ -1642,15 +1649,27 @@ long vhost_vring_ioctl(struct vhost_dev *d, unsigned int ioctl, void __user *arg
 			r = -EINVAL;
 			break;
 		}
+
+		pr_debug(
+			"VHOST_SET_VRING_BASE [vq=%p][s.index=%u][s.num=%u][vq->avail_idx=%d][vq->last_avail_idx=%d][vq-
>ndescs=%d][vq->first_desc=%d]",
+			vq, s.index, s.num, vq->avail_idx, vq->last_avail_idx,
+			vq->ndescs, vq->first_desc);
+		dump_stack();
 		vq->last_avail_idx = s.num;
 		/* Forget the cached index value. */
 		vq->avail_idx = vq->last_avail_idx;
+		vq->ndescs = vq->first_desc = 0;
 		break;
 	case VHOST_GET_VRING_BASE:
 		s.index = idx;
 		s.num = vq->last_avail_idx;
 		if (copy_to_user(argp, &s, sizeof s))
 			r = -EFAULT;
+		pr_debug(
+			"VHOST_GET_VRING_BASE [vq=%p][s.index=%u][s.num=%u][vq->avail_idx=%d][vq->last_avail_idx=%d][vq-
>ndescs=%d][vq->first_desc=%d]",
+			vq, s.index, s.num, vq->avail_idx, vq->last_avail_idx,
+			vq->ndescs, vq->first_desc);
+		dump_stack();
 		break;
 	case VHOST_SET_VRING_KICK:
 		if (copy_from_user(&f, argp, sizeof f)) {
@@ -2239,8 +2258,8 @@ static int fetch_buf(struct vhost_virtqueue *vq)
 		vq->avail_idx = vhost16_to_cpu(vq, avail_idx);
 
 		if (unlikely((u16)(vq->avail_idx - last_avail_idx) > vq->num)) {
-			vq_err(vq, "Guest moved used index from %u to %u",
-				last_avail_idx, vq->avail_idx);
+			vq_err(vq, "Guest moved vq %p used index from %u to %u",
+				vq, last_avail_idx, vq->avail_idx);
 			return -EFAULT;
 		}
 
@@ -2316,6 +2335,9 @@ static int fetch_buf(struct vhost_virtqueue *vq)
 	BUG_ON(!(vq->used_flags & VRING_USED_F_NO_NOTIFY));
 
 	/* On success, increment avail index. */
+	pr_debug(
+		"[vq=%p][vq->last_avail_idx=%u][vq->avail_idx=%u][vq->ndescs=%d][vq->first_desc=%d]",
+		vq, vq->last_avail_idx, vq->avail_idx, vq->ndescs, vq->first_desc);
 	vq->last_avail_idx++;
 
 	return 0;
@@ -2333,7 +2355,7 @@ static int fetch_descs(struct vhost_virtqueue *vq)
 	if (vq->ndescs)
 		return 0;
 
-	while (!ret && vq->ndescs <= vq->batch_descs)
+	while (!ret && vq->ndescs <= vhost_vq_num_batch_descs(vq))
 		ret = fetch_buf(vq);
 
 	return vq->ndescs ? 0 : ret;
@@ -2432,6 +2454,9 @@ EXPORT_SYMBOL_GPL(vhost_get_vq_desc);
 /* Reverse the effect of vhost_get_vq_desc. Useful for error handling. */
 void vhost_discard_vq_desc(struct vhost_virtqueue *vq, int n)
 {
+	pr_debug(
+		"DISCARD [vq=%p][vq->last_avail_idx=%u][vq->avail_idx=%u][n=%d]",
+		vq, vq->last_avail_idx, vq->avail_idx, n);
 	vq->last_avail_idx -= n;
 }
 EXPORT_SYMBOL_GPL(vhost_discard_vq_desc);
diff --git a/drivers/vhost/vhost.h b/drivers/vhost/vhost.h
index 661088ae6dc7..e648b9b997d4 100644
--- a/drivers/vhost/vhost.h
+++ b/drivers/vhost/vhost.h
@@ -102,7 +102,6 @@ struct vhost_virtqueue {
 	int ndescs;
 	int first_desc;
 	int max_descs;
-	int batch_descs;
 
 	const struct vhost_umem_node *meta_iotlb[VHOST_NUM_ADDRS];
 	struct file *kick;

Re: vhost changes (batched) in linux-next after 12/13 trigger random crashes in KVM guests after reboot

[Christian Borntraeger]

On 27.03.20 12:08, Eugenio Pérez wrote:
> Hi Christian.
> 
> Sorry for the late response. Could we try this one over eccb852f1fe6bede630e2e4f1a121a81e34354ab, and see if you still
> can reproduce the bug?

To much time has passed and too many things have changed on that system.
I have trouble reproducing this with either
eccb852f1fe6bede630e2e4f1a121a81e34354ab or 52c36ce7f334.
I will try to reproduce this again :-/

> 
> Apart from that, could you print me the backtrace when qemu calls vhost_kernel_set_vring_base and
> vhost_kernel_get_vring_base functions?
> 
> Thank you very much!
> 
> diff --git a/drivers/vhost/net.c b/drivers/vhost/net.c
> index e158159671fa..a1a4239512bb 100644
> --- a/drivers/vhost/net.c
> +++ b/drivers/vhost/net.c
> @@ -1505,10 +1505,13 @@ static long vhost_net_set_backend(struct vhost_net *n, unsigned index, int fd)
>  
>  	mutex_lock(&n->dev.mutex);
>  	r = vhost_dev_check_owner(&n->dev);
> -	if (r)
> +	if (r) {
> +		pr_debug("vhost_dev_check_owner index=%u fd=%d rc r=%d", index, fd, r);
>  		goto err;
> +	}
>  
>  	if (index >= VHOST_NET_VQ_MAX) {
> +		pr_debug("vhost_dev_check_owner index=%u fd=%d MAX=%d", index, fd, VHOST_NET_VQ_MAX);
>  		r = -ENOBUFS;
>  		goto err;
>  	}
> @@ -1518,22 +1521,26 @@ static long vhost_net_set_backend(struct vhost_net *n, unsigned index, int fd)
>  
>  	/* Verify that ring has been setup correctly. */
>  	if (!vhost_vq_access_ok(vq)) {
> +		pr_debug("vhost_net_set_backend index=%u fd=%d !vhost_vq_access_ok", index, fd);
>  		r = -EFAULT;
>  		goto err_vq;
>  	}
>  	sock = get_socket(fd);
>  	if (IS_ERR(sock)) {
>  		r = PTR_ERR(sock);
> +		pr_debug("vhost_net_set_backend index=%u fd=%d get_socket err r=%d", index, fd, r);
>  		goto err_vq;
>  	}
>  
>  	/* start polling new socket */
>  	oldsock = vq->private_data;
>  	if (sock != oldsock) {
> +		pr_debug("sock=%p != oldsock=%p index=%u fd=%d vq=%p", sock, oldsock, index, fd, vq);
>  		ubufs = vhost_net_ubuf_alloc(vq,
>  					     sock && vhost_sock_zcopy(sock));
>  		if (IS_ERR(ubufs)) {
>  			r = PTR_ERR(ubufs);
> +			pr_debug("ubufs index=%u fd=%d err r=%d vq=%p", index, fd, r, vq);
>  			goto err_ubufs;
>  		}
>  
> @@ -1541,11 +1548,15 @@ static long vhost_net_set_backend(struct vhost_net *n, unsigned index, int fd)
>  		vq->private_data = sock;
>  		vhost_net_buf_unproduce(nvq);
>  		r = vhost_vq_init_access(vq);
> -		if (r)
> +		if (r) {
> +			pr_debug("init_access index=%u fd=%d r=%d vq=%p", index, fd, r, vq);
>  			goto err_used;
> +		}
>  		r = vhost_net_enable_vq(n, vq);
> -		if (r)
> +		if (r) {
> +			pr_debug("enable_vq index=%u fd=%d r=%d vq=%p", index, fd, r, vq);
>  			goto err_used;
> +		}
>  		if (index == VHOST_NET_VQ_RX)
>  			nvq->rx_ring = get_tap_ptr_ring(fd);
>  
> @@ -1559,6 +1570,8 @@ static long vhost_net_set_backend(struct vhost_net *n, unsigned index, int fd)
>  
>  	mutex_unlock(&vq->mutex);
>  
> +	pr_debug("sock=%p", sock);
> +
>  	if (oldubufs) {
>  		vhost_net_ubuf_put_wait_and_free(oldubufs);
>  		mutex_lock(&vq->mutex);
> @@ -1712,6 +1725,9 @@ static long vhost_net_ioctl(struct file *f, unsigned int ioctl,
>  	case VHOST_NET_SET_BACKEND:
>  		if (copy_from_user(&backend, argp, sizeof backend))
>  			return -EFAULT;
> +		pr_debug("VHOST_NET_SET_BACKEND [b.index=%u][b.fd=%d]",
> +			 backend.index, backend.fd);
> +		dump_stack();
>  		return vhost_net_set_backend(n, backend.index, backend.fd);
>  	case VHOST_GET_FEATURES:
>  		features = VHOST_NET_FEATURES;
> diff --git a/drivers/vhost/vhost.c b/drivers/vhost/vhost.c
> index b5a51b1f2e79..9dd0bcae0b22 100644
> --- a/drivers/vhost/vhost.c
> +++ b/drivers/vhost/vhost.c
> @@ -372,6 +372,11 @@ static int vhost_worker(void *data)
>  	return 0;
>  }
>  
> +static int vhost_vq_num_batch_descs(struct vhost_virtqueue *vq)
> +{
> +	return vq->max_descs - UIO_MAXIOV;
> +}
> +
>  static void vhost_vq_free_iovecs(struct vhost_virtqueue *vq)
>  {
>  	kfree(vq->descs);
> @@ -394,7 +399,9 @@ static long vhost_dev_alloc_iovecs(struct vhost_dev *dev)
>  	for (i = 0; i < dev->nvqs; ++i) {
>  		vq = dev->vqs[i];
>  		vq->max_descs = dev->iov_limit;
> -		vq->batch_descs = dev->iov_limit - UIO_MAXIOV;
> +		if (vhost_vq_num_batch_descs(vq) < 0) {
> +			return -EINVAL;
> +		}
>  		vq->descs = kmalloc_array(vq->max_descs,
>  					  sizeof(*vq->descs),
>  					  GFP_KERNEL);
> @@ -1642,15 +1649,27 @@ long vhost_vring_ioctl(struct vhost_dev *d, unsigned int ioctl, void __user *arg
>  			r = -EINVAL;
>  			break;
>  		}
> +
> +		pr_debug(
> +			"VHOST_SET_VRING_BASE [vq=%p][s.index=%u][s.num=%u][vq->avail_idx=%d][vq->last_avail_idx=%d][vq-
>> ndescs=%d][vq->first_desc=%d]",
> +			vq, s.index, s.num, vq->avail_idx, vq->last_avail_idx,
> +			vq->ndescs, vq->first_desc);
> +		dump_stack();
>  		vq->last_avail_idx = s.num;
>  		/* Forget the cached index value. */
>  		vq->avail_idx = vq->last_avail_idx;
> +		vq->ndescs = vq->first_desc = 0;
>  		break;
>  	case VHOST_GET_VRING_BASE:
>  		s.index = idx;
>  		s.num = vq->last_avail_idx;
>  		if (copy_to_user(argp, &s, sizeof s))
>  			r = -EFAULT;
> +		pr_debug(
> +			"VHOST_GET_VRING_BASE [vq=%p][s.index=%u][s.num=%u][vq->avail_idx=%d][vq->last_avail_idx=%d][vq-
>> ndescs=%d][vq->first_desc=%d]",
> +			vq, s.index, s.num, vq->avail_idx, vq->last_avail_idx,
> +			vq->ndescs, vq->first_desc);
> +		dump_stack();
>  		break;
>  	case VHOST_SET_VRING_KICK:
>  		if (copy_from_user(&f, argp, sizeof f)) {
> @@ -2239,8 +2258,8 @@ static int fetch_buf(struct vhost_virtqueue *vq)
>  		vq->avail_idx = vhost16_to_cpu(vq, avail_idx);
>  
>  		if (unlikely((u16)(vq->avail_idx - last_avail_idx) > vq->num)) {
> -			vq_err(vq, "Guest moved used index from %u to %u",
> -				last_avail_idx, vq->avail_idx);
> +			vq_err(vq, "Guest moved vq %p used index from %u to %u",
> +				vq, last_avail_idx, vq->avail_idx);
>  			return -EFAULT;
>  		}
>  
> @@ -2316,6 +2335,9 @@ static int fetch_buf(struct vhost_virtqueue *vq)
>  	BUG_ON(!(vq->used_flags & VRING_USED_F_NO_NOTIFY));
>  
>  	/* On success, increment avail index. */
> +	pr_debug(
> +		"[vq=%p][vq->last_avail_idx=%u][vq->avail_idx=%u][vq->ndescs=%d][vq->first_desc=%d]",
> +		vq, vq->last_avail_idx, vq->avail_idx, vq->ndescs, vq->first_desc);
>  	vq->last_avail_idx++;
>  
>  	return 0;
> @@ -2333,7 +2355,7 @@ static int fetch_descs(struct vhost_virtqueue *vq)
>  	if (vq->ndescs)
>  		return 0;
>  
> -	while (!ret && vq->ndescs <= vq->batch_descs)
> +	while (!ret && vq->ndescs <= vhost_vq_num_batch_descs(vq))
>  		ret = fetch_buf(vq);
>  
>  	return vq->ndescs ? 0 : ret;
> @@ -2432,6 +2454,9 @@ EXPORT_SYMBOL_GPL(vhost_get_vq_desc);
>  /* Reverse the effect of vhost_get_vq_desc. Useful for error handling. */
>  void vhost_discard_vq_desc(struct vhost_virtqueue *vq, int n)
>  {
> +	pr_debug(
> +		"DISCARD [vq=%p][vq->last_avail_idx=%u][vq->avail_idx=%u][n=%d]",
> +		vq, vq->last_avail_idx, vq->avail_idx, n);
>  	vq->last_avail_idx -= n;
>  }
>  EXPORT_SYMBOL_GPL(vhost_discard_vq_desc);
> diff --git a/drivers/vhost/vhost.h b/drivers/vhost/vhost.h
> index 661088ae6dc7..e648b9b997d4 100644
> --- a/drivers/vhost/vhost.h
> +++ b/drivers/vhost/vhost.h
> @@ -102,7 +102,6 @@ struct vhost_virtqueue {
>  	int ndescs;
>  	int first_desc;
>  	int max_descs;
> -	int batch_descs;
>  
>  	const struct vhost_umem_node *meta_iotlb[VHOST_NUM_ADDRS];
>  	struct file *kick;
> 

Re: vhost changes (batched) in linux-next after 12/13 trigger random crashes in KVM guests after reboot

[Michael S. Tsirkin]

On Tue, Feb 11, 2020 at 02:04:54PM +0100, Eugenio Pérez wrote:
> On Mon, 2020-02-10 at 12:01 +0100, Christian Borntraeger wrote:
> > 
> > On 10.02.20 10:47, Eugenio Perez Martin wrote:
> > > Hi Christian.
> > > 
> > > I'm not able to reproduce the failure with eccb852f1fe6bede630e2e4f1a121a81e34354ab commit. Could you add more data?
> > > Your configuration (libvirt or qemu line), and host's dmesg output if any?
> > > 
> > > Thanks!
> > 
> > If it was not obvious, this is on s390x, a big endian system.
> > 
> 
> Hi Christian. Thank you very much for your fast responses.
> 
> Could you try this patch on top of eccb852f1fe6bede630e2e4f1a121a81e34354ab?
> 
> Thanks!
> 
> >From 71d0f9108a18aa894cc0c0c1c7efbad39f465a27 Mon Sep 17 00:00:00 2001
> From: =?UTF-8?q?Eugenio=20P=C3=A9rez?= <
> eperezma@redhat.com>
> Date: Tue, 11 Feb 2020 13:19:10 +0100
> Subject: [PATCH] vhost: fix return value of vhost_get_vq_desc
> 
> Before of the batch change, it was the chain's head. Need to keep that
> way or we will not be able to free a chain of descriptors.

I think it's cleaner to have all descriptors in the chain
have the same id.

> Fixes: eccb852f1fe6 ("vhost: batching fetches")
> ---
>  drivers/vhost/vhost.c | 3 +--
>  1 file changed, 1 insertion(+), 2 deletions(-)
> 
> diff --git a/drivers/vhost/vhost.c b/drivers/vhost/vhost.c
> index b5a51b1f2e79..fc422c3e5c08 100644
> --- a/drivers/vhost/vhost.c
> +++ b/drivers/vhost/vhost.c
> @@ -2409,12 +2409,11 @@ int vhost_get_vq_desc(struct vhost_virtqueue *vq,
>  			*out_num += ret;
>  		}
>  
> -		ret = desc->id;
> -
>  		if (!(desc->flags & VRING_DESC_F_NEXT))
>  			break;
>  	}
>  
> +	ret = vq->descs[vq->first_desc].id;
>  	vq->first_desc = i + 1;
>  
>  	return ret;
> -- 
> 2.18.1

Re: vhost changes (batched) in linux-next after 12/13 trigger random crashes in KVM guests after reboot

[Eugenio Perez Martin]

On Tue, Feb 11, 2020 at 2:58 PM Michael S. Tsirkin <mst@redhat.com> wrote:
>
> On Tue, Feb 11, 2020 at 02:04:54PM +0100, Eugenio Pérez wrote:
> > On Mon, 2020-02-10 at 12:01 +0100, Christian Borntraeger wrote:
> > >
> > > On 10.02.20 10:47, Eugenio Perez Martin wrote:
> > > > Hi Christian.
> > > >
> > > > I'm not able to reproduce the failure with eccb852f1fe6bede630e2e4f1a121a81e34354ab commit. Could you add more data?
> > > > Your configuration (libvirt or qemu line), and host's dmesg output if any?
> > > >
> > > > Thanks!
> > >
> > > If it was not obvious, this is on s390x, a big endian system.
> > >
> >
> > Hi Christian. Thank you very much for your fast responses.
> >
> > Could you try this patch on top of eccb852f1fe6bede630e2e4f1a121a81e34354ab?
> >
> > Thanks!
> >
> > >From 71d0f9108a18aa894cc0c0c1c7efbad39f465a27 Mon Sep 17 00:00:00 2001
> > From: =?UTF-8?q?Eugenio=20P=C3=A9rez?= <
> > eperezma@redhat.com>
> > Date: Tue, 11 Feb 2020 13:19:10 +0100
> > Subject: [PATCH] vhost: fix return value of vhost_get_vq_desc
> >
> > Before of the batch change, it was the chain's head. Need to keep that
> > way or we will not be able to free a chain of descriptors.
>
> I think it's cleaner to have all descriptors in the chain
> have the same id.
>

Hi Michael.

Not sure if I understood you correctly. All the descriptors in the
chain still have the same id, this just returns the first one instead
of update it in every loop iteration. Can left unapplied in the final
version though, is up to you.

> > Fixes: eccb852f1fe6 ("vhost: batching fetches")
> > ---
> >  drivers/vhost/vhost.c | 3 +--
> >  1 file changed, 1 insertion(+), 2 deletions(-)
> >
> > diff --git a/drivers/vhost/vhost.c b/drivers/vhost/vhost.c
> > index b5a51b1f2e79..fc422c3e5c08 100644
> > --- a/drivers/vhost/vhost.c
> > +++ b/drivers/vhost/vhost.c
> > @@ -2409,12 +2409,11 @@ int vhost_get_vq_desc(struct vhost_virtqueue *vq,
> >                       *out_num += ret;
> >               }
> >
> > -             ret = desc->id;
> > -
> >               if (!(desc->flags & VRING_DESC_F_NEXT))
> >                       break;
> >       }
> >
> > +     ret = vq->descs[vq->first_desc].id;
> >       vq->first_desc = i + 1;
> >
> >       return ret;
> > --
> > 2.18.1
>

Re: vhost changes (batched) in linux-next after 12/13 trigger random crashes in KVM guests after reboot

[Michael S. Tsirkin]

On Thu, Feb 06, 2020 at 03:22:39PM +0100, eperezma@redhat.com wrote:
> Hi Christian.
> 
> Could you try this patch on top of ("38ced0208491 vhost: use batched version by default")?
> 
> It will not solve your first random crash but it should help with the lost of network connectivity.
> 
> Please let me know how does it goes.
> 
> Thanks!
> 
> >From 99f0f543f3939dbe803988c9153a95616ccccacd Mon Sep 17 00:00:00 2001
> From: =?UTF-8?q?Eugenio=20P=C3=A9rez?= <eperezma@redhat.com>
> Date: Thu, 6 Feb 2020 15:13:42 +0100
> Subject: [PATCH] vhost: filter valid vhost descriptors flags
> MIME-Version: 1.0
> Content-Type: text/plain; charset=UTF-8
> Content-Transfer-Encoding: 8bit
> 
> Previous commit copy _NEXT flag, and it complains if a copied descriptor
> contains it.
> 
> Signed-off-by: Eugenio Pérez <eperezma@redhat.com>
> ---
>  drivers/vhost/vhost.c | 6 ++++--
>  1 file changed, 4 insertions(+), 2 deletions(-)
> 
> diff --git a/drivers/vhost/vhost.c b/drivers/vhost/vhost.c
> index 27ae5b4872a0..56c5253056ee 100644
> --- a/drivers/vhost/vhost.c
> +++ b/drivers/vhost/vhost.c
> @@ -2125,6 +2125,8 @@ static void pop_split_desc(struct vhost_virtqueue *vq)
>  	--vq->ndescs;
>  }
>  
> +#define VHOST_DESC_FLAGS (VRING_DESC_F_INDIRECT | VRING_DESC_F_WRITE | \
> +			  VRING_DESC_F_NEXT)
>  static int push_split_desc(struct vhost_virtqueue *vq, struct vring_desc *desc, u16 id)
>  {
>  	struct vhost_desc *h;
> @@ -2134,7 +2136,7 @@ static int push_split_desc(struct vhost_virtqueue *vq, struct vring_desc *desc,
>  	h = &vq->descs[vq->ndescs++];
>  	h->addr = vhost64_to_cpu(vq, desc->addr);
>  	h->len = vhost32_to_cpu(vq, desc->len);
> -	h->flags = vhost16_to_cpu(vq, desc->flags);
> +	h->flags = vhost16_to_cpu(vq, desc->flags) & VHOST_DESC_FLAGS;
>  	h->id = id;
>  
>  	return 0;



> @@ -2343,7 +2345,7 @@ int vhost_get_vq_desc(struct vhost_virtqueue *vq,
>  		struct vhost_desc *desc = &vq->descs[i];
>  		int access;
>  
> -		if (desc->flags & ~(VRING_DESC_F_INDIRECT | VRING_DESC_F_WRITE)) {
> +		if (desc->flags & ~VHOST_DESC_FLAGS) {
>  			vq_err(vq, "Unexpected flags: 0x%x at descriptor id 0x%x\n",
>  			       desc->flags, desc->id);
>  			ret = -EINVAL;
> -- 
> 2.18.1

Thanks for catching this!

Do we need the 1st chunk though?

It seems preferable to just muck with flags in 1 place, when we
validate them ...

> 
> On Wed, 2020-01-22 at 20:32 +0100, Christian Borntraeger wrote:
> > 
> > On 20.01.20 07:27, Michael S. Tsirkin wrote:
> > > On Tue, Jan 07, 2020 at 01:16:50PM +0100, Christian Borntraeger wrote:
> > > > On 07.01.20 12:55, Michael S. Tsirkin wrote:
> > > > 
> > > > > I pushed batched-v3 - same head but bisect should work now.
> > > > > 
> > > > 
> > > > With 
> > > > commit 38ced0208491103b50f1056f0d1c8f28e2e13d08 (HEAD)
> > > > Author:     Michael S. Tsirkin <mst@redhat.com>
> > > > AuthorDate: Wed Dec 11 12:19:26 2019 -0500
> > > > Commit:     Michael S. Tsirkin <mst@redhat.com>
> > > > CommitDate: Tue Jan 7 06:52:42 2020 -0500
> > > > 
> > > >     vhost: use batched version by default
> > > > 
> > > > 
> > > > I have exactly one successful ping and then the network inside the guest is broken (no packet
> > > > anymore).
> > > 
> > > Does anything appear in host's dmesg when this happens?
> > 
> > I think there was nothing, but I am not sure. I would need to redo the test if this is important to know.
> > 
> > > 
> > > > So you could consider this commit broken (but in a different way and also without any
> > > > guest reboot necessary).
> > > > 
> > > > 
> > > > bisect log:
> > > > git bisect start
> > > > # bad: [d2f6175f52062ee51ee69754a6925608213475d2] vhost: use vhost_desc instead of vhost_log
> > > > git bisect bad d2f6175f52062ee51ee69754a6925608213475d2
> > > > # good: [d1281e3a562ec6a08f944a876481dd043ba739b9] virtio-blk: remove VIRTIO_BLK_F_SCSI support
> > > > git bisect good d1281e3a562ec6a08f944a876481dd043ba739b9
> > > > # good: [fac7c0f46996e32d996f5c46121df24a6b95ec3b] vhost: option to fetch descriptors through an independent
> > > > struct
> > > > git bisect good fac7c0f46996e32d996f5c46121df24a6b95ec3b
> > > > # bad: [539eb9d738f048cd7be61f404e8f9c7d9d2ff3cc] vhost: batching fetches
> > > > git bisect bad 539eb9d738f048cd7be61f404e8f9c7d9d2ff3cc