* Re: [Qemu-devel] issues of region cache and iommu reset
2017-03-29 8:09 ` Paolo Bonzini
@ 2017-03-29 8:37 ` Jason Wang
2017-03-29 8:41 ` Paolo Bonzini
2017-03-29 8:45 ` Cornelia Huck
2017-03-29 21:28 ` Michael S. Tsirkin
2 siblings, 1 reply; 16+ messages in thread
From: Jason Wang @ 2017-03-29 8:37 UTC (permalink / raw)
To: Paolo Bonzini, Michael S. Tsirkin; +Cc: Peter Xu, qemu devel, Cornelia Huck
On 2017年03月29日 16:09, Paolo Bonzini wrote:
>
> On 29/03/2017 10:00, Jason Wang wrote:
>>
>> 1) vtd was reset first
>>
>> 2) during the reset of virtio-net-pci #1, deletion of msix subregion
>> will cause a commit of all memory listeners
>>
>> 3) virito-net-pci #2's region cache will be update, but since vtd has
>> already been reset, it can't get a valid mappings here
>>
>> Any ideas on how to fix this? Need region cache be aware of IOMMU/IOTLB
>> state in this case? Or can we simply reset IOMMU as the last one?
> Something like this?
>
> diff --git a/hw/virtio/virtio.c b/hw/virtio/virtio.c
> index 03592c5..73e69ac 100644
> --- a/hw/virtio/virtio.c
> +++ b/hw/virtio/virtio.c
> @@ -176,6 +176,10 @@ err_used:
> address_space_cache_destroy(&new->desc);
> err_desc:
> g_free(new);
> + atomic_rcu_set(&vq->vring.caches, NULL);
> + if (old) {
> + call_rcu(old, virtio_free_region_cache, rcu);
> + }
> }
>
> /* virt queue functions */
>
> Paolo
This looks a good fix but may not solve this issue completely. Depends
on the iova that guest uses, address_space_cache_init() may succeed even
in this case since vtd does a passthrough translation in this case.
Thanks
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [Qemu-devel] issues of region cache and iommu reset
2017-03-29 8:37 ` Jason Wang
@ 2017-03-29 8:41 ` Paolo Bonzini
2017-03-29 9:09 ` Jason Wang
0 siblings, 1 reply; 16+ messages in thread
From: Paolo Bonzini @ 2017-03-29 8:41 UTC (permalink / raw)
To: Jason Wang, Michael S. Tsirkin; +Cc: Peter Xu, qemu devel, Cornelia Huck
On 29/03/2017 10:37, Jason Wang wrote:
>>
>> diff --git a/hw/virtio/virtio.c b/hw/virtio/virtio.c
>> index 03592c5..73e69ac 100644
>> --- a/hw/virtio/virtio.c
>> +++ b/hw/virtio/virtio.c
>> @@ -176,6 +176,10 @@ err_used:
>> address_space_cache_destroy(&new->desc);
>> err_desc:
>> g_free(new);
>> + atomic_rcu_set(&vq->vring.caches, NULL);
>> + if (old) {
>> + call_rcu(old, virtio_free_region_cache, rcu);
>> + }
>> }
>>
>> /* virt queue functions */
>>
>> Paolo
>
> This looks a good fix but may not solve this issue completely. Depends
> on the iova that guest uses, address_space_cache_init() may succeed even
> in this case since vtd does a passthrough translation in this case.
Can you explain this more?
Paolo
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [Qemu-devel] issues of region cache and iommu reset
2017-03-29 8:41 ` Paolo Bonzini
@ 2017-03-29 9:09 ` Jason Wang
2017-03-29 9:11 ` Paolo Bonzini
0 siblings, 1 reply; 16+ messages in thread
From: Jason Wang @ 2017-03-29 9:09 UTC (permalink / raw)
To: Paolo Bonzini, Michael S. Tsirkin; +Cc: Peter Xu, qemu devel, Cornelia Huck
On 2017年03月29日 16:41, Paolo Bonzini wrote:
>
> On 29/03/2017 10:37, Jason Wang wrote:
>>> diff --git a/hw/virtio/virtio.c b/hw/virtio/virtio.c
>>> index 03592c5..73e69ac 100644
>>> --- a/hw/virtio/virtio.c
>>> +++ b/hw/virtio/virtio.c
>>> @@ -176,6 +176,10 @@ err_used:
>>> address_space_cache_destroy(&new->desc);
>>> err_desc:
>>> g_free(new);
>>> + atomic_rcu_set(&vq->vring.caches, NULL);
>>> + if (old) {
>>> + call_rcu(old, virtio_free_region_cache, rcu);
>>> + }
>>> }
>>>
>>> /* virt queue functions */
>>>
>>> Paolo
>> This looks a good fix but may not solve this issue completely. Depends
>> on the iova that guest uses, address_space_cache_init() may succeed even
>> in this case since vtd does a passthrough translation in this case.
> Can you explain this more?
>
> Paolo
Yes. In this case, virtio-net-pci is not reset, but vtd has been reset.
So virtio-net-pci will still try to use the iova of ring to setup the
cache but now vtd treat iova as gpa since dmar has been disabled during
reset. (E.g In my test with Linux driver, desc map succeed but used fail.)
Thanks
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [Qemu-devel] issues of region cache and iommu reset
2017-03-29 9:09 ` Jason Wang
@ 2017-03-29 9:11 ` Paolo Bonzini
2017-03-29 9:24 ` Jason Wang
0 siblings, 1 reply; 16+ messages in thread
From: Paolo Bonzini @ 2017-03-29 9:11 UTC (permalink / raw)
To: Jason Wang, Michael S. Tsirkin; +Cc: Peter Xu, qemu devel, Cornelia Huck
On 29/03/2017 11:09, Jason Wang wrote:
>>>>
>>> This looks a good fix but may not solve this issue completely. Depends
>>> on the iova that guest uses, address_space_cache_init() may succeed even
>>> in this case since vtd does a passthrough translation in this case.
>> Can you explain this more?
>
> Yes. In this case, virtio-net-pci is not reset, but vtd has been reset.
> So virtio-net-pci will still try to use the iova of ring to setup the
> cache but now vtd treat iova as gpa since dmar has been disabled during
> reset. (E.g In my test with Linux driver, desc map succeed but used fail.)
That would be a Linux bug, the devices below the IOMMU have to be reset
first.
Paolo
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [Qemu-devel] issues of region cache and iommu reset
2017-03-29 9:11 ` Paolo Bonzini
@ 2017-03-29 9:24 ` Jason Wang
2017-03-29 21:31 ` Michael S. Tsirkin
0 siblings, 1 reply; 16+ messages in thread
From: Jason Wang @ 2017-03-29 9:24 UTC (permalink / raw)
To: Paolo Bonzini, Michael S. Tsirkin; +Cc: Peter Xu, qemu devel, Cornelia Huck
On 2017年03月29日 17:11, Paolo Bonzini wrote:
>
> On 29/03/2017 11:09, Jason Wang wrote:
>>>> This looks a good fix but may not solve this issue completely. Depends
>>>> on the iova that guest uses, address_space_cache_init() may succeed even
>>>> in this case since vtd does a passthrough translation in this case.
>>> Can you explain this more?
>> Yes. In this case, virtio-net-pci is not reset, but vtd has been reset.
>> So virtio-net-pci will still try to use the iova of ring to setup the
>> cache but now vtd treat iova as gpa since dmar has been disabled during
>> reset. (E.g In my test with Linux driver, desc map succeed but used fail.)
> That would be a Linux bug, the devices below the IOMMU have to be reset
> first.
>
> Paolo
Maybe, but I meet the same issue if I do a "system_reset". I think we
should guarantee IOMMU was reset after any pci devices in this case too?
Thanks
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [Qemu-devel] issues of region cache and iommu reset
2017-03-29 9:24 ` Jason Wang
@ 2017-03-29 21:31 ` Michael S. Tsirkin
0 siblings, 0 replies; 16+ messages in thread
From: Michael S. Tsirkin @ 2017-03-29 21:31 UTC (permalink / raw)
To: Jason Wang; +Cc: Paolo Bonzini, Peter Xu, qemu devel, Cornelia Huck
On Wed, Mar 29, 2017 at 05:24:18PM +0800, Jason Wang wrote:
>
>
> On 2017年03月29日 17:11, Paolo Bonzini wrote:
> >
> > On 29/03/2017 11:09, Jason Wang wrote:
> > > > > This looks a good fix but may not solve this issue completely. Depends
> > > > > on the iova that guest uses, address_space_cache_init() may succeed even
> > > > > in this case since vtd does a passthrough translation in this case.
> > > > Can you explain this more?
> > > Yes. In this case, virtio-net-pci is not reset, but vtd has been reset.
> > > So virtio-net-pci will still try to use the iova of ring to setup the
> > > cache but now vtd treat iova as gpa since dmar has been disabled during
> > > reset. (E.g In my test with Linux driver, desc map succeed but used fail.)
> > That would be a Linux bug, the devices below the IOMMU have to be reset
> > first.
> >
> > Paolo
>
> Maybe, but I meet the same issue if I do a "system_reset". I think we should
> guarantee IOMMU was reset after any pci devices in this case too?
>
> Thanks
>
system reset would indeed appear to be a big issue.
--
MST
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [Qemu-devel] issues of region cache and iommu reset
2017-03-29 8:09 ` Paolo Bonzini
2017-03-29 8:37 ` Jason Wang
@ 2017-03-29 8:45 ` Cornelia Huck
2017-03-29 9:18 ` Jason Wang
2017-03-29 21:28 ` Michael S. Tsirkin
2 siblings, 1 reply; 16+ messages in thread
From: Cornelia Huck @ 2017-03-29 8:45 UTC (permalink / raw)
To: Paolo Bonzini; +Cc: Jason Wang, Michael S. Tsirkin, Peter Xu, qemu devel
On Wed, 29 Mar 2017 10:09:10 +0200
Paolo Bonzini <pbonzini@redhat.com> wrote:
> On 29/03/2017 10:00, Jason Wang wrote:
> >
> >
> > 1) vtd was reset first
> >
> > 2) during the reset of virtio-net-pci #1, deletion of msix subregion
> > will cause a commit of all memory listeners
> >
> > 3) virito-net-pci #2's region cache will be update, but since vtd has
> > already been reset, it can't get a valid mappings here
> >
> > Any ideas on how to fix this? Need region cache be aware of IOMMU/IOTLB
> > state in this case? Or can we simply reset IOMMU as the last one?
>
> Something like this?
>
> diff --git a/hw/virtio/virtio.c b/hw/virtio/virtio.c
> index 03592c5..73e69ac 100644
> --- a/hw/virtio/virtio.c
> +++ b/hw/virtio/virtio.c
> @@ -176,6 +176,10 @@ err_used:
> address_space_cache_destroy(&new->desc);
> err_desc:
> g_free(new);
> + atomic_rcu_set(&vq->vring.caches, NULL);
> + if (old) {
> + call_rcu(old, virtio_free_region_cache, rcu);
> + }
Maybe I'm just confused here, but isn't ->broken enough to prevent
further accesses? And a reset will unset both ->broken and the caches
anyway? What am I missing?
> }
>
> /* virt queue functions */
>
> Paolo
>
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [Qemu-devel] issues of region cache and iommu reset
2017-03-29 8:45 ` Cornelia Huck
@ 2017-03-29 9:18 ` Jason Wang
2017-03-29 11:39 ` Cornelia Huck
0 siblings, 1 reply; 16+ messages in thread
From: Jason Wang @ 2017-03-29 9:18 UTC (permalink / raw)
To: Cornelia Huck, Paolo Bonzini; +Cc: Michael S. Tsirkin, Peter Xu, qemu devel
On 2017年03月29日 16:45, Cornelia Huck wrote:
> On Wed, 29 Mar 2017 10:09:10 +0200
> Paolo Bonzini <pbonzini@redhat.com> wrote:
>
>> On 29/03/2017 10:00, Jason Wang wrote:
>>>
>>> 1) vtd was reset first
>>>
>>> 2) during the reset of virtio-net-pci #1, deletion of msix subregion
>>> will cause a commit of all memory listeners
>>>
>>> 3) virito-net-pci #2's region cache will be update, but since vtd has
>>> already been reset, it can't get a valid mappings here
>>>
>>> Any ideas on how to fix this? Need region cache be aware of IOMMU/IOTLB
>>> state in this case? Or can we simply reset IOMMU as the last one?
>> Something like this?
>>
>> diff --git a/hw/virtio/virtio.c b/hw/virtio/virtio.c
>> index 03592c5..73e69ac 100644
>> --- a/hw/virtio/virtio.c
>> +++ b/hw/virtio/virtio.c
>> @@ -176,6 +176,10 @@ err_used:
>> address_space_cache_destroy(&new->desc);
>> err_desc:
>> g_free(new);
>> + atomic_rcu_set(&vq->vring.caches, NULL);
>> + if (old) {
>> + call_rcu(old, virtio_free_region_cache, rcu);
>> + }
> Maybe I'm just confused here, but isn't ->broken enough to prevent
> further accesses?
Looks not (e.g virtio_queue_update_used_idx() that was called by vhost).
We only have pfn check for all helpers now.
> And a reset will unset both ->broken and the caches
> anyway? What am I missing?
Yes, but is it good to store invalid or even wrong mappings in the cache?
Thanks
>
>> }
>>
>> /* virt queue functions */
>>
>> Paolo
>>
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [Qemu-devel] issues of region cache and iommu reset
2017-03-29 9:18 ` Jason Wang
@ 2017-03-29 11:39 ` Cornelia Huck
2017-03-30 2:14 ` Jason Wang
0 siblings, 1 reply; 16+ messages in thread
From: Cornelia Huck @ 2017-03-29 11:39 UTC (permalink / raw)
To: Jason Wang; +Cc: Paolo Bonzini, Michael S. Tsirkin, Peter Xu, qemu devel
On Wed, 29 Mar 2017 17:18:00 +0800
Jason Wang <jasowang@redhat.com> wrote:
> On 2017年03月29日 16:45, Cornelia Huck wrote:
> > On Wed, 29 Mar 2017 10:09:10 +0200
> > Paolo Bonzini <pbonzini@redhat.com> wrote:
> >
> >> On 29/03/2017 10:00, Jason Wang wrote:
> >>>
> >>> 1) vtd was reset first
> >>>
> >>> 2) during the reset of virtio-net-pci #1, deletion of msix subregion
> >>> will cause a commit of all memory listeners
> >>>
> >>> 3) virito-net-pci #2's region cache will be update, but since vtd has
> >>> already been reset, it can't get a valid mappings here
> >>>
> >>> Any ideas on how to fix this? Need region cache be aware of IOMMU/IOTLB
> >>> state in this case? Or can we simply reset IOMMU as the last one?
> >> Something like this?
> >>
> >> diff --git a/hw/virtio/virtio.c b/hw/virtio/virtio.c
> >> index 03592c5..73e69ac 100644
> >> --- a/hw/virtio/virtio.c
> >> +++ b/hw/virtio/virtio.c
> >> @@ -176,6 +176,10 @@ err_used:
> >> address_space_cache_destroy(&new->desc);
> >> err_desc:
> >> g_free(new);
> >> + atomic_rcu_set(&vq->vring.caches, NULL);
> >> + if (old) {
> >> + call_rcu(old, virtio_free_region_cache, rcu);
> >> + }
> > Maybe I'm just confused here, but isn't ->broken enough to prevent
> > further accesses?
>
> Looks not (e.g virtio_queue_update_used_idx() that was called by vhost).
We probably should take care that we don't do things like that for
broken devices. But that ties into the virtio_error() discussion, and
we probably want to revisit that in that context.
> We only have pfn check for all helpers now.
So we'll assert() after that change?
>
> > And a reset will unset both ->broken and the caches
> > anyway? What am I missing?
>
> Yes, but is it good to store invalid or even wrong mappings in the cache?
Not really; but I'd like to see ->broken as a kind of master indicator
"this device does not work, do not trust/use anything until it has been
reset".
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [Qemu-devel] issues of region cache and iommu reset
2017-03-29 11:39 ` Cornelia Huck
@ 2017-03-30 2:14 ` Jason Wang
0 siblings, 0 replies; 16+ messages in thread
From: Jason Wang @ 2017-03-30 2:14 UTC (permalink / raw)
To: Cornelia Huck; +Cc: Paolo Bonzini, qemu devel, Peter Xu, Michael S. Tsirkin
On 2017年03月29日 19:39, Cornelia Huck wrote:
> On Wed, 29 Mar 2017 17:18:00 +0800
> Jason Wang <jasowang@redhat.com> wrote:
>
>> On 2017年03月29日 16:45, Cornelia Huck wrote:
>>> On Wed, 29 Mar 2017 10:09:10 +0200
>>> Paolo Bonzini <pbonzini@redhat.com> wrote:
>>>
>>>> On 29/03/2017 10:00, Jason Wang wrote:
>>>>> 1) vtd was reset first
>>>>>
>>>>> 2) during the reset of virtio-net-pci #1, deletion of msix subregion
>>>>> will cause a commit of all memory listeners
>>>>>
>>>>> 3) virito-net-pci #2's region cache will be update, but since vtd has
>>>>> already been reset, it can't get a valid mappings here
>>>>>
>>>>> Any ideas on how to fix this? Need region cache be aware of IOMMU/IOTLB
>>>>> state in this case? Or can we simply reset IOMMU as the last one?
>>>> Something like this?
>>>>
>>>> diff --git a/hw/virtio/virtio.c b/hw/virtio/virtio.c
>>>> index 03592c5..73e69ac 100644
>>>> --- a/hw/virtio/virtio.c
>>>> +++ b/hw/virtio/virtio.c
>>>> @@ -176,6 +176,10 @@ err_used:
>>>> address_space_cache_destroy(&new->desc);
>>>> err_desc:
>>>> g_free(new);
>>>> + atomic_rcu_set(&vq->vring.caches, NULL);
>>>> + if (old) {
>>>> + call_rcu(old, virtio_free_region_cache, rcu);
>>>> + }
>>> Maybe I'm just confused here, but isn't ->broken enough to prevent
>>> further accesses?
>> Looks not (e.g virtio_queue_update_used_idx() that was called by vhost).
> We probably should take care that we don't do things like that for
> broken devices. But that ties into the virtio_error() discussion, and
> we probably want to revisit that in that context.
Yes, but the issue here is, the device should not be treated as broken
since it does nothing wrong.
>
>> We only have pfn check for all helpers now.
> So we'll assert() after that change?
I think we don't want assert too, since it was a indicator of bug somewhere.
>
>>> And a reset will unset both ->broken and the caches
>>> anyway? What am I missing?
>> Yes, but is it good to store invalid or even wrong mappings in the cache?
> Not really; but I'd like to see ->broken as a kind of master indicator
> "this device does not work, do not trust/use anything until it has been
> reset".
>
Then I guess we probably check broken for all exported helpers like what
we did for gfn.
Thanks
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [Qemu-devel] issues of region cache and iommu reset
2017-03-29 8:09 ` Paolo Bonzini
2017-03-29 8:37 ` Jason Wang
2017-03-29 8:45 ` Cornelia Huck
@ 2017-03-29 21:28 ` Michael S. Tsirkin
2017-03-30 9:02 ` Paolo Bonzini
2 siblings, 1 reply; 16+ messages in thread
From: Michael S. Tsirkin @ 2017-03-29 21:28 UTC (permalink / raw)
To: Paolo Bonzini; +Cc: Jason Wang, Peter Xu, qemu devel, Cornelia Huck
On Wed, Mar 29, 2017 at 10:09:10AM +0200, Paolo Bonzini wrote:
>
>
> On 29/03/2017 10:00, Jason Wang wrote:
> >
> >
> > 1) vtd was reset first
> >
> > 2) during the reset of virtio-net-pci #1, deletion of msix subregion
> > will cause a commit of all memory listeners
> >
> > 3) virito-net-pci #2's region cache will be update, but since vtd has
> > already been reset, it can't get a valid mappings here
> >
> > Any ideas on how to fix this? Need region cache be aware of IOMMU/IOTLB
> > state in this case? Or can we simply reset IOMMU as the last one?
>
> Something like this?
>
> diff --git a/hw/virtio/virtio.c b/hw/virtio/virtio.c
> index 03592c5..73e69ac 100644
> --- a/hw/virtio/virtio.c
> +++ b/hw/virtio/virtio.c
> @@ -176,6 +176,10 @@ err_used:
> address_space_cache_destroy(&new->desc);
> err_desc:
> g_free(new);
> + atomic_rcu_set(&vq->vring.caches, NULL);
> + if (old) {
> + call_rcu(old, virtio_free_region_cache, rcu);
> + }
> }
>
> /* virt queue functions */
I would be worried about call_rcu here - this means
something can hang on to and use the old cache,
and reset really must act as a sync/flush point.
> Paolo
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [Qemu-devel] issues of region cache and iommu reset
2017-03-29 21:28 ` Michael S. Tsirkin
@ 2017-03-30 9:02 ` Paolo Bonzini
0 siblings, 0 replies; 16+ messages in thread
From: Paolo Bonzini @ 2017-03-30 9:02 UTC (permalink / raw)
To: Michael S. Tsirkin; +Cc: Jason Wang, Peter Xu, qemu devel, Cornelia Huck
On 29/03/2017 23:28, Michael S. Tsirkin wrote:
>> Something like this?
>>
>> diff --git a/hw/virtio/virtio.c b/hw/virtio/virtio.c
>> index 03592c5..73e69ac 100644
>> --- a/hw/virtio/virtio.c
>> +++ b/hw/virtio/virtio.c
>> @@ -176,6 +176,10 @@ err_used:
>> address_space_cache_destroy(&new->desc);
>> err_desc:
>> g_free(new);
>> + atomic_rcu_set(&vq->vring.caches, NULL);
>> + if (old) {
>> + call_rcu(old, virtio_free_region_cache, rcu);
>> + }
>> }
>>
>> /* virt queue functions */
> I would be worried about call_rcu here - this means
> something can hang on to and use the old cache,
> and reset really must act as a sync/flush point.
The flush is done later in virtio_reset. Here it's just reacting
asynchronously to the IOMMU reset.
I'm thinking of adding a global generation count for IOMMU mappings, and
forcing an update when the IOMMU mappings have changed.
Paolo
^ permalink raw reply [flat|nested] 16+ messages in thread