vfio/pci: guest error recovery proposal

vfio/pci: guest error recovery proposal

[Michael S. Tsirkin]

>  1) We need to do the right thing for the guest, I don't think we
>     should be presuming that different reset types are equivalent,
>     leaving gaps where we expect the guest/host to do a reset and don't
>     follow through on other reset requests, and we need to notify the
>     guest immediately for the error.
c>  2) We need to do the right thing for the host, that means we should
>     not give the user the opportunity to leave a device in a state
>     where we haven't at least performed a bus reset on link error (this
>     may be our current state and if so we should fix it).

Ok so here is a concrete proposal for improving guest device error
recovery (1).  This is not trying to fix current bugs for 2, but
also does not lock us into not fixing them.

I'll write up proposal for (2) but I feel we can't properly
fix host without fixing (1) first and without breaking compatibility.

Background:

non-fatal errors:

- These errors are due to data link problems.
  The problem is that a transaction was lost, so driver and device are
  out of sync. Device reset is in theory enough to recover from these,
  in practice some drivers might try to do link level reset instead.


fatal errors:

- These errors are due to physical problems.
  The problem is that a transaction was lost, so driver and device are
  out of sync. Link reset might be necessary to recover from these,
  sometimes device reset might be enough for very simple devices.
  If a link above the device reports errors, device might have went away,
  link reset is the only thing that might being it back.

current behaviour:

- vfio will always report that it recovered function from an error.
- whether link reset will trigger depends on whether any other
  function on the same link has a host driver that reports an error.
- also, if there's a host driver that can't handle errors,
  link reset will never trigger


proposed enhancement:

1- allow userspace to request reporting non fatal/fatal errors separately
2- report errors on monitor as events as well
3- forward correct error type to guest
4- set link error flag in userspace (this is optional, used for 5 below)
5- if guest requests link reset, and error flag is set,
  stop vm (I hope we can distinguish this
  from resets that happen on reboot here.
  if yes we might not need error flag in 4 above)


Results:
The advantage of this is that we don't need to manage any state at all.
Most drivers will handle non fatal errors by FLR and will recover fine.
Drivers that attempt link reset will get vmstop which is not
worse than what we have now.

I don't see how this can break any reasonable configuration
that is not already broken, but we might want a flag
to suppress aer reports to guest and just do vmstop
unconditionally.
Alternatively, management can pause vm itself when it sees the error.


Pls remember to Cc qemu list on discussion, not just kvm.

-- 
MST

vfio/pci: host error recovery proposal

[Michael S. Tsirkin]

On Fri, Dec 16, 2016 at 01:02:14AM +0200, Michael S. Tsirkin wrote:
> 
> >  1) We need to do the right thing for the guest, I don't think we
> >     should be presuming that different reset types are equivalent,
> >     leaving gaps where we expect the guest/host to do a reset and don't
> >     follow through on other reset requests, and we need to notify the
> >     guest immediately for the error.
> c>  2) We need to do the right thing for the host, that means we should
> >     not give the user the opportunity to leave a device in a state
> >     where we haven't at least performed a bus reset on link error (this
> >     may be our current state and if so we should fix it).
> 
> Ok so here is a concrete proposal for improving guest device error
> recovery (1).  This is not trying to fix current bugs for 2, but
> also does not lock us into not fixing them.
> 
> I'll write up proposal for (2) but I feel we can't properly
> fix host without fixing (1) first and without breaking compatibility.
> 
> Background:
> 
> non-fatal errors:
> 
> - These errors are due to data link problems.
>   The problem is that a transaction was lost, so driver and device are
>   out of sync. Device reset is in theory enough to recover from these,
>   in practice some drivers might try to do link level reset instead.
> 
> 
> fatal errors:
> 
> - These errors are due to physical problems.
>   The problem is that a transaction was lost, so driver and device are
>   out of sync. Link reset might be necessary to recover from these,
>   sometimes device reset might be enough for very simple devices.
>   If a link above the device reports errors, device might have went away,
>   link reset is the only thing that might being it back.
> 
> current behaviour:
> 
> - vfio will always report that it recovered function from an error.
> - whether link reset will trigger depends on whether any other
>   function on the same link has a host driver that reports an error.
> - also, if there's a host driver that can't handle errors,
>   link reset will never trigger
> 
> 
> proposed enhancement:
> 
> 1- allow userspace to request reporting non fatal/fatal errors separately
> 2- report errors on monitor as events as well
> 3- forward correct error type to guest
> 4- set link error flag in userspace (this is optional, used for 5 below)
> 5- if guest requests link reset, and error flag is set,
>   stop vm (I hope we can distinguish this
>   from resets that happen on reboot here.
>   if yes we might not need error flag in 4 above)
> 
> 
> Results:
> The advantage of this is that we don't need to manage any state at all.
> Most drivers will handle non fatal errors by FLR and will recover fine.
> Drivers that attempt link reset will get vmstop which is not
> worse than what we have now.
> 
> I don't see how this can break any reasonable configuration
> that is not already broken, but we might want a flag
> to suppress aer reports to guest and just do vmstop
> unconditionally.
> Alternatively, management can pause vm itself when it sees the error.
> 
> 
> Pls remember to Cc qemu list on discussion, not just kvm.

Proposed enhancement to improve host level error handling.
This might need more work.
The issue we are trying to address is that devices might
need link reset.

1. detect slot (possibly link - tbd) reset
2. report to guest that link has been lost
3. at this point we can allow guest to continue,
   converting reset to flr

What's difficult is deciding whether link reset on host
is required.

Three possible ideas:

1. we can whitelist devices based on ID
2. For a generic device, we can have an option to assume
   that non fatal errors do not need link resets.
   Can be an option for devices linux does not know about
3. we can add a PV interface to tell host that 2 above
   can be used for this device


> -- 
> MST

Re: vfio/pci: guest error recovery proposal

[Cao jin]

On 12/16/2016 07:02 AM, Michael S. Tsirkin wrote:
> 
>>  1) We need to do the right thing for the guest, I don't think we
>>     should be presuming that different reset types are equivalent,
>>     leaving gaps where we expect the guest/host to do a reset and don't
>>     follow through on other reset requests, and we need to notify the
>>     guest immediately for the error.
> c>  2) We need to do the right thing for the host, that means we should
>>     not give the user the opportunity to leave a device in a state
>>     where we haven't at least performed a bus reset on link error (this
>>     may be our current state and if so we should fix it).
> 
> Ok so here is a concrete proposal for improving guest device error
> recovery (1).  This is not trying to fix current bugs for 2, but
> also does not lock us into not fixing them.
> 
> I'll write up proposal for (2) but I feel we can't properly
> fix host without fixing (1) first and without breaking compatibility.
> 
> Background:
> 
> non-fatal errors:
> 
> - These errors are due to data link problems.
>   The problem is that a transaction was lost, so driver and device are
>   out of sync. Device reset is in theory enough to recover from these,
>   in practice some drivers might try to do link level reset instead.
> 
> 
> fatal errors:
> 
> - These errors are due to physical problems.
>   The problem is that a transaction was lost, so driver and device are
>   out of sync. Link reset might be necessary to recover from these,
>   sometimes device reset might be enough for very simple devices.
>   If a link above the device reports errors, device might have went away,
>   link reset is the only thing that might being it back.
> 
> current behaviour:
> 
> - vfio will always report that it recovered function from an error.
> - whether link reset will trigger depends on whether any other
>   function on the same link has a host driver that reports an error.
> - also, if there's a host driver that can't handle errors,
>   link reset will never trigger
> 
> 
> proposed enhancement:
> 
> 1- allow userspace to request reporting non fatal/fatal errors separately
> 2- report errors on monitor as events as well
> 3- forward correct error type to guest
> 4- set link error flag in userspace (this is optional, used for 5 below)
> 5- if guest requests link reset, and error flag is set,
>   stop vm (I hope we can distinguish this
>   from resets that happen on reboot here.
>   if yes we might not need error flag in 4 above)
> 

Hi,

I have a question about vm stop on fatal error.
Recently, When test my patches, I often saw fatal error(Malformed TLP
Status) happens, which disturbed my test. So I am wondering: why vm stop
is a better choice than qdev_unplug? Although we told user "Please
collect any data possible and then kill the guest", I still don't know
how to save any possible data. For example, if user is editing document,
vm_stop caused by a device fatal error will destroy user's effort.

-- 
Sincerely,
Cao jin
> 
> Results:
> The advantage of this is that we don't need to manage any state at all.
> Most drivers will handle non fatal errors by FLR and will recover fine.
> Drivers that attempt link reset will get vmstop which is not
> worse than what we have now.
> 
> I don't see how this can break any reasonable configuration
> that is not already broken, but we might want a flag
> to suppress aer reports to guest and just do vmstop
> unconditionally.
> Alternatively, management can pause vm itself when it sees the error.
> 
> 
> Pls remember to Cc qemu list on discussion, not just kvm.
> 

Re: vfio/pci: guest error recovery proposal

[Michael S. Tsirkin]

On Wed, Dec 28, 2016 at 10:52:13AM +0800, Cao jin wrote:
> 
> 
> On 12/16/2016 07:02 AM, Michael S. Tsirkin wrote:
> > 
> >>  1) We need to do the right thing for the guest, I don't think we
> >>     should be presuming that different reset types are equivalent,
> >>     leaving gaps where we expect the guest/host to do a reset and don't
> >>     follow through on other reset requests, and we need to notify the
> >>     guest immediately for the error.
> > c>  2) We need to do the right thing for the host, that means we should
> >>     not give the user the opportunity to leave a device in a state
> >>     where we haven't at least performed a bus reset on link error (this
> >>     may be our current state and if so we should fix it).
> > 
> > Ok so here is a concrete proposal for improving guest device error
> > recovery (1).  This is not trying to fix current bugs for 2, but
> > also does not lock us into not fixing them.
> > 
> > I'll write up proposal for (2) but I feel we can't properly
> > fix host without fixing (1) first and without breaking compatibility.
> > 
> > Background:
> > 
> > non-fatal errors:
> > 
> > - These errors are due to data link problems.
> >   The problem is that a transaction was lost, so driver and device are
> >   out of sync. Device reset is in theory enough to recover from these,
> >   in practice some drivers might try to do link level reset instead.
> > 
> > 
> > fatal errors:
> > 
> > - These errors are due to physical problems.
> >   The problem is that a transaction was lost, so driver and device are
> >   out of sync. Link reset might be necessary to recover from these,
> >   sometimes device reset might be enough for very simple devices.
> >   If a link above the device reports errors, device might have went away,
> >   link reset is the only thing that might being it back.
> > 
> > current behaviour:
> > 
> > - vfio will always report that it recovered function from an error.
> > - whether link reset will trigger depends on whether any other
> >   function on the same link has a host driver that reports an error.
> > - also, if there's a host driver that can't handle errors,
> >   link reset will never trigger
> > 
> > 
> > proposed enhancement:
> > 
> > 1- allow userspace to request reporting non fatal/fatal errors separately
> > 2- report errors on monitor as events as well
> > 3- forward correct error type to guest
> > 4- set link error flag in userspace (this is optional, used for 5 below)
> > 5- if guest requests link reset, and error flag is set,
> >   stop vm (I hope we can distinguish this
> >   from resets that happen on reboot here.
> >   if yes we might not need error flag in 4 above)
> > 
> 
> Hi,
> 
> I have a question about vm stop on fatal error.
> Recently, When test my patches, I often saw fatal error(Malformed TLP
> Status) happens, which disturbed my test. So I am wondering: why vm stop
> is a better choice than qdev_unplug? Although we told user "Please
> collect any data possible and then kill the guest", I still don't know
> how to save any possible data. For example, if user is editing document,
> vm_stop caused by a device fatal error will destroy user's effort.
> 
> -- 
> Sincerely,
> Cao jin

Why vm stop might not always be the right thing to do
it happens to be what we already do.

This patchset isn't making any progress for a long time.

Focusing on incremental enhancements with minimal changes
at each step is probably the only
chance there is to make meaningful progress.


> > 
> > Results:
> > The advantage of this is that we don't need to manage any state at all.
> > Most drivers will handle non fatal errors by FLR and will recover fine.
> > Drivers that attempt link reset will get vmstop which is not
> > worse than what we have now.
> > 
> > I don't see how this can break any reasonable configuration
> > that is not already broken, but we might want a flag
> > to suppress aer reports to guest and just do vmstop
> > unconditionally.
> > Alternatively, management can pause vm itself when it sees the error.
> > 
> > 
> > Pls remember to Cc qemu list on discussion, not just kvm.
> > 
> 
> 
>