[Qemu-devel] [RFC PATCH 0/4] pass aer error to guest

[Qemu-devel] [RFC PATCH 0/4] pass aer error to guest

[Chen Fan]

For now, for vfio pci passthough devices when qemu receives
an error from host aer report, there just terminate the guest,
but usually user want to know what error occured but stop the
guest, so this patch add aer capability support for vfio device,
and pass the error to guest, and have guest driver to recover
from the error.

Chen Fan (4):
  pcie_aer: fix typos in pcie_aer_inject_error comment
  pcie-aer: Fix command pcie_aer_inject_error is invalid
  vfio-pci: add aer capability support
  vfio-pci: pass the aer error to guest

 hw/pci/pcie_aer.c         | 11 +++----
 hw/vfio/pci.c             | 75 ++++++++++++++++++++++++++++++++++++++++++-----
 include/hw/pci/pcie_aer.h |  3 +-
 3 files changed, 76 insertions(+), 13 deletions(-)

-- 
1.9.3

[Qemu-devel] [RFC PATCH 1/4] pcie_aer: fix typos in pcie_aer_inject_error comment

[Chen Fan]

Refer to "PCI Express Base Spec3.0", this comments can't
fit the description in spec, so we should fix them.

Signed-off-by: Chen Fan <chen.fan.fnst@cn.fujitsu.com>
---
 hw/pci/pcie_aer.c | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/hw/pci/pcie_aer.c b/hw/pci/pcie_aer.c
index 1f4be16..7ca077a 100644
--- a/hw/pci/pcie_aer.c
+++ b/hw/pci/pcie_aer.c
@@ -618,11 +618,11 @@ static bool pcie_aer_inject_uncor_error(PCIEAERInject *inj, bool is_fatal)
  * non-Function specific error must be recorded in all functions.
  * It is the responsibility of the caller of this function.
  * It is also caller's responsibility to determine which function should
- * report the rerror.
+ * report the error.
  *
  * 6.2.4 Error Logging
- * 6.2.5 Sqeunce of Device Error Signaling and Logging Operations
- * table 6-2: Flowchard Showing Sequence of Device Error Signaling and Logging
+ * 6.2.5 Sequence of Device Error Signaling and Logging Operations
+ * table 6-2: Flowchart Showing Sequence of Device Error Signaling and Logging
  *            Operations
  */
 int pcie_aer_inject_error(PCIDevice *dev, const PCIEAERErr *err)
-- 
1.9.3

[Qemu-devel] [RFC PATCH 2/4] pcie-aer: Fix command pcie_aer_inject_error is invalid

[Chen Fan]

in spec "PCI Express 3.0" section 6.2.6 Figure 6-3 virtual bridge part,
the flowchart showing tell us SERR# enable at Bridge Control register
associate with system error at Secondary Status register can send error
message. but bridge_control from dev->config is NULL, and SERR# was set
in dev->wmask in pcie_aer_init() which was implemented by root port and
swith devices, so we should add wmask (for w/r) bit set for bridge control.
we can user command like:
qemu_system_x86_64:
-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,id=bridge1
-device x3130-upstream,bus=bridge1,id=up.1,addr=00.0
-device xio3130-downstream,bus=up.1,id=down.1,port=1,addr=00.0,chassis=5

(qemu) pcie_aer_inject_error net0 POISON_TLP

after that,
guest can output the error message.

Signed-off-by: Chen Fan <chen.fan.fnst@cn.fujitsu.com>
---
 hw/pci/pcie_aer.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/hw/pci/pcie_aer.c b/hw/pci/pcie_aer.c
index 7ca077a..571dc92 100644
--- a/hw/pci/pcie_aer.c
+++ b/hw/pci/pcie_aer.c
@@ -231,7 +231,8 @@ pcie_aer_msg_alldev(PCIDevice *dev, const PCIEAERMsg *msg)
  */
 static bool pcie_aer_msg_vbridge(PCIDevice *dev, const PCIEAERMsg *msg)
 {
-    uint16_t bridge_control = pci_get_word(dev->config + PCI_BRIDGE_CONTROL);
+    uint16_t bridge_control = pci_get_word(dev->config + PCI_BRIDGE_CONTROL) |
+                              pci_get_word(dev->wmask + PCI_BRIDGE_CONTROL);
 
     if (pcie_aer_msg_is_uncor(msg)) {
         /* Received System Error */
-- 
1.9.3

[Qemu-devel] [RFC PATCH 3/4] vfio-pci: add aer capability support

[Chen Fan]

Signed-off-by: Chen Fan <chen.fan.fnst@cn.fujitsu.com>
---
 hw/vfio/pci.c | 40 ++++++++++++++++++++++++++++++++++++++++
 1 file changed, 40 insertions(+)

diff --git a/hw/vfio/pci.c b/hw/vfio/pci.c
index b4e73d1..0ee6326 100644
--- a/hw/vfio/pci.c
+++ b/hw/vfio/pci.c
@@ -2667,6 +2667,41 @@ static int vfio_add_capabilities(VFIOPCIDevice *vdev)
     return vfio_add_std_cap(vdev, pdev->config[PCI_CAPABILITY_LIST]);
 }
 
+static int vfio_add_ext_capabilities(VFIOPCIDevice *vdev)
+{
+    PCIDevice *pdev = &vdev->pdev;
+    PCIExpressDevice *exp;
+    uint32_t header;
+    uint16_t next = PCI_CONFIG_SPACE_SIZE;
+
+    if (pci_config_size(pdev) <= PCI_CONFIG_SPACE_SIZE) {
+        return 0;
+    }
+
+    header = pci_get_long(pdev->config + next);
+    while (header) {
+        switch (PCI_EXT_CAP_ID(header)) {
+        case PCI_EXT_CAP_ID_ERR:
+             exp = &pdev->exp;
+             exp->aer_cap = next;
+
+             /* enable the error report */
+             vfio_add_emulated_long(vdev, exp->exp_cap + PCI_EXP_DEVCTL,
+                 PCI_EXP_DEVCTL_CERE | PCI_EXP_DEVCTL_NFERE |
+                 PCI_EXP_DEVCTL_FERE | PCI_EXP_DEVCTL_URRE, ~0);
+             break;
+        };
+
+        next = PCI_EXT_CAP_NEXT(header);
+        if (!next) {
+            return 0;
+        }
+        header = pci_get_long(pdev->config + next);
+    }
+
+    return 0;
+}
+
 static void vfio_pci_pre_reset(VFIOPCIDevice *vdev)
 {
     PCIDevice *pdev = &vdev->pdev;
@@ -3293,6 +3328,11 @@ static int vfio_initfn(PCIDevice *pdev)
         goto out_teardown;
     }
 
+    ret = vfio_add_ext_capabilities(vdev);
+    if (ret) {
+        goto out_teardown;
+    }
+
     /* QEMU emulates all of MSI & MSIX */
     if (pdev->cap_present & QEMU_PCI_CAP_MSIX) {
         memset(vdev->emulated_config_bits + pdev->msix_cap, 0xff,
-- 
1.9.3

[Qemu-devel] [RFC PATCH 4/4] vfio-pci: pass the aer error to guest

[Chen Fan]

when the vfio device encounters an uncorrectable error in host,
 the vfio_pci driver will signal the eventfd registered by this
 vfio device, the results in the qemu eventfd handler getting
 invoked.

this patch is to pass the error to guest and have the guest driver
recover from the error.

Signed-off-by: Chen Fan <chen.fan.fnst@cn.fujitsu.com>
---
 hw/pci/pcie_aer.c         |  2 +-
 hw/vfio/pci.c             | 35 ++++++++++++++++++++++++++++-------
 include/hw/pci/pcie_aer.h |  3 ++-
 3 files changed, 31 insertions(+), 9 deletions(-)

diff --git a/hw/pci/pcie_aer.c b/hw/pci/pcie_aer.c
index 571dc92..4812e3d 100644
--- a/hw/pci/pcie_aer.c
+++ b/hw/pci/pcie_aer.c
@@ -371,7 +371,7 @@ static void pcie_aer_msg_root_port(PCIDevice *dev, const PCIEAERMsg *msg)
  *
  * Walk up the bus tree from the device, propagate the error message.
  */
-static void pcie_aer_msg(PCIDevice *dev, const PCIEAERMsg *msg)
+void pcie_aer_msg(PCIDevice *dev, const PCIEAERMsg *msg)
 {
     uint8_t type;
 
diff --git a/hw/vfio/pci.c b/hw/vfio/pci.c
index 0ee6326..6f0b265 100644
--- a/hw/vfio/pci.c
+++ b/hw/vfio/pci.c
@@ -3106,20 +3106,41 @@ static void vfio_put_device(VFIOPCIDevice *vdev)
 static void vfio_err_notifier_handler(void *opaque)
 {
     VFIOPCIDevice *vdev = opaque;
+    PCIDevice *dev = &vdev->pdev;
+    PCIEAERMsg msg = {
+        .severity = 0,
+        .source_id = (pci_bus_num(dev->bus) << 8) | dev->devfn,
+    };
 
     if (!event_notifier_test_and_clear(&vdev->err_notifier)) {
         return;
     }
 
-    /*
-     * TBD. Retrieve the error details and decide what action
-     * needs to be taken. One of the actions could be to pass
-     * the error to the guest and have the guest driver recover
-     * from the error. This requires that PCIe capabilities be
-     * exposed to the guest. For now, we just terminate the
-     * guest to contain the error.
+    /* we should read the error details from the real hardware
+     * configration spaces, here we only need to do is signaling
+     * to guest an uncorrectable error has occurred.
      */
+    if (dev->exp.aer_cap) {
+        uint8_t *aer_cap = dev->config + dev->exp.aer_cap;
+        uint32_t uncor_status;
+        bool isfatal;
+
+        uncor_status = vfio_pci_read_config(dev,
+                           dev->exp.aer_cap + PCI_ERR_UNCOR_STATUS, 4);
+
+        isfatal = uncor_status & pci_get_long(aer_cap + PCI_ERR_UNCOR_SEVER);
+
+        msg.severity = isfatal ? PCI_ERR_ROOT_CMD_FATAL_EN :
+                                 PCI_ERR_ROOT_CMD_NONFATAL_EN;
 
+        pcie_aer_msg(dev, &msg);
+        return;
+    }
+
+    /*
+     * If the aer capability is not exposed to the guest. we just
+     * terminate the guest to contain the error.
+     */
     error_report("%s(%04x:%02x:%02x.%x) Unrecoverable error detected.  "
                  "Please collect any data possible and then kill the guest",
                  __func__, vdev->host.domain, vdev->host.bus,
diff --git a/include/hw/pci/pcie_aer.h b/include/hw/pci/pcie_aer.h
index bcac80a..6c4bf3b 100644
--- a/include/hw/pci/pcie_aer.h
+++ b/include/hw/pci/pcie_aer.h
@@ -51,7 +51,7 @@ struct PCIEAERLog {
     PCIEAERErr *log;
 };
 
-/* aer error message: error signaling message has only error sevirity and
+/* aer error message: error signaling message has only error severity and
    source id. See 2.2.8.3 error signaling messages */
 struct PCIEAERMsg {
     /*
@@ -102,5 +102,6 @@ void pcie_aer_root_write_config(PCIDevice *dev,
 
 /* error injection */
 int pcie_aer_inject_error(PCIDevice *dev, const PCIEAERErr *err);
+void pcie_aer_msg(PCIDevice *dev, const PCIEAERMsg *msg);
 
 #endif /* QEMU_PCIE_AER_H */
-- 
1.9.3

Re: [Qemu-devel] [RFC PATCH 3/4] vfio-pci: add aer capability support

[Paolo Bonzini]

On 12/01/2015 04:04, Chen Fan wrote:
> +static int vfio_add_ext_capabilities(VFIOPCIDevice *vdev)
> +{
> +    PCIDevice *pdev = &vdev->pdev;
> +    PCIExpressDevice *exp;
> +    uint32_t header;
> +    uint16_t next = PCI_CONFIG_SPACE_SIZE;
> +
> +    if (pci_config_size(pdev) <= PCI_CONFIG_SPACE_SIZE) {
> +        return 0;
> +    }
> +
> +    header = pci_get_long(pdev->config + next);
> +    while (header) {
> +        switch (PCI_EXT_CAP_ID(header)) {
> +        case PCI_EXT_CAP_ID_ERR:
> +             exp = &pdev->exp;
> +             exp->aer_cap = next;
> +
> +             /* enable the error report */
> +             vfio_add_emulated_long(vdev, exp->exp_cap + PCI_EXP_DEVCTL,
> +                 PCI_EXP_DEVCTL_CERE | PCI_EXP_DEVCTL_NFERE |
> +                 PCI_EXP_DEVCTL_FERE | PCI_EXP_DEVCTL_URRE, ~0);
> +             break;
> +        };
> +
> +        next = PCI_EXT_CAP_NEXT(header);
> +        if (!next) {
> +            return 0;
> +        }
> +        header = pci_get_long(pdev->config + next);
> +    }
> +
> +    return 0;
> +}
> +

Please add a property to the VFIO device, defaulting to true, and
disable it for older machine types.

Paolo

Re: [Qemu-devel] [RFC PATCH 2/4] pcie-aer: Fix command pcie_aer_inject_error is invalid

[Marcel Apfelbaum]

On 01/12/2015 05:04 AM, Chen Fan wrote:
> in spec "PCI Express 3.0" section 6.2.6 Figure 6-3 virtual bridge part,
> the flowchart showing tell us SERR# enable at Bridge Control register
> associate with system error at Secondary Status register can send error
> message. but bridge_control from dev->config is NULL, and SERR# was set
> in dev->wmask in pcie_aer_init()
wmask denotes the register bits that can be written by the guest.

If you are referring to:
        pci_word_test_and_set_mask(dev->wmask + PCI_BRIDGE_CONTROL,
                                   PCI_BRIDGE_CTL_SERR);
that means that the OS *is able* to turn on/off SERR forwarding.


  which was implemented by root port and
> swith devices, so we should add wmask (for w/r) bit set for bridge control.
> we can user command like:
> qemu_system_x86_64:
> -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,id=bridge1
> -device x3130-upstream,bus=bridge1,id=up.1,addr=00.0
> -device xio3130-downstream,bus=up.1,id=down.1,port=1,addr=00.0,chassis=5
>
> (qemu) pcie_aer_inject_error net0 POISON_TLP
>
> after that,
> guest can output the error message.
>
> Signed-off-by: Chen Fan <chen.fan.fnst@cn.fujitsu.com>
> ---
>   hw/pci/pcie_aer.c | 3 ++-
>   1 file changed, 2 insertions(+), 1 deletion(-)
>
> diff --git a/hw/pci/pcie_aer.c b/hw/pci/pcie_aer.c
> index 7ca077a..571dc92 100644
> --- a/hw/pci/pcie_aer.c
> +++ b/hw/pci/pcie_aer.c
> @@ -231,7 +231,8 @@ pcie_aer_msg_alldev(PCIDevice *dev, const PCIEAERMsg *msg)
>    */
>   static bool pcie_aer_msg_vbridge(PCIDevice *dev, const PCIEAERMsg *msg)
>   {
> -    uint16_t bridge_control = pci_get_word(dev->config + PCI_BRIDGE_CONTROL);
Here we check if the Guest OS/firmware actually turned the #SERR forwarding on.

> +    uint16_t bridge_control = pci_get_word(dev->config + PCI_BRIDGE_CONTROL) |
> +                              pci_get_word(dev->wmask + PCI_BRIDGE_CONTROL);
I don't think that this check is correct given the above comments.
Please correct me if I mislead you,
Thanks,
Marcel


>
>       if (pcie_aer_msg_is_uncor(msg)) {
>           /* Received System Error */
>

Re: [Qemu-devel] [RFC PATCH 4/4] vfio-pci: pass the aer error to guest

[Alex Williamson]

On Mon, 2015-01-12 at 11:04 +0800, Chen Fan wrote:
> when the vfio device encounters an uncorrectable error in host,
>  the vfio_pci driver will signal the eventfd registered by this
>  vfio device, the results in the qemu eventfd handler getting
>  invoked.
> 
> this patch is to pass the error to guest and have the guest driver
> recover from the error.
> 
> Signed-off-by: Chen Fan <chen.fan.fnst@cn.fujitsu.com>
> ---
>  hw/pci/pcie_aer.c         |  2 +-
>  hw/vfio/pci.c             | 35 ++++++++++++++++++++++++++++-------
>  include/hw/pci/pcie_aer.h |  3 ++-
>  3 files changed, 31 insertions(+), 9 deletions(-)
> 
> diff --git a/hw/pci/pcie_aer.c b/hw/pci/pcie_aer.c
> index 571dc92..4812e3d 100644
> --- a/hw/pci/pcie_aer.c
> +++ b/hw/pci/pcie_aer.c
> @@ -371,7 +371,7 @@ static void pcie_aer_msg_root_port(PCIDevice *dev, const PCIEAERMsg *msg)
>   *
>   * Walk up the bus tree from the device, propagate the error message.
>   */
> -static void pcie_aer_msg(PCIDevice *dev, const PCIEAERMsg *msg)
> +void pcie_aer_msg(PCIDevice *dev, const PCIEAERMsg *msg)
>  {
>      uint8_t type;
>  

Better to split the AER core changes to a separate patch.

> diff --git a/hw/vfio/pci.c b/hw/vfio/pci.c
> index 0ee6326..6f0b265 100644
> --- a/hw/vfio/pci.c
> +++ b/hw/vfio/pci.c
> @@ -3106,20 +3106,41 @@ static void vfio_put_device(VFIOPCIDevice *vdev)
>  static void vfio_err_notifier_handler(void *opaque)
>  {
>      VFIOPCIDevice *vdev = opaque;
> +    PCIDevice *dev = &vdev->pdev;
> +    PCIEAERMsg msg = {
> +        .severity = 0,
> +        .source_id = (pci_bus_num(dev->bus) << 8) | dev->devfn,
> +    };
>  
>      if (!event_notifier_test_and_clear(&vdev->err_notifier)) {
>          return;
>      }
>  
> -    /*
> -     * TBD. Retrieve the error details and decide what action
> -     * needs to be taken. One of the actions could be to pass
> -     * the error to the guest and have the guest driver recover
> -     * from the error. This requires that PCIe capabilities be
> -     * exposed to the guest. For now, we just terminate the
> -     * guest to contain the error.
> +    /* we should read the error details from the real hardware
> +     * configration spaces, here we only need to do is signaling

s/configration/configuration/

> +     * to guest an uncorrectable error has occurred.
>       */
> +    if (dev->exp.aer_cap) {
> +        uint8_t *aer_cap = dev->config + dev->exp.aer_cap;
> +        uint32_t uncor_status;
> +        bool isfatal;
> +
> +        uncor_status = vfio_pci_read_config(dev,
> +                           dev->exp.aer_cap + PCI_ERR_UNCOR_STATUS, 4);
> +
> +        isfatal = uncor_status & pci_get_long(aer_cap + PCI_ERR_UNCOR_SEVER);
> +
> +        msg.severity = isfatal ? PCI_ERR_ROOT_CMD_FATAL_EN :
> +                                 PCI_ERR_ROOT_CMD_NONFATAL_EN;
>  
> +        pcie_aer_msg(dev, &msg);
> +        return;

Two things here, first, what happens when the guest attempts to do bus
resets or any sort of operation with the parent downstream port to
recover the device?  Those ops aren't passed through to hardware.  Is a
guest reboot necessarily going to be sufficient to clear the error and
doesn't that depend on what kind of reset vfio can do to the device on
the host?

Second, what happens on 440fx?  It looks like the message goes nowhere,
which causes a regression from our current error containment there.  So
any 440fx machine would need to disable the property that Paolo is
requesting.

> +    }
> +
> +    /*
> +     * If the aer capability is not exposed to the guest. we just
> +     * terminate the guest to contain the error.
> +     */
>      error_report("%s(%04x:%02x:%02x.%x) Unrecoverable error detected.  "
>                   "Please collect any data possible and then kill the guest",
>                   __func__, vdev->host.domain, vdev->host.bus,
> diff --git a/include/hw/pci/pcie_aer.h b/include/hw/pci/pcie_aer.h
> index bcac80a..6c4bf3b 100644
> --- a/include/hw/pci/pcie_aer.h
> +++ b/include/hw/pci/pcie_aer.h
> @@ -51,7 +51,7 @@ struct PCIEAERLog {
>      PCIEAERErr *log;
>  };
>  
> -/* aer error message: error signaling message has only error sevirity and
> +/* aer error message: error signaling message has only error severity and
>     source id. See 2.2.8.3 error signaling messages */
>  struct PCIEAERMsg {
>      /*
> @@ -102,5 +102,6 @@ void pcie_aer_root_write_config(PCIDevice *dev,
>  
>  /* error injection */
>  int pcie_aer_inject_error(PCIDevice *dev, const PCIEAERErr *err);
> +void pcie_aer_msg(PCIDevice *dev, const PCIEAERMsg *msg);
>  
>  #endif /* QEMU_PCIE_AER_H */

Re: [Qemu-devel] [RFC PATCH 3/4] vfio-pci: add aer capability support

[Alex Williamson]

On Mon, 2015-01-12 at 11:04 +0800, Chen Fan wrote:

This patch isn't trivial enough for a blank commit log.  Why do we need
to make those bits emulated?  Do we only care about AER for now?

> Signed-off-by: Chen Fan <chen.fan.fnst@cn.fujitsu.com>
> ---
>  hw/vfio/pci.c | 40 ++++++++++++++++++++++++++++++++++++++++
>  1 file changed, 40 insertions(+)
> 
> diff --git a/hw/vfio/pci.c b/hw/vfio/pci.c
> index b4e73d1..0ee6326 100644
> --- a/hw/vfio/pci.c
> +++ b/hw/vfio/pci.c
> @@ -2667,6 +2667,41 @@ static int vfio_add_capabilities(VFIOPCIDevice *vdev)
>      return vfio_add_std_cap(vdev, pdev->config[PCI_CAPABILITY_LIST]);
>  }
>  
> +static int vfio_add_ext_capabilities(VFIOPCIDevice *vdev)
> +{
> +    PCIDevice *pdev = &vdev->pdev;
> +    PCIExpressDevice *exp;
> +    uint32_t header;
> +    uint16_t next = PCI_CONFIG_SPACE_SIZE;
> +
> +    if (pci_config_size(pdev) <= PCI_CONFIG_SPACE_SIZE) {
> +        return 0;
> +    }
> +
> +    header = pci_get_long(pdev->config + next);
> +    while (header) {
> +        switch (PCI_EXT_CAP_ID(header)) {
> +        case PCI_EXT_CAP_ID_ERR:
> +             exp = &pdev->exp;
> +             exp->aer_cap = next;
> +
> +             /* enable the error report */
> +             vfio_add_emulated_long(vdev, exp->exp_cap + PCI_EXP_DEVCTL,
> +                 PCI_EXP_DEVCTL_CERE | PCI_EXP_DEVCTL_NFERE |
> +                 PCI_EXP_DEVCTL_FERE | PCI_EXP_DEVCTL_URRE, ~0);
> +             break;
> +        };
> +
> +        next = PCI_EXT_CAP_NEXT(header);
> +        if (!next) {
> +            return 0;
> +        }
> +        header = pci_get_long(pdev->config + next);
> +    }
> +
> +    return 0;
> +}
> +
>  static void vfio_pci_pre_reset(VFIOPCIDevice *vdev)
>  {
>      PCIDevice *pdev = &vdev->pdev;
> @@ -3293,6 +3328,11 @@ static int vfio_initfn(PCIDevice *pdev)
>          goto out_teardown;
>      }
>  
> +    ret = vfio_add_ext_capabilities(vdev);
> +    if (ret) {
> +        goto out_teardown;
> +    }
> +
>      /* QEMU emulates all of MSI & MSIX */
>      if (pdev->cap_present & QEMU_PCI_CAP_MSIX) {
>          memset(vdev->emulated_config_bits + pdev->msix_cap, 0xff,

Re: [Qemu-devel] [RFC PATCH 2/4] pcie-aer: Fix command pcie_aer_inject_error is invalid

[Chen Fan]

On 01/12/2015 09:56 PM, Marcel Apfelbaum wrote:
> On 01/12/2015 05:04 AM, Chen Fan wrote:
>> in spec "PCI Express 3.0" section 6.2.6 Figure 6-3 virtual bridge part,
>> the flowchart showing tell us SERR# enable at Bridge Control register
>> associate with system error at Secondary Status register can send error
>> message. but bridge_control from dev->config is NULL, and SERR# was set
>> in dev->wmask in pcie_aer_init()
> wmask denotes the register bits that can be written by the guest.
>
> If you are referring to:
>        pci_word_test_and_set_mask(dev->wmask + PCI_BRIDGE_CONTROL,
>                                   PCI_BRIDGE_CTL_SERR);
> that means that the OS *is able* to turn on/off SERR forwarding.
>
>
>  which was implemented by root port and
>> swith devices, so we should add wmask (for w/r) bit set for bridge 
>> control.
>> we can user command like:
>> qemu_system_x86_64:
>> -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,id=bridge1
>> -device x3130-upstream,bus=bridge1,id=up.1,addr=00.0
>> -device xio3130-downstream,bus=up.1,id=down.1,port=1,addr=00.0,chassis=5
>>
>> (qemu) pcie_aer_inject_error net0 POISON_TLP
>>
>> after that,
>> guest can output the error message.
>>
>> Signed-off-by: Chen Fan <chen.fan.fnst@cn.fujitsu.com>
>> ---
>>   hw/pci/pcie_aer.c | 3 ++-
>>   1 file changed, 2 insertions(+), 1 deletion(-)
>>
>> diff --git a/hw/pci/pcie_aer.c b/hw/pci/pcie_aer.c
>> index 7ca077a..571dc92 100644
>> --- a/hw/pci/pcie_aer.c
>> +++ b/hw/pci/pcie_aer.c
>> @@ -231,7 +231,8 @@ pcie_aer_msg_alldev(PCIDevice *dev, const 
>> PCIEAERMsg *msg)
>>    */
>>   static bool pcie_aer_msg_vbridge(PCIDevice *dev, const PCIEAERMsg 
>> *msg)
>>   {
>> -    uint16_t bridge_control = pci_get_word(dev->config + 
>> PCI_BRIDGE_CONTROL);
> Here we check if the Guest OS/firmware actually turned the #SERR 
> forwarding on.
>
>> +    uint16_t bridge_control = pci_get_word(dev->config + 
>> PCI_BRIDGE_CONTROL) |
>> +                              pci_get_word(dev->wmask + 
>> PCI_BRIDGE_CONTROL);
> I don't think that this check is correct given the above comments.
> Please correct me if I mislead you,
after sweep the code again,  I think you are right.

And for pcie spec  6.2.5 chapter. I think we should add a check for
validating the "Fatal Error Reporting Enable" bit in Device Control register
or whether #SERR is enabled either.

Thanks,
Chen



> Thanks,
> Marcel
>
>
>>
>>       if (pcie_aer_msg_is_uncor(msg)) {
>>           /* Received System Error */
>>
>
> .
>

Re: [Qemu-devel] [RFC PATCH 3/4] vfio-pci: add aer capability support

[Chen Fan]

On 01/12/2015 11:26 PM, Alex Williamson wrote:
> On Mon, 2015-01-12 at 11:04 +0800, Chen Fan wrote:
>
> This patch isn't trivial enough for a blank commit log.  Why do we need
> to make those bits emulated?  Do we only care about AER for now?
I think the vfio extend capabilities control registers should be 
manipulated by qemu self.
BTW, it is guest driver's responsibility to set PCI_EXP_DEVCTL bits, 
right? so here
only need to initialize configration sapces and set corresponding bits 
attributes.
I will change it in my next version patches.

I think AER is a good start.

Thanks,
Chen
>
>> Signed-off-by: Chen Fan <chen.fan.fnst@cn.fujitsu.com>
>> ---
>>   hw/vfio/pci.c | 40 ++++++++++++++++++++++++++++++++++++++++
>>   1 file changed, 40 insertions(+)
>>
>> diff --git a/hw/vfio/pci.c b/hw/vfio/pci.c
>> index b4e73d1..0ee6326 100644
>> --- a/hw/vfio/pci.c
>> +++ b/hw/vfio/pci.c
>> @@ -2667,6 +2667,41 @@ static int vfio_add_capabilities(VFIOPCIDevice *vdev)
>>       return vfio_add_std_cap(vdev, pdev->config[PCI_CAPABILITY_LIST]);
>>   }
>>   
>> +static int vfio_add_ext_capabilities(VFIOPCIDevice *vdev)
>> +{
>> +    PCIDevice *pdev = &vdev->pdev;
>> +    PCIExpressDevice *exp;
>> +    uint32_t header;
>> +    uint16_t next = PCI_CONFIG_SPACE_SIZE;
>> +
>> +    if (pci_config_size(pdev) <= PCI_CONFIG_SPACE_SIZE) {
>> +        return 0;
>> +    }
>> +
>> +    header = pci_get_long(pdev->config + next);
>> +    while (header) {
>> +        switch (PCI_EXT_CAP_ID(header)) {
>> +        case PCI_EXT_CAP_ID_ERR:
>> +             exp = &pdev->exp;
>> +             exp->aer_cap = next;
>> +
>> +             /* enable the error report */
>> +             vfio_add_emulated_long(vdev, exp->exp_cap + PCI_EXP_DEVCTL,
>> +                 PCI_EXP_DEVCTL_CERE | PCI_EXP_DEVCTL_NFERE |
>> +                 PCI_EXP_DEVCTL_FERE | PCI_EXP_DEVCTL_URRE, ~0);
>> +             break;
>> +        };
>> +
>> +        next = PCI_EXT_CAP_NEXT(header);
>> +        if (!next) {
>> +            return 0;
>> +        }
>> +        header = pci_get_long(pdev->config + next);
>> +    }
>> +
>> +    return 0;
>> +}
>> +
>>   static void vfio_pci_pre_reset(VFIOPCIDevice *vdev)
>>   {
>>       PCIDevice *pdev = &vdev->pdev;
>> @@ -3293,6 +3328,11 @@ static int vfio_initfn(PCIDevice *pdev)
>>           goto out_teardown;
>>       }
>>   
>> +    ret = vfio_add_ext_capabilities(vdev);
>> +    if (ret) {
>> +        goto out_teardown;
>> +    }
>> +
>>       /* QEMU emulates all of MSI & MSIX */
>>       if (pdev->cap_present & QEMU_PCI_CAP_MSIX) {
>>           memset(vdev->emulated_config_bits + pdev->msix_cap, 0xff,
>
>
> .
>

Re: [Qemu-devel] [RFC PATCH 3/4] vfio-pci: add aer capability support

[Chen Fan]

On 01/12/2015 09:14 PM, Paolo Bonzini wrote:
>
> On 12/01/2015 04:04, Chen Fan wrote:
>> +static int vfio_add_ext_capabilities(VFIOPCIDevice *vdev)
>> +{
>> +    PCIDevice *pdev = &vdev->pdev;
>> +    PCIExpressDevice *exp;
>> +    uint32_t header;
>> +    uint16_t next = PCI_CONFIG_SPACE_SIZE;
>> +
>> +    if (pci_config_size(pdev) <= PCI_CONFIG_SPACE_SIZE) {
>> +        return 0;
>> +    }
>> +
>> +    header = pci_get_long(pdev->config + next);
>> +    while (header) {
>> +        switch (PCI_EXT_CAP_ID(header)) {
>> +        case PCI_EXT_CAP_ID_ERR:
>> +             exp = &pdev->exp;
>> +             exp->aer_cap = next;
>> +
>> +             /* enable the error report */
>> +             vfio_add_emulated_long(vdev, exp->exp_cap + PCI_EXP_DEVCTL,
>> +                 PCI_EXP_DEVCTL_CERE | PCI_EXP_DEVCTL_NFERE |
>> +                 PCI_EXP_DEVCTL_FERE | PCI_EXP_DEVCTL_URRE, ~0);
>> +             break;
>> +        };
>> +
>> +        next = PCI_EXT_CAP_NEXT(header);
>> +        if (!next) {
>> +            return 0;
>> +        }
>> +        header = pci_get_long(pdev->config + next);
>> +    }
>> +
>> +    return 0;
>> +}
>> +
> Please add a property to the VFIO device, defaulting to true, and
> disable it for older machine types.
Thanks for your suggestion.

Chen
>
> Paolo
>

Re: [Qemu-devel] [RFC PATCH 2/4] pcie-aer: Fix command pcie_aer_inject_error is invalid

[Chen Fan]

On 01/12/2015 09:56 PM, Marcel Apfelbaum wrote:
> On 01/12/2015 05:04 AM, Chen Fan wrote:
>> in spec "PCI Express 3.0" section 6.2.6 Figure 6-3 virtual bridge part,
>> the flowchart showing tell us SERR# enable at Bridge Control register
>> associate with system error at Secondary Status register can send error
>> message. but bridge_control from dev->config is NULL, and SERR# was set
>> in dev->wmask in pcie_aer_init()
> wmask denotes the register bits that can be written by the guest.
>
> If you are referring to:
>        pci_word_test_and_set_mask(dev->wmask + PCI_BRIDGE_CONTROL,
>                                   PCI_BRIDGE_CTL_SERR);
> that means that the OS *is able* to turn on/off SERR forwarding.
Hi marcel,

I saw the OS that turn on SERR# is to via evaluate _HPP (Hot Plug 
Parameters) method in ACPI.
is it the only way to turn on SERR#?

Thanks,
Chen

>
>
>  which was implemented by root port and
>> swith devices, so we should add wmask (for w/r) bit set for bridge 
>> control.
>> we can user command like:
>> qemu_system_x86_64:
>> -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,id=bridge1
>> -device x3130-upstream,bus=bridge1,id=up.1,addr=00.0
>> -device xio3130-downstream,bus=up.1,id=down.1,port=1,addr=00.0,chassis=5
>>
>> (qemu) pcie_aer_inject_error net0 POISON_TLP
>>
>> after that,
>> guest can output the error message.
>>
>> Signed-off-by: Chen Fan <chen.fan.fnst@cn.fujitsu.com>
>> ---
>>   hw/pci/pcie_aer.c | 3 ++-
>>   1 file changed, 2 insertions(+), 1 deletion(-)
>>
>> diff --git a/hw/pci/pcie_aer.c b/hw/pci/pcie_aer.c
>> index 7ca077a..571dc92 100644
>> --- a/hw/pci/pcie_aer.c
>> +++ b/hw/pci/pcie_aer.c
>> @@ -231,7 +231,8 @@ pcie_aer_msg_alldev(PCIDevice *dev, const 
>> PCIEAERMsg *msg)
>>    */
>>   static bool pcie_aer_msg_vbridge(PCIDevice *dev, const PCIEAERMsg 
>> *msg)
>>   {
>> -    uint16_t bridge_control = pci_get_word(dev->config + 
>> PCI_BRIDGE_CONTROL);
> Here we check if the Guest OS/firmware actually turned the #SERR 
> forwarding on.
>
>> +    uint16_t bridge_control = pci_get_word(dev->config + 
>> PCI_BRIDGE_CONTROL) |
>> +                              pci_get_word(dev->wmask + 
>> PCI_BRIDGE_CONTROL);
> I don't think that this check is correct given the above comments.
> Please correct me if I mislead you,
> Thanks,
> Marcel
>
>
>>
>>       if (pcie_aer_msg_is_uncor(msg)) {
>>           /* Received System Error */
>>
>
> .
>

Re: [Qemu-devel] [RFC PATCH 2/4] pcie-aer: Fix command pcie_aer_inject_error is invalid

[Chen Fan]

[-- Attachment #1: Type: text/plain, Size: 2749 bytes --]


On 01/16/2015 03:56 PM, Chen Fan wrote:
>
> On 01/12/2015 09:56 PM, Marcel Apfelbaum wrote:
>> On 01/12/2015 05:04 AM, Chen Fan wrote:
>>> in spec "PCI Express 3.0" section 6.2.6 Figure 6-3 virtual bridge part,
>>> the flowchart showing tell us SERR# enable at Bridge Control register
>>> associate with system error at Secondary Status register can send error
>>> message. but bridge_control from dev->config is NULL, and SERR# was set
>>> in dev->wmask in pcie_aer_init()
>> wmask denotes the register bits that can be written by the guest.
>>
>> If you are referring to:
>>        pci_word_test_and_set_mask(dev->wmask + PCI_BRIDGE_CONTROL,
>>                                   PCI_BRIDGE_CTL_SERR);
>> that means that the OS *is able* to turn on/off SERR forwarding.
> Hi marcel,
>
> I saw the OS that turn on SERR# is to via evaluate _HPP (Hot Plug 
> Parameters) method in ACPI.
> is it the only way to turn on SERR#?

I saw there was one option called*//*PCI SERR# Generation **searched 
from web pages in firmware on hardware****.
Do it turn on the SERR#? if so, we can enable it in seabios.

Thanks,
Chen

>
> Thanks,
> Chen
>
>>
>>
>>  which was implemented by root port and
>>> swith devices, so we should add wmask (for w/r) bit set for bridge 
>>> control.
>>> we can user command like:
>>> qemu_system_x86_64:
>>> -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,id=bridge1
>>> -device x3130-upstream,bus=bridge1,id=up.1,addr=00.0
>>> -device 
>>> xio3130-downstream,bus=up.1,id=down.1,port=1,addr=00.0,chassis=5
>>>
>>> (qemu) pcie_aer_inject_error net0 POISON_TLP
>>>
>>> after that,
>>> guest can output the error message.
>>>
>>> Signed-off-by: Chen Fan <chen.fan.fnst@cn.fujitsu.com>
>>> ---
>>>   hw/pci/pcie_aer.c | 3 ++-
>>>   1 file changed, 2 insertions(+), 1 deletion(-)
>>>
>>> diff --git a/hw/pci/pcie_aer.c b/hw/pci/pcie_aer.c
>>> index 7ca077a..571dc92 100644
>>> --- a/hw/pci/pcie_aer.c
>>> +++ b/hw/pci/pcie_aer.c
>>> @@ -231,7 +231,8 @@ pcie_aer_msg_alldev(PCIDevice *dev, const 
>>> PCIEAERMsg *msg)
>>>    */
>>>   static bool pcie_aer_msg_vbridge(PCIDevice *dev, const PCIEAERMsg 
>>> *msg)
>>>   {
>>> -    uint16_t bridge_control = pci_get_word(dev->config + 
>>> PCI_BRIDGE_CONTROL);
>> Here we check if the Guest OS/firmware actually turned the #SERR 
>> forwarding on.
>>
>>> +    uint16_t bridge_control = pci_get_word(dev->config + 
>>> PCI_BRIDGE_CONTROL) |
>>> +                              pci_get_word(dev->wmask + 
>>> PCI_BRIDGE_CONTROL);
>> I don't think that this check is correct given the above comments.
>> Please correct me if I mislead you,
>> Thanks,
>> Marcel
>>
>>
>>>
>>>       if (pcie_aer_msg_is_uncor(msg)) {
>>>           /* Received System Error */
>>>
>>
>> .
>>
>
>
>


[-- Attachment #2: Type: text/html, Size: 4977 bytes --]

Re: [Qemu-devel] [RFC PATCH 2/4] pcie-aer: Fix command pcie_aer_inject_error is invalid

[Marcel Apfelbaum]

On 01/21/2015 11:56 AM, Chen Fan wrote:
>
> On 01/16/2015 03:56 PM, Chen Fan wrote:
>>
>> On 01/12/2015 09:56 PM, Marcel Apfelbaum wrote:
>>> On 01/12/2015 05:04 AM, Chen Fan wrote:
>>>> in spec "PCI Express 3.0" section 6.2.6 Figure 6-3 virtual bridge part,
>>>> the flowchart showing tell us SERR# enable at Bridge Control register
>>>> associate with system error at Secondary Status register can send error
>>>> message. but bridge_control from dev->config is NULL, and SERR# was set
>>>> in dev->wmask in pcie_aer_init()
>>> wmask denotes the register bits that can be written by the guest.
>>>
>>> If you are referring to:
>>>        pci_word_test_and_set_mask(dev->wmask + PCI_BRIDGE_CONTROL,
>>>                                   PCI_BRIDGE_CTL_SERR);
>>> that means that the OS *is able* to turn on/off SERR forwarding.
>> Hi marcel,
>>
>> I saw the OS that turn on SERR# is to via evaluate _HPP (Hot Plug Parameters) method in ACPI.
>>   it the only way to turn on SERR#?
This is strange, I don't see how it is connected.

>
> I saw there was one option do it, called*//*PCI SERR# Generation **searched from web pages in firmware on hardware****.
> Do it turn on the SERR#? if so, we can enable it in seabios.
Indeed, OS/firmware are in charge of setting this bit.
We *could* do it in BIOS, but not before checking how the OS is handling it
I suggest checking the pci(e) bridge initialization code in Linux Kernel and only
then decide how to proceed.
You can also look(grep) for this ...CTL_ERR in the kernel code
and try to figure that out.

Thanks,
Marcel

> Thanks,
> Chen
>
[...]

Re: [Qemu-devel] [RFC PATCH 4/4] vfio-pci: pass the aer error to guest

[Chen Fan]

On 01/12/2015 11:24 PM, Alex Williamson wrote:
> On Mon, 2015-01-12 at 11:04 +0800, Chen Fan wrote:
>> when the vfio device encounters an uncorrectable error in host,
>>   the vfio_pci driver will signal the eventfd registered by this
>>   vfio device, the results in the qemu eventfd handler getting
>>   invoked.
>>
>> this patch is to pass the error to guest and have the guest driver
>> recover from the error.
>>
>> Signed-off-by: Chen Fan <chen.fan.fnst@cn.fujitsu.com>
>> ---
>>   hw/pci/pcie_aer.c         |  2 +-
>>   hw/vfio/pci.c             | 35 ++++++++++++++++++++++++++++-------
>>   include/hw/pci/pcie_aer.h |  3 ++-
>>   3 files changed, 31 insertions(+), 9 deletions(-)
>>
>> diff --git a/hw/pci/pcie_aer.c b/hw/pci/pcie_aer.c
>> index 571dc92..4812e3d 100644
>> --- a/hw/pci/pcie_aer.c
>> +++ b/hw/pci/pcie_aer.c
>> @@ -371,7 +371,7 @@ static void pcie_aer_msg_root_port(PCIDevice *dev, const PCIEAERMsg *msg)
>>    *
>>    * Walk up the bus tree from the device, propagate the error message.
>>    */
>> -static void pcie_aer_msg(PCIDevice *dev, const PCIEAERMsg *msg)
>> +void pcie_aer_msg(PCIDevice *dev, const PCIEAERMsg *msg)
>>   {
>>       uint8_t type;
>>   
> Better to split the AER core changes to a separate patch.
>
>> diff --git a/hw/vfio/pci.c b/hw/vfio/pci.c
>> index 0ee6326..6f0b265 100644
>> --- a/hw/vfio/pci.c
>> +++ b/hw/vfio/pci.c
>> @@ -3106,20 +3106,41 @@ static void vfio_put_device(VFIOPCIDevice *vdev)
>>   static void vfio_err_notifier_handler(void *opaque)
>>   {
>>       VFIOPCIDevice *vdev = opaque;
>> +    PCIDevice *dev = &vdev->pdev;
>> +    PCIEAERMsg msg = {
>> +        .severity = 0,
>> +        .source_id = (pci_bus_num(dev->bus) << 8) | dev->devfn,
>> +    };
>>   
>>       if (!event_notifier_test_and_clear(&vdev->err_notifier)) {
>>           return;
>>       }
>>   
>> -    /*
>> -     * TBD. Retrieve the error details and decide what action
>> -     * needs to be taken. One of the actions could be to pass
>> -     * the error to the guest and have the guest driver recover
>> -     * from the error. This requires that PCIe capabilities be
>> -     * exposed to the guest. For now, we just terminate the
>> -     * guest to contain the error.
>> +    /* we should read the error details from the real hardware
>> +     * configration spaces, here we only need to do is signaling
> s/configration/configuration/
>
>> +     * to guest an uncorrectable error has occurred.
>>        */
>> +    if (dev->exp.aer_cap) {
>> +        uint8_t *aer_cap = dev->config + dev->exp.aer_cap;
>> +        uint32_t uncor_status;
>> +        bool isfatal;
>> +
>> +        uncor_status = vfio_pci_read_config(dev,
>> +                           dev->exp.aer_cap + PCI_ERR_UNCOR_STATUS, 4);
>> +
>> +        isfatal = uncor_status & pci_get_long(aer_cap + PCI_ERR_UNCOR_SEVER);
>> +
>> +        msg.severity = isfatal ? PCI_ERR_ROOT_CMD_FATAL_EN :
>> +                                 PCI_ERR_ROOT_CMD_NONFATAL_EN;
>>   
>> +        pcie_aer_msg(dev, &msg);
>> +        return;
Hi Alex,

> Two things here, first, what happens when the guest attempts to do bus
> resets or any sort of operation with the parent downstream port to
> recover the device?  Those ops aren't passed through to hardware.  Is a
> guest reboot necessarily going to be sufficient to clear the error and
> doesn't that depend on what kind of reset vfio can do to the device on
> the host?
for now, the supported aer capability bridges are ioh3420, and 
upstream/downstream,
when need to reset link for fatal error, which will cause these device 
reset by trigger secondary bus reset,
then do qbus_reset_all(): which will trigger vfio_pci_reset to reset 
vfio device,
in vfio_pci_reset, vfio device will via ioctl to notify hardware to 
reset by method VFIO_DEVICE_RESET.
I think it seems be ok.

>
> Second, what happens on 440fx?  It looks like the message goes nowhere,
> which causes a regression from our current error containment there.  So
> any 440fx machine would need to disable the property that Paolo is
> requesting.
For 440fx, it don't support pcie root port, so there will not trigger
aer report in qemu.

Thanks,
Chen

>> +    }
>> +
>> +    /*
>> +     * If the aer capability is not exposed to the guest. we just
>> +     * terminate the guest to contain the error.
>> +     */
>>       error_report("%s(%04x:%02x:%02x.%x) Unrecoverable error detected.  "
>>                    "Please collect any data possible and then kill the guest",
>>                    __func__, vdev->host.domain, vdev->host.bus,
>> diff --git a/include/hw/pci/pcie_aer.h b/include/hw/pci/pcie_aer.h
>> index bcac80a..6c4bf3b 100644
>> --- a/include/hw/pci/pcie_aer.h
>> +++ b/include/hw/pci/pcie_aer.h
>> @@ -51,7 +51,7 @@ struct PCIEAERLog {
>>       PCIEAERErr *log;
>>   };
>>   
>> -/* aer error message: error signaling message has only error sevirity and
>> +/* aer error message: error signaling message has only error severity and
>>      source id. See 2.2.8.3 error signaling messages */
>>   struct PCIEAERMsg {
>>       /*
>> @@ -102,5 +102,6 @@ void pcie_aer_root_write_config(PCIDevice *dev,
>>   
>>   /* error injection */
>>   int pcie_aer_inject_error(PCIDevice *dev, const PCIEAERErr *err);
>> +void pcie_aer_msg(PCIDevice *dev, const PCIEAERMsg *msg);
>>   
>>   #endif /* QEMU_PCIE_AER_H */
>
>
> .
>