Re: [PATCH v13 0/7] hot-unplug failure handle mechanism

["Ananyev, Konstantin" <konstantin.ananyev@intel.com>]

> -----Original Message-----
> From: Guo, Jia
> Sent: Thursday, October 4, 2018 7:31 AM
> To: stephen@networkplumber.org; Richardson, Bruce <bruce.richardson@intel.com>; Yigit, Ferruh <ferruh.yigit@intel.com>; Ananyev,
> Konstantin <konstantin.ananyev@intel.com>; gaetan.rivet@6wind.com; Wu, Jingjing <jingjing.wu@intel.com>; thomas@monjalon.net;
> motih@mellanox.com; matan@mellanox.com; Van Haaren, Harry <harry.van.haaren@intel.com>; Zhang, Qi Z <qi.z.zhang@intel.com>; He,
> Shaopeng <shaopeng.he@intel.com>; Iremonger, Bernard <bernard.iremonger@intel.com>; arybchenko@solarflare.com; Lu, Wenzhuo
> <wenzhuo.lu@intel.com>; Burakov, Anatoly <anatoly.burakov@intel.com>; jerin.jacob@caviumnetworks.com
> Cc: jblunck@infradead.org; shreyansh.jain@nxp.com; dev@dpdk.org; Guo, Jia <jia.guo@intel.com>; Zhang, Helin <helin.zhang@intel.com>
> Subject: [PATCH v13 0/7] hot-unplug failure handle mechanism
> 
> Hotplug is an important feature for use-cases like the datacenter device's
> fail-safe and for SRIOV Live Migration in SDN/NFV. It could bring higher
> flexibility and continuality to networking services in multiple use-cases
> in the industry. So let's see how DPDK can help users implement hotplug
> solutions.
> 
> We already have a general device-event monitor mechanism, failsafe driver,
> and hot plug/unplug API in DPDK. We have already got the solution of
> “ethdev event + kernel PMD hotplug handler + failsafe”, but we still not
> got “eal event + hotplug handler for pci PMD + failsafe” implement, and we
> need to considerate 2 different solutions between uio pci and vfio pci.
> 
> In the case of hotplug for igb_uio, when a hardware device be removed
> physically or disabled in software, the application needs to be notified
> and detach the device out of the bus, and then make the device invalidate.
> The problem is that, the removal of the device is not instantaneous in
> software. If the application data path tries to read/write to the device
> when removal is still in process, it will cause an MMIO error and
> application will crash.
> 
> In this patch set, we propose a PCIe bus failure handler mechanism for
> hot-unplug in igb_uio. It aims to guarantee that, when a hot-unplug occurs,
> the application will not crash.
> 
> The mechanism should work as below:
> 
> First, the application enables the device event monitor, registers the
> hotplug event’s callback and enable hotplug handling before running the
> data path. Once the hot-unplug occurs, the mechanism will detect the
> removal event and then accordingly do the failure handling. In order to
> do that, the below functionality will be required:
>  - Add a new bus ops “hot_unplug_handler” to handle hot-unplug failure.
>  - Implement pci bus specific ops “pci_hot_unplug_handler”. For uio pci,
>    it will be based on the failure address to remap memory for the corresponding
>    device that unplugged. For vfio pci, could seperate implement case by case.
> 
> For the data path or other unexpected behaviors from the control path
> when a hot unplug occurs:
>  - Add a new bus ops “sigbus_handler”, that is responsible for handling
>    the sigbus error which is either an original memory error, or a specific
>    memory error that is caused by a hot unplug. When a sigbus error is
>    captured, it will call this function to handle sigbus error.
>  - Implement PCI bus specific ops “pci_sigbus_handler”. It will iterate all
>    device on PCI bus to find which device encounter the failure.
>  - Implement a "rte_bus_sigbus_handler" to iterate all buses to find a bus
>    to handle the failure.
>  - Add a couple of APIs “rte_dev_hotplug_handle_enable” and
>    “rte_dev_hotplug_handle_diable” to enable/disable hotplug handling.
>    It will monitor the sigbus error by a handler which is per-process.
>    Based on the signal event principle, the control path thread and the
>    data path thread will randomly receive the sigbus error, but will call the
>    common sigbus process. When sigbus be captured, it will call the above API
>    to find bus to handle it.
> 
> The mechanism could be used by app or PMDs. For example, the whole process
> of hotplug in testpmd is:
>  - Enable device event monitor->Enable hotplug handle->Register event callback
>    ->attach port->start port->start forwarding->Device unplug->failure handle
>    ->stop forwarding->stop port->close port->detach port.
> 
> This patch set would not cover hotplug insert and binding, and it is only
> implement the igb_uio failure handler, the vfio hotplug failure handler
> will be in next coming patch set.
> 
> patchset history:
> v13->v12:
> use local variable to rewrite the func to be more readable.
> add sa_flag check when invoke generic sigbus handler
> modify some typo
> delete needless helper in app
> 
> v12->v11:
> add and delete some checking about sigbus recover.
> 
> v11->v10:
> change the ops name, since both uio and vfio will use the hot-unplug ops.
> since we plan to abandon RTE_ETH_EVENT_INTR_RMV, change to use
> RTE_DEV_EVENT_REMOVE, so modify the hotplug event and callback usage.
> move the igb_uio fixing part, since it is random issue and should be considarate
> as kernel driver defect but not include as this failure handler mechanism.
> 
> v10->v9:
> modify the api name and exposure out for public use.
> add hotplug handle enable/disable APIs
> refine commit log
> 
> v9->v8:
> refine commit log to be more readable.
> 
> v8->v7:
> refine errno process in sigbus handler.
> refine igb uio release process
> 
> v7->v6:
> delete some unused part
> 
> v6->v5:
> refine some description about bus ops
> refine commit log
> add some entry check.
> 
> v5->v4:
> split patches to focus on the failure handle, remove the event usage
> by testpmd to another patch.
> change the hotplug failure handler name.
> refine the sigbus handle logic.
> add lock for udev state in igb uio driver.
> 
> v4->v3:
> split patches to be small and clear.
> change to use new parameter "--hotplug-mode" in testpmd to identify
> the eal hotplug and ethdev hotplug.
> 
> v3->v2:
> change bus ops name to bus_hotplug_handler.
> add new API and bus ops of bus_signal_handler distingush handle generic.
> sigbus and hotplug sigbus.
> 
> v2->v1(v21):
> refine some doc and commit log.
> fix igb uio kernel issue for control path failure rebase testpmd code.
> 
> Since the hot plug solution be discussed serval around in the public,
> the scope be changed and the patch set be split into many times. Coming
> to the recently RFC and feature design, it just focus on the hot unplug
> failure handler at this patch set, so in order let this topic more clear
> and focus, summarize privours patch set in history “v1(v21)”, the v2 here
> go ahead for further track.
> 
> "v1(21)" == v21 as below:
> v21->v20:
> split function in hot unplug ops.
> sync failure hanlde to fix multiple process issue fix attach port issue for multiple devices case.
> combind rmv callback function to be only one.
> 
> v20->v19:
> clean the code.
> refine the remap logic for multiple device.
> remove the auto binding.
> 
> v19->18:
> note for limitation of multiple hotplug, fix some typo, sqeeze patch.
> 
> v18->v15:
> add document, add signal bus handler, refine the code to be more clear.
> 
> the prior patch history please check the patch set "add device event monitor framework".
> 
> Jeff Guo (7):
>   bus: add hot-unplug handler
>   bus/pci: implement hot-unplug handler ops
>   bus: add sigbus handler
>   bus/pci: implement sigbus handler ops
>   bus: add helper to handle sigbus
>   eal: add failure handle mechanism for hot-unplug
>   app/testpmd: use hotplug failure handler
> 
>  app/test-pmd/testpmd.c                  |  86 ++++++++--------
>  doc/guides/rel_notes/release_18_08.rst  |   5 +
>  drivers/bus/pci/pci_common.c            |  82 +++++++++++++++
>  drivers/bus/pci/pci_common_uio.c        |  33 +++++++
>  drivers/bus/pci/private.h               |  12 +++
>  lib/librte_eal/bsdapp/eal/eal_dev.c     |  14 +++
>  lib/librte_eal/common/eal_common_bus.c  |  43 ++++++++
>  lib/librte_eal/common/eal_private.h     |  39 ++++++++
>  lib/librte_eal/common/include/rte_bus.h |  34 +++++++
>  lib/librte_eal/common/include/rte_dev.h |  26 +++++
>  lib/librte_eal/linuxapp/eal/eal_dev.c   | 170 +++++++++++++++++++++++++++++++-
>  lib/librte_eal/rte_eal_version.map      |   2 +
>  12 files changed, 499 insertions(+), 47 deletions(-)
> 
> --

Acked-by: Konstantin Ananyev <konstantin.ananyev@intel.com>
> 2.7.4