DPDK-dev Archive on lore.kernel.org
 help / color / Atom feed
From: David Marchand <david.marchand@redhat.com>
To: Nithin Dabilpuram <ndabilpuram@marvell.com>
Cc: Hyong Youb Kim <hyonkim@cisco.com>,
	Thomas Monjalon <thomas@monjalon.net>,
	 Ferruh Yigit <ferruh.yigit@intel.com>,
	Bruce Richardson <bruce.richardson@intel.com>,
	 Jerin Jacob Kollanukkaran <jerinj@marvell.com>,
	John Daley <johndale@cisco.com>,
	Shahed Shaikh <shshaikh@marvell.com>, dev <dev@dpdk.org>
Subject: Re: [dpdk-dev] [PATCH v4 2/3] eal: add ack interrupt API
Date: Mon, 22 Jul 2019 21:28:33 +0200
Message-ID: <CAJFAV8ySX=a2SgVdGtwwrEGRMuNWvCeMxA3LonCB23wJAQ-M=g@mail.gmail.com> (raw)
In-Reply-To: <20190718143628.169623-3-ndabilpuram@marvell.com>

On Thu, Jul 18, 2019 at 4:36 PM Nithin Dabilpuram
<ndabilpuram@marvell.com> wrote:
>
> Add new ack interrupt API to avoid using
> VFIO_IRQ_SET_ACTION_TRIGGER(rte_intr_enable()) for
> acking interrupt purpose for VFIO based interrupt handlers.
> This implementation is specific to Linux.
>
> Using rte_intr_enable() for acking interrupt has below issues
>
>  * Time consuming to do for every interrupt received as it will
>    free_irq() followed by request_irq() and all other initializations
>  * A race condition because of a window between free_irq() and
>    request_irq() with packet reception still on and device still
>    enabled and would throw warning messages like below.
>    [158764.159833] do_IRQ: 9.34 No irq handler for vector
>
> In this patch, rte_intr_ack() is a no-op for VFIO_MSIX/VFIO_MSI interrupts
> as they are edge triggered and kernel would not mask the interrupt before
> delivering the event to userspace and we don't need to ack.
>
> Signed-off-by: Nithin Dabilpuram <ndabilpuram@marvell.com>
> Signed-off-by: Jerin Jacob <jerinj@marvell.com>
> Tested-by: Shahed Shaikh <shshaikh@marvell.com>
> ---
> v4:
> * Move note to implementation and change
>   the expectation to must call for new api.
> v3:
> * Update note on new api
> v2:
> * No change
>
>  lib/librte_eal/common/include/rte_interrupts.h | 18 ++++++
>  lib/librte_eal/freebsd/eal/eal_interrupts.c    |  9 +++
>  lib/librte_eal/linux/eal/eal_interrupts.c      | 90 ++++++++++++++++++++++++++
>  lib/librte_eal/rte_eal_version.map             |  1 +
>  4 files changed, 118 insertions(+)
>
> diff --git a/lib/librte_eal/common/include/rte_interrupts.h b/lib/librte_eal/common/include/rte_interrupts.h
> index c1e912c..c463265 100644
> --- a/lib/librte_eal/common/include/rte_interrupts.h
> +++ b/lib/librte_eal/common/include/rte_interrupts.h
> @@ -118,6 +118,24 @@ int rte_intr_enable(const struct rte_intr_handle *intr_handle);
>   */
>  int rte_intr_disable(const struct rte_intr_handle *intr_handle);
>
> +/**

Missing a banner.
 * @warning
 * @b EXPERIMENTAL: this API may change without prior notice

> + * It acks an interrupt raised for the specified handle.

acknowledges

> + *
> + * This function should be called at the end of each interrupt
> + * handler either from application or driver, so that
> + * currently raised interrupt is acked and further
> + * new interrupts are raised.
> + *
> + * @param intr_handle
> + *  pointer to the interrupt handle.
> + *
> + * @return
> + *  - On success, zero.
> + *  - On failure, a negative value.
> + */
> +__rte_experimental
> +int rte_intr_ack(const struct rte_intr_handle *intr_handle);
> +
>  #ifdef __cplusplus
>  }
>  #endif
> diff --git a/lib/librte_eal/freebsd/eal/eal_interrupts.c b/lib/librte_eal/freebsd/eal/eal_interrupts.c
> index 10375bd..f6831b7 100644
> --- a/lib/librte_eal/freebsd/eal/eal_interrupts.c
> +++ b/lib/librte_eal/freebsd/eal/eal_interrupts.c
> @@ -387,6 +387,15 @@ rte_intr_disable(const struct rte_intr_handle *intr_handle)
>         return 0;
>  }
>
> +int
> +rte_intr_ack(const struct rte_intr_handle *intr_handle)
> +{
> +       if (intr_handle && intr_handle->type == RTE_INTR_HANDLE_VDEV)
> +               return 0;
> +
> +       return -1;
> +}
> +
>  static void
>  eal_intr_process_interrupts(struct kevent *events, int nfds)
>  {
> diff --git a/lib/librte_eal/linux/eal/eal_interrupts.c b/lib/librte_eal/linux/eal/eal_interrupts.c
> index 79ad5e8..794374e 100644
> --- a/lib/librte_eal/linux/eal/eal_interrupts.c
> +++ b/lib/librte_eal/linux/eal/eal_interrupts.c
> @@ -197,6 +197,35 @@ vfio_disable_intx(const struct rte_intr_handle *intr_handle) {
>         return 0;
>  }
>
> +/* unmask/ack legacy (INTx) interrupts */
> +static int
> +vfio_ack_intx(const struct rte_intr_handle *intr_handle)
> +{
> +       struct vfio_irq_set *irq_set;
> +       char irq_set_buf[IRQ_SET_BUF_LEN];
> +       int len, ret;
> +
> +       len = sizeof(struct vfio_irq_set);
> +
> +       /* unmask INTx */
> +       irq_set = (struct vfio_irq_set *) irq_set_buf;
> +       memset(irq_set, 0, len);
> +       irq_set->argsz = len;
> +       irq_set->count = 1;
> +       irq_set->flags = VFIO_IRQ_SET_DATA_NONE | VFIO_IRQ_SET_ACTION_UNMASK;
> +       irq_set->index = VFIO_PCI_INTX_IRQ_INDEX;
> +       irq_set->start = 0;
> +
> +       ret = ioctl(intr_handle->vfio_dev_fd, VFIO_DEVICE_SET_IRQS, irq_set);
> +
> +       if (ret) {
> +               RTE_LOG(ERR, EAL, "Error unmasking INTx interrupts for fd %d\n",
> +                       intr_handle->fd);
> +               return -1;
> +       }
> +       return 0;
> +}
> +

We are not setting any fd.
No need to have the irq_set_buf[] array, you can directly declare
irq_set on the stack and remove the len and ret variables.

Something like:

/* unmask/ack legacy (INTx) interrupts */
static int
vfio_ack_intx(const struct rte_intr_handle *intr_handle)
{
        struct vfio_irq_set irq_set;

        /* unmask INTx */
        memset(&irq_set, 0, sizeof(irq_set));
        irq_set.argsz = sizeof(irq_set);
        irq_set.count = 1;
        irq_set.flags = VFIO_IRQ_SET_DATA_NONE | VFIO_IRQ_SET_ACTION_UNMASK;
        irq_set.index = VFIO_PCI_INTX_IRQ_INDEX;
        irq_set.start = 0;

        if (ioctl(intr_handle->vfio_dev_fd, VFIO_DEVICE_SET_IRQS, &irq_set)) {
                RTE_LOG(ERR, EAL, "Error unmasking INTx interrupts for fd %d\n",
                        intr_handle->fd);
                return -1;
        }
        return 0;
}

>  /* enable MSI interrupts */
>  static int
>  vfio_enable_msi(const struct rte_intr_handle *intr_handle) {
> @@ -693,6 +722,67 @@ rte_intr_enable(const struct rte_intr_handle *intr_handle)
>         return 0;
>  }
>
> +

Double empty line.

> +/**
> + *  PMD generally calls this function at the end of its IRQ callback.
> + *  Internally, it unmasks the interrupt if possible. For INTx, unmasking
> + *  is required as the interrupt is auto-masked prior to invoking
> + *  callback. For MSI/MSI-X, unmasking is typically not needed as the
> + *  interrupt is not auto-masked. In fact, for interrupt handle types
> + *  VFIO_MSIX and VFIO_MSI, this function is no-op.

No double space at the beginning of the lines.
How about separating the cases, like:

/**
 * PMD generally calls this function at the end of its IRQ callback.
 * Internally, it unmasks the interrupt if possible.
 *
 * For INTx, unmasking is required as the interrupt is auto-masked prior to
 * invoking callback.
 *
 * For MSI/MSI-X, unmasking is typically not needed as the interrupt is not
 * auto-masked. In fact, for interrupt handle types VFIO_MSIX and VFIO_MSI,
 * this function is no-op.
 */


> + */
> +int
> +rte_intr_ack(const struct rte_intr_handle *intr_handle)
> +{
> +       if (intr_handle && intr_handle->type == RTE_INTR_HANDLE_VDEV)
> +               return 0;
> +
> +       if (!intr_handle || intr_handle->fd < 0 || intr_handle->uio_cfg_fd < 0)
> +               return -1;
> +
> +       switch (intr_handle->type) {
> +       /* Both acking and disabling are same for UIO */

enabling?

> +       case RTE_INTR_HANDLE_UIO:
> +               if (uio_intr_enable(intr_handle))
> +                       return -1;
> +               break;
> +       case RTE_INTR_HANDLE_UIO_INTX:
> +               if (uio_intx_intr_enable(intr_handle))
> +                       return -1;
> +               break;
> +       /* not used at this moment */
> +       case RTE_INTR_HANDLE_ALARM:
> +               return -1;
> +#ifdef VFIO_PRESENT
> +               /* Since VFIO_MSIX is implicitly acked
> +                * unlike INTx, we report success
> +                */

Indent is weird.
How about:

        /* VFIO MSI* is implicitly acked unlike INTx, nothing to do */

> +       case RTE_INTR_HANDLE_VFIO_MSIX:
> +       case RTE_INTR_HANDLE_VFIO_MSI:
> +               return 0;
> +       case RTE_INTR_HANDLE_VFIO_LEGACY:
> +               if (vfio_ack_intx(intr_handle))
> +                       return -1;
> +               break;
> +#ifdef HAVE_VFIO_DEV_REQ_INTERFACE
> +       case RTE_INTR_HANDLE_VFIO_REQ:
> +               return -1;
> +#endif
> +#endif
> +       /* not used at this moment */
> +       case RTE_INTR_HANDLE_DEV_EVENT:
> +               return -1;
> +       /* unknown handle type */
> +       default:
> +               RTE_LOG(ERR, EAL,
> +                       "Unknown handle type of fd %d\n",
> +                                       intr_handle->fd);
> +               return -1;
> +       }
> +
> +       return 0;
> +}
> +
>  int
>  rte_intr_disable(const struct rte_intr_handle *intr_handle)
>  {
> diff --git a/lib/librte_eal/rte_eal_version.map b/lib/librte_eal/rte_eal_version.map
> index 1892d9e..0537a6d 100644
> --- a/lib/librte_eal/rte_eal_version.map
> +++ b/lib/librte_eal/rte_eal_version.map
> @@ -407,4 +407,5 @@ EXPERIMENTAL {
>         rte_lcore_to_cpu_id;
>         rte_mcfg_timer_lock;
>         rte_mcfg_timer_unlock;
> +       rte_intr_ack;

This should be alphabetically sorted.


>  };

> --
> 2.8.4
>


--
David Marchand

  reply index

Thread overview: 38+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2019-07-17 11:58 [dpdk-dev] [PATCH 0/3] vfio: fix broken msix interrupt initialization Nithin Dabilpuram
2019-07-17 11:58 ` [dpdk-dev] [PATCH 1/3] vfio: revert change that does intr eventfd setup at probe Nithin Dabilpuram
2019-07-19  9:13   ` Nowak, DamianX
2019-07-17 11:58 ` [dpdk-dev] [PATCH 2/3] eal: add ack interrupt API Nithin Dabilpuram
2019-07-17 11:58 ` [dpdk-dev] [PATCH 3/3] drivers/net: use ack API in interrupt handlers Nithin Dabilpuram
2019-07-17 12:47   ` Hyong Youb Kim (hyonkim)
2019-07-17 17:18   ` Shahed Shaikh
2019-07-17 12:43 ` [dpdk-dev] [PATCH v2 0/3] vfio: fix broken msix interrupt initialization Nithin Dabilpuram
2019-07-17 12:43   ` [dpdk-dev] [PATCH v2 1/3] vfio: revert change that does intr eventfd setup at probe Nithin Dabilpuram
2019-07-17 12:43   ` [dpdk-dev] [PATCH v2 2/3] eal: add ack interrupt API Nithin Dabilpuram
2019-07-17 12:57     ` Hyong Youb Kim (hyonkim)
2019-07-17 14:35       ` [dpdk-dev] [EXT] " Nithin Kumar Dabilpuram
2019-07-17 15:05         ` Hyong Youb Kim (hyonkim)
2019-07-17 15:16           ` Nithin Kumar Dabilpuram
2019-07-18 10:22           ` Burakov, Anatoly
2019-07-17 12:43   ` [dpdk-dev] [PATCH v2 3/3] drivers/net: use ack API in interrupt handlers Nithin Dabilpuram
2019-07-18  8:46 ` [dpdk-dev] [PATCH v3 0/3] vfio: fix broken msix interrupt initialization Nithin Dabilpuram
2019-07-18  8:46   ` [dpdk-dev] [PATCH v3 1/3] vfio: revert change that does intr eventfd setup at probe Nithin Dabilpuram
2019-07-18  8:46   ` [dpdk-dev] [PATCH v3 2/3] eal: add ack interrupt API Nithin Dabilpuram
2019-07-18 13:13     ` Burakov, Anatoly
2019-07-18 13:27       ` [dpdk-dev] [EXT] " Jerin Jacob Kollanukkaran
2019-07-18 13:30         ` Nithin Kumar Dabilpuram
2019-07-18  8:46   ` [dpdk-dev] [PATCH v3 3/3] drivers/net: use ack API in interrupt handlers Nithin Dabilpuram
2019-07-18 14:36 ` [dpdk-dev] [PATCH v4 0/3] vfio: fix broken msix interrupt initialization Nithin Dabilpuram
2019-07-18 14:36   ` [dpdk-dev] [PATCH v4 1/3] vfio: revert change that does intr eventfd setup at probe Nithin Dabilpuram
2019-07-19  8:28     ` Yao, Lei A
2019-07-22 19:27     ` David Marchand
2019-07-18 14:36   ` [dpdk-dev] [PATCH v4 2/3] eal: add ack interrupt API Nithin Dabilpuram
2019-07-22 19:28     ` David Marchand [this message]
2019-07-18 14:36   ` [dpdk-dev] [PATCH v4 3/3] drivers/net: use ack API in interrupt handlers Nithin Dabilpuram
2019-07-22 19:29     ` David Marchand
2019-07-22 19:38   ` [dpdk-dev] [PATCH v4 0/3] vfio: fix broken msix interrupt initialization David Marchand
2019-07-23  5:32     ` [dpdk-dev] [EXT] " Nithin Kumar Dabilpuram
2019-07-23  8:04 ` [dpdk-dev] [PATCH v5 " David Marchand
2019-07-23  8:04   ` [dpdk-dev] [PATCH v5 1/3] vfio: revert change that does intr eventfd setup at probe David Marchand
2019-07-23  8:04   ` [dpdk-dev] [PATCH v5 2/3] eal: add ack interrupt API David Marchand
2019-07-23  8:04   ` [dpdk-dev] [PATCH v5 3/3] drivers/net: use ack API in interrupt handlers David Marchand
2019-07-23 10:01   ` [dpdk-dev] [PATCH v5 0/3] vfio: fix broken msix interrupt initialization Thomas Monjalon

Reply instructions:

You may reply publically to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to='CAJFAV8ySX=a2SgVdGtwwrEGRMuNWvCeMxA3LonCB23wJAQ-M=g@mail.gmail.com' \
    --to=david.marchand@redhat.com \
    --cc=bruce.richardson@intel.com \
    --cc=dev@dpdk.org \
    --cc=ferruh.yigit@intel.com \
    --cc=hyonkim@cisco.com \
    --cc=jerinj@marvell.com \
    --cc=johndale@cisco.com \
    --cc=ndabilpuram@marvell.com \
    --cc=shshaikh@marvell.com \
    --cc=thomas@monjalon.net \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

DPDK-dev Archive on lore.kernel.org

Archives are clonable:
	git clone --mirror https://lore.kernel.org/dpdk-dev/0 dpdk-dev/git/0.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 dpdk-dev dpdk-dev/ https://lore.kernel.org/dpdk-dev \
		dev@dpdk.org dpdk-dev@archiver.kernel.org
	public-inbox-index dpdk-dev


Newsgroup available over NNTP:
	nntp://nntp.lore.kernel.org/org.dpdk.dev


AGPL code for this site: git clone https://public-inbox.org/ public-inbox