From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.8 required=3.0 tests=HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_PATCH,MAILING_LIST_MULTI,SIGNED_OFF_BY,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2E196C7618B for ; Tue, 23 Jul 2019 08:04:53 +0000 (UTC) Received: from dpdk.org (dpdk.org [92.243.14.124]) by mail.kernel.org (Postfix) with ESMTP id C28292070B for ; Tue, 23 Jul 2019 08:04:52 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org C28292070B Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=dev-bounces@dpdk.org Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 16BA01BFD6; Tue, 23 Jul 2019 10:04:40 +0200 (CEST) Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by dpdk.org (Postfix) with ESMTP id BB8E31BFC4 for ; Tue, 23 Jul 2019 10:04:36 +0200 (CEST) Received: from smtp.corp.redhat.com (int-mx07.intmail.prod.int.phx2.redhat.com [10.5.11.22]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 194868553F; Tue, 23 Jul 2019 08:04:36 +0000 (UTC) Received: from dmarchan.remote.csb (ovpn-204-177.brq.redhat.com [10.40.204.177]) by smtp.corp.redhat.com (Postfix) with ESMTP id B165E1001DD7; Tue, 23 Jul 2019 08:04:33 +0000 (UTC) From: David Marchand To: dev@dpdk.org Cc: thomas@monjalon.net, hyonkim@cisco.com, ferruh.yigit@intel.com, jerinj@marvell.com, johndale@cisco.com, shshaikh@marvell.com, Nithin Dabilpuram , Bruce Richardson Date: Tue, 23 Jul 2019 10:04:18 +0200 Message-Id: <1563869059-12618-3-git-send-email-david.marchand@redhat.com> In-Reply-To: <1563869059-12618-1-git-send-email-david.marchand@redhat.com> References: <20190717115852.171416-1-ndabilpuram@marvell.com> <1563869059-12618-1-git-send-email-david.marchand@redhat.com> X-Scanned-By: MIMEDefang 2.84 on 10.5.11.22 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.28]); Tue, 23 Jul 2019 08:04:36 +0000 (UTC) Subject: [dpdk-dev] [PATCH v5 2/3] eal: add ack interrupt API X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" From: Nithin Dabilpuram Add new ack interrupt API to avoid using VFIO_IRQ_SET_ACTION_TRIGGER(rte_intr_enable()) for acking interrupt purpose for VFIO based interrupt handlers. This implementation is specific to Linux. Using rte_intr_enable() for acking interrupt has below issues * Time consuming to do for every interrupt received as it will free_irq() followed by request_irq() and all other initializations * A race condition because of a window between free_irq() and request_irq() with packet reception still on and device still enabled and would throw warning messages like below. [158764.159833] do_IRQ: 9.34 No irq handler for vector In this patch, rte_intr_ack() is a no-op for VFIO_MSIX/VFIO_MSI interrupts as they are edge triggered and kernel would not mask the interrupt before delivering the event to userspace and we don't need to ack. Signed-off-by: Nithin Dabilpuram Signed-off-by: Jerin Jacob Tested-by: Shahed Shaikh Signed-off-by: David Marchand --- Changelog since v4: - added EXPERIMENTAL banner + little rewording in header - simplified vfio ioctl code by shrinking the passed buffer - fixed comments - sorted new symbol in map file Changelog since v3: - Move note to implementation and change the expectation to must call for new api. Changelog since v2: - Update note on new api --- lib/librte_eal/common/include/rte_interrupts.h | 20 +++++++ lib/librte_eal/freebsd/eal/eal_interrupts.c | 9 +++ lib/librte_eal/linux/eal/eal_interrupts.c | 82 ++++++++++++++++++++++++++ lib/librte_eal/rte_eal_version.map | 1 + 4 files changed, 112 insertions(+) diff --git a/lib/librte_eal/common/include/rte_interrupts.h b/lib/librte_eal/common/include/rte_interrupts.h index c1e912c..e3b406a 100644 --- a/lib/librte_eal/common/include/rte_interrupts.h +++ b/lib/librte_eal/common/include/rte_interrupts.h @@ -118,6 +118,26 @@ int rte_intr_callback_unregister(const struct rte_intr_handle *intr_handle, */ int rte_intr_disable(const struct rte_intr_handle *intr_handle); +/** + * @warning + * @b EXPERIMENTAL: this API may change without prior notice + * + * It acknowledges an interrupt raised for the specified handle. + * + * This function should be called at the end of each interrupt handler either + * from application or driver, so that currently raised interrupt is acked and + * further new interrupts are raised. + * + * @param intr_handle + * pointer to the interrupt handle. + * + * @return + * - On success, zero. + * - On failure, a negative value. + */ +__rte_experimental +int rte_intr_ack(const struct rte_intr_handle *intr_handle); + #ifdef __cplusplus } #endif diff --git a/lib/librte_eal/freebsd/eal/eal_interrupts.c b/lib/librte_eal/freebsd/eal/eal_interrupts.c index 10375bd..f6831b7 100644 --- a/lib/librte_eal/freebsd/eal/eal_interrupts.c +++ b/lib/librte_eal/freebsd/eal/eal_interrupts.c @@ -387,6 +387,15 @@ struct rte_intr_source { return 0; } +int +rte_intr_ack(const struct rte_intr_handle *intr_handle) +{ + if (intr_handle && intr_handle->type == RTE_INTR_HANDLE_VDEV) + return 0; + + return -1; +} + static void eal_intr_process_interrupts(struct kevent *events, int nfds) { diff --git a/lib/librte_eal/linux/eal/eal_interrupts.c b/lib/librte_eal/linux/eal/eal_interrupts.c index 79ad5e8..1955324 100644 --- a/lib/librte_eal/linux/eal/eal_interrupts.c +++ b/lib/librte_eal/linux/eal/eal_interrupts.c @@ -197,6 +197,28 @@ struct rte_intr_source { return 0; } +/* unmask/ack legacy (INTx) interrupts */ +static int +vfio_ack_intx(const struct rte_intr_handle *intr_handle) +{ + struct vfio_irq_set irq_set; + + /* unmask INTx */ + memset(&irq_set, 0, sizeof(irq_set)); + irq_set.argsz = sizeof(irq_set); + irq_set.count = 1; + irq_set.flags = VFIO_IRQ_SET_DATA_NONE | VFIO_IRQ_SET_ACTION_UNMASK; + irq_set.index = VFIO_PCI_INTX_IRQ_INDEX; + irq_set.start = 0; + + if (ioctl(intr_handle->vfio_dev_fd, VFIO_DEVICE_SET_IRQS, &irq_set)) { + RTE_LOG(ERR, EAL, "Error unmasking INTx interrupts for fd %d\n", + intr_handle->fd); + return -1; + } + return 0; +} + /* enable MSI interrupts */ static int vfio_enable_msi(const struct rte_intr_handle *intr_handle) { @@ -693,6 +715,66 @@ struct rte_intr_source { return 0; } +/** + * PMD generally calls this function at the end of its IRQ callback. + * Internally, it unmasks the interrupt if possible. + * + * For INTx, unmasking is required as the interrupt is auto-masked prior to + * invoking callback. + * + * For MSI/MSI-X, unmasking is typically not needed as the interrupt is not + * auto-masked. In fact, for interrupt handle types VFIO_MSIX and VFIO_MSI, + * this function is no-op. + */ +int +rte_intr_ack(const struct rte_intr_handle *intr_handle) +{ + if (intr_handle && intr_handle->type == RTE_INTR_HANDLE_VDEV) + return 0; + + if (!intr_handle || intr_handle->fd < 0 || intr_handle->uio_cfg_fd < 0) + return -1; + + switch (intr_handle->type) { + /* Both acking and enabling are same for UIO */ + case RTE_INTR_HANDLE_UIO: + if (uio_intr_enable(intr_handle)) + return -1; + break; + case RTE_INTR_HANDLE_UIO_INTX: + if (uio_intx_intr_enable(intr_handle)) + return -1; + break; + /* not used at this moment */ + case RTE_INTR_HANDLE_ALARM: + return -1; +#ifdef VFIO_PRESENT + /* VFIO MSI* is implicitly acked unlike INTx, nothing to do */ + case RTE_INTR_HANDLE_VFIO_MSIX: + case RTE_INTR_HANDLE_VFIO_MSI: + return 0; + case RTE_INTR_HANDLE_VFIO_LEGACY: + if (vfio_ack_intx(intr_handle)) + return -1; + break; +#ifdef HAVE_VFIO_DEV_REQ_INTERFACE + case RTE_INTR_HANDLE_VFIO_REQ: + return -1; +#endif +#endif + /* not used at this moment */ + case RTE_INTR_HANDLE_DEV_EVENT: + return -1; + /* unknown handle type */ + default: + RTE_LOG(ERR, EAL, "Unknown handle type of fd %d\n", + intr_handle->fd); + return -1; + } + + return 0; +} + int rte_intr_disable(const struct rte_intr_handle *intr_handle) { diff --git a/lib/librte_eal/rte_eal_version.map b/lib/librte_eal/rte_eal_version.map index 1892d9e..2344877 100644 --- a/lib/librte_eal/rte_eal_version.map +++ b/lib/librte_eal/rte_eal_version.map @@ -403,6 +403,7 @@ EXPERIMENTAL { rte_realloc_socket; # added in 19.08 + rte_intr_ack; rte_lcore_cpuset; rte_lcore_to_cpu_id; rte_mcfg_timer_lock; -- 1.8.3.1