From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2747AC433EF for ; Mon, 11 Oct 2021 08:35:41 +0000 (UTC) Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by mail.kernel.org (Postfix) with ESMTP id 504AB60C4B for ; Mon, 11 Oct 2021 08:35:40 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 504AB60C4B Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=oktetlabs.ru Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=dpdk.org Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 79F93410E2; Mon, 11 Oct 2021 10:35:39 +0200 (CEST) Received: from shelob.oktetlabs.ru (shelob.oktetlabs.ru [91.220.146.113]) by mails.dpdk.org (Postfix) with ESMTP id 7A9D3410DB for ; Mon, 11 Oct 2021 10:35:38 +0200 (CEST) Received: from [192.168.38.17] (aros.oktetlabs.ru [192.168.38.17]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by shelob.oktetlabs.ru (Postfix) with ESMTPSA id CED017F514; Mon, 11 Oct 2021 11:35:37 +0300 (MSK) DKIM-Filter: OpenDKIM Filter v2.11.0 shelob.oktetlabs.ru CED017F514 DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=oktetlabs.ru; s=default; t=1633941338; bh=3ZqgfkLRSjUCeeleqoSnj7gfUS3YiAOF6C1BrzwPQvM=; h=Subject:To:Cc:References:From:Date:In-Reply-To; b=ktH2J4E9i8vwZs6kzbuLzOVGjTuXeDZ+0/lODWideQivF3xg4jmLGlFcUAuQlN4tk CnxTXXdMHQVh+cYjmLPGrJuJ6kbjy2Z3MLTDDIdQJKHVGOug+j1e/lPtfQ47EO3M0N /j9h4aK79/uw6U2ho+kOJzzdp0Kg5XodqHCDd68U= To: fengchengwen , Konstantin Ananyev , dev@dpdk.org Cc: xiaoyun.li@intel.com, anoobj@marvell.com, jerinj@marvell.com, ndabilpuram@marvell.com, adwivedi@marvell.com, shepard.siegel@atomicrules.com, ed.czeck@atomicrules.com, john.miller@atomicrules.com, irusskikh@marvell.com, ajit.khaparde@broadcom.com, somnath.kotur@broadcom.com, rahul.lakkireddy@chelsio.com, hemant.agrawal@nxp.com, sachin.saxena@oss.nxp.com, haiyue.wang@intel.com, johndale@cisco.com, hyonkim@cisco.com, qi.z.zhang@intel.com, xiao.w.wang@intel.com, humin29@huawei.com, yisen.zhuang@huawei.com, oulijun@huawei.com, beilei.xing@intel.com, jingjing.wu@intel.com, qiming.yang@intel.com, matan@nvidia.com, viacheslavo@nvidia.com, sthemmin@microsoft.com, longli@microsoft.com, heinrich.kuhn@corigine.com, kirankumark@marvell.com, mczekaj@marvell.com, jiawenwu@trustnetic.com, jianwang@trustnetic.com, maxime.coquelin@redhat.com, chenbo.xia@intel.com, thomas@monjalon.net, ferruh.yigit@intel.com, mdr@ashroe.eu, jay.jayatheerthan@intel.com References: <20211004135603.20593-1-konstantin.ananyev@intel.com> <20211007112750.25526-1-konstantin.ananyev@intel.com> <20211007112750.25526-5-konstantin.ananyev@intel.com> <4c57bb9d-21d6-0722-92b8-987283bb8fe6@huawei.com> From: Andrew Rybchenko Organization: OKTET Labs Message-ID: <6493c54a-23bb-7f4a-6645-102db6303ab2@oktetlabs.ru> Date: Mon, 11 Oct 2021 11:35:37 +0300 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101 Thunderbird/78.14.0 MIME-Version: 1.0 In-Reply-To: <4c57bb9d-21d6-0722-92b8-987283bb8fe6@huawei.com> Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 7bit Subject: Re: [dpdk-dev] [PATCH v5 4/7] ethdev: copy fast-path API into separate structure X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" On 10/9/21 3:05 PM, fengchengwen wrote: > On 2021/10/7 19:27, Konstantin Ananyev wrote: >> Copy public function pointers (rx_pkt_burst(), etc.) and related >> pointers to internal data from rte_eth_dev structure into a >> separate flat array. That array will remain in a public header. >> The intention here is to make rte_eth_dev and related structures internal. >> That should allow future possible changes to core eth_dev structures >> to be transparent to the user and help to avoid ABI/API breakages. >> The plan is to keep minimal part of data from rte_eth_dev public, >> so we still can use inline functions for fast-path calls >> (like rte_eth_rx_burst(), etc.) to avoid/minimize slowdown. >> The whole idea beyond this new schema: >> 1. PMDs keep to setup fast-path function pointers and related data >> inside rte_eth_dev struct in the same way they did it before. >> 2. Inside rte_eth_dev_start() and inside rte_eth_dev_probing_finish() >> (for secondary process) we call eth_dev_fp_ops_setup, which >> copies these function and data pointers into rte_eth_fp_ops[port_id]. >> 3. Inside rte_eth_dev_stop() and inside rte_eth_dev_release_port() >> we call eth_dev_fp_ops_reset(), which resets rte_eth_fp_ops[port_id] >> into some dummy values. >> 4. fast-path ethdev API (rte_eth_rx_burst(), etc.) will use that new >> flat array to call PMD specific functions. >> That approach should allow us to make rte_eth_devices[] private >> without introducing regression and help to avoid changes in drivers code. >> >> Signed-off-by: Konstantin Ananyev >> --- >> lib/ethdev/ethdev_private.c | 52 ++++++++++++++++++++++++++++++++++ >> lib/ethdev/ethdev_private.h | 7 +++++ >> lib/ethdev/rte_ethdev.c | 27 ++++++++++++++++++ >> lib/ethdev/rte_ethdev_core.h | 55 ++++++++++++++++++++++++++++++++++++ >> 4 files changed, 141 insertions(+) >> >> diff --git a/lib/ethdev/ethdev_private.c b/lib/ethdev/ethdev_private.c >> index 012cf73ca2..3eeda6e9f9 100644 >> --- a/lib/ethdev/ethdev_private.c >> +++ b/lib/ethdev/ethdev_private.c >> @@ -174,3 +174,55 @@ rte_eth_devargs_parse_representor_ports(char *str, void *data) >> RTE_LOG(ERR, EAL, "wrong representor format: %s\n", str); >> return str == NULL ? -1 : 0; >> } >> + >> +static uint16_t >> +dummy_eth_rx_burst(__rte_unused void *rxq, >> + __rte_unused struct rte_mbuf **rx_pkts, >> + __rte_unused uint16_t nb_pkts) >> +{ >> + RTE_ETHDEV_LOG(ERR, "rx_pkt_burst for unconfigured port\n"); >> + rte_errno = ENOTSUP; >> + return 0; >> +} >> + >> +static uint16_t >> +dummy_eth_tx_burst(__rte_unused void *txq, >> + __rte_unused struct rte_mbuf **tx_pkts, >> + __rte_unused uint16_t nb_pkts) >> +{ >> + RTE_ETHDEV_LOG(ERR, "tx_pkt_burst for unconfigured port\n"); >> + rte_errno = ENOTSUP; >> + return 0; >> +} >> + >> +void >> +eth_dev_fp_ops_reset(struct rte_eth_fp_ops *fpo) > > The port_id parameter is preferable, this will hide rte_eth_fp_ops as much as possible. Sorry, but I see no point to hide it inside ethdev. Of course, prototype should be reconsidered if we make it ethdev-internal API available for drivers. If so, I agree that the parameter should be port_id. [snip] >> diff --git a/lib/ethdev/ethdev_private.h b/lib/ethdev/ethdev_private.h >> index 3724429577..5721be7bdc 100644 >> --- a/lib/ethdev/ethdev_private.h >> +++ b/lib/ethdev/ethdev_private.h >> @@ -26,4 +26,11 @@ eth_find_device(const struct rte_eth_dev *_start, rte_eth_cmp_t cmp, >> /* Parse devargs value for representor parameter. */ >> int rte_eth_devargs_parse_representor_ports(char *str, void *data); >> >> +/* reset eth fast-path API to dummy values */ >> +void eth_dev_fp_ops_reset(struct rte_eth_fp_ops *fpo); >> + >> +/* setup eth fast-path API to ethdev values */ >> +void eth_dev_fp_ops_setup(struct rte_eth_fp_ops *fpo, >> + const struct rte_eth_dev *dev); > > Some drivers control the transmit/receive function during operation. E.g. > for hns3 driver, when detect reset, primary process will set rx/tx burst to dummy, after > process reset, primary process will set the correct rx/tx burst. During this process, the > send and receive threads are still working, but the bursts they call are changed. So: > 1. it is recommended that trace be deleted from the dummy function. > 2. public the eth_dev_fp_ops_reset/setup interface for driver usage. Good point. [snip] >> diff --git a/lib/ethdev/rte_ethdev_core.h b/lib/ethdev/rte_ethdev_core.h >> index 51cd68de94..d5853dff86 100644 >> --- a/lib/ethdev/rte_ethdev_core.h >> +++ b/lib/ethdev/rte_ethdev_core.h >> @@ -50,6 +50,61 @@ typedef int (*eth_rx_descriptor_status_t)(void *rxq, uint16_t offset); >> typedef int (*eth_tx_descriptor_status_t)(void *txq, uint16_t offset); >> /**< @internal Check the status of a Tx descriptor */ >> >> +/** >> + * @internal >> + * Structure used to hold opaque pointers to internal ethdev Rx/Tx >> + * queues data. >> + * The main purpose to expose these pointers at all - allow compiler >> + * to fetch this data for fast-path ethdev inline functions in advance. >> + */ >> +struct rte_ethdev_qdata { >> + void **data; >> + /**< points to array of internal queue data pointers */ >> + void **clbk; >> + /**< points to array of queue callback data pointers */ >> +}; >> + >> +/** >> + * @internal >> + * fast-path ethdev functions and related data are hold in a flat array. >> + * One entry per ethdev. >> + * On 64-bit systems contents of this structure occupy exactly two 64B lines. >> + * On 32-bit systems contents of this structure fits into one 64B line. >> + */ >> +struct rte_eth_fp_ops { >> + >> + /** >> + * Rx fast-path functions and related data. >> + * 64-bit systems: occupies first 64B line >> + */ >> + eth_rx_burst_t rx_pkt_burst; >> + /**< PMD receive function. */ >> + eth_rx_queue_count_t rx_queue_count; >> + /**< Get the number of used RX descriptors. */ >> + eth_rx_descriptor_status_t rx_descriptor_status; >> + /**< Check the status of a Rx descriptor. */ >> + struct rte_ethdev_qdata rxq; >> + /**< Rx queues data. */ >> + uintptr_t reserved1[3]; >> + >> + /** >> + * Tx fast-path functions and related data. >> + * 64-bit systems: occupies second 64B line >> + */ >> + eth_tx_burst_t tx_pkt_burst; > > Why not place rx_pkt_burst/tx_pkt_burst/rxq /txq to the first cacheline ? > Other function, e.g. rx_queue_count/descriptor_status are low frequency call functions. +1 Very good question If so, tx_pkt_prepare should be on the first cache-line as well. >> + /**< PMD transmit function. */ >> + eth_tx_prep_t tx_pkt_prepare; >> + /**< PMD transmit prepare function. */ >> + eth_tx_descriptor_status_t tx_descriptor_status; >> + /**< Check the status of a Tx descriptor. */ >> + struct rte_ethdev_qdata txq; >> + /**< Tx queues data. */ >> + uintptr_t reserved2[3]; >> + >> +} __rte_cache_aligned; >> + >> +extern struct rte_eth_fp_ops rte_eth_fp_ops[RTE_MAX_ETHPORTS]; >> + >> >> /** >> * @internal >>