From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4E6B4C2B9F4 for ; Fri, 25 Jun 2021 05:57:57 +0000 (UTC) Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by mail.kernel.org (Postfix) with ESMTP id CBA926140F for ; Fri, 25 Jun 2021 05:57:56 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org CBA926140F Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=marvell.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=dev-bounces@dpdk.org Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 68102410F8; Fri, 25 Jun 2021 07:57:36 +0200 (CEST) Received: from mx0b-0016f401.pphosted.com (mx0b-0016f401.pphosted.com [67.231.156.173]) by mails.dpdk.org (Postfix) with ESMTP id CF95D410F9 for ; Fri, 25 Jun 2021 07:57:34 +0200 (CEST) Received: from pps.filterd (m0045851.ppops.net [127.0.0.1]) by mx0b-0016f401.pphosted.com (8.16.0.43/8.16.0.43) with SMTP id 15P5vFf7018374; Thu, 24 Jun 2021 22:57:34 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=marvell.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : content-type; s=pfpt0220; bh=K5XlYL0UVWATANuP2BR8Y+B0JM9zk5zyu6exbJLUSYY=; b=jKkA2X+kohM4hbkkEfpU76oFU9EgFBC6O1obngKl8kpXxarvfyEho5OtTaICaBrWJxmw qma/xEgKs25NPeRLLmY3LJA2IeMGQVJ6ZDabWtTzsWBIeVMvRZ43cXFhSEeJicIUcMiI j85FUgAOhyopZTsZWt6ElkXgk4cK1+bxWBZ6NzueFfDhBIkRP4/PLsgs1EpH7xRCTjGD OKwwN1Tf301gb/eRF24v9HQW0I6KPM0jPGkJkXAfjs9NC0VO1xqBazk5Y3QXq1LhGmmo J4l7zvZWBEdeoumaKefhEJVWlLBfOw8xEfrqIIiwwZZC+lX/q6keoLGbmBV3K6urrgbn Yw== Received: from dc5-exch01.marvell.com ([199.233.59.181]) by mx0b-0016f401.pphosted.com with ESMTP id 39d241shrn-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-SHA384 bits=256 verify=NOT); Thu, 24 Jun 2021 22:57:34 -0700 Received: from DC5-EXCH01.marvell.com (10.69.176.38) by DC5-EXCH01.marvell.com (10.69.176.38) with Microsoft SMTP Server (TLS) id 15.0.1497.18; Thu, 24 Jun 2021 22:57:31 -0700 Received: from maili.marvell.com (10.69.176.80) by DC5-EXCH01.marvell.com (10.69.176.38) with Microsoft SMTP Server id 15.0.1497.18 via Frontend Transport; Thu, 24 Jun 2021 22:57:31 -0700 Received: from HY-LT1002.marvell.com (HY-LT1002.marvell.com [10.28.176.218]) by maili.marvell.com (Postfix) with ESMTP id AD8FE3F7041; Thu, 24 Jun 2021 22:57:28 -0700 (PDT) From: Anoob Joseph To: Akhil Goyal , Thomas Monjalon CC: Anoob Joseph , Jerin Jacob , "Ankur Dwivedi" , Tejasree Kondoj , , Archana Muniganti Date: Fri, 25 Jun 2021 11:26:17 +0530 Message-ID: <1624600591-29841-7-git-send-email-anoobj@marvell.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1624600591-29841-1-git-send-email-anoobj@marvell.com> References: <1624600591-29841-1-git-send-email-anoobj@marvell.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Content-Type: text/plain X-Proofpoint-GUID: 1kZhHnOF5uu8uzr1Mkujr7bWSyXm_64_ X-Proofpoint-ORIG-GUID: 1kZhHnOF5uu8uzr1Mkujr7bWSyXm_64_ X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.391, 18.0.790 definitions=2021-06-25_02:2021-06-24, 2021-06-25 signatures=0 Subject: [dpdk-dev] [PATCH v2 06/20] crypto/cnxk: add enqueue burst op X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Add enqueue_burst op in cn9k & cn10k. Signed-off-by: Ankur Dwivedi Signed-off-by: Anoob Joseph Signed-off-by: Archana Muniganti Signed-off-by: Tejasree Kondoj --- drivers/crypto/cnxk/cn10k_cryptodev.c | 2 + drivers/crypto/cnxk/cn10k_cryptodev_ops.c | 189 ++++++++++++++++++++++++++++++ drivers/crypto/cnxk/cn10k_cryptodev_ops.h | 2 + drivers/crypto/cnxk/cn9k_cryptodev.c | 2 + drivers/crypto/cnxk/cn9k_cryptodev_ops.c | 154 ++++++++++++++++++++++++ drivers/crypto/cnxk/cn9k_cryptodev_ops.h | 2 + drivers/crypto/cnxk/cnxk_cryptodev_ops.h | 9 ++ 7 files changed, 360 insertions(+) diff --git a/drivers/crypto/cnxk/cn10k_cryptodev.c b/drivers/crypto/cnxk/cn10k_cryptodev.c index a66b777..53f7a94 100644 --- a/drivers/crypto/cnxk/cn10k_cryptodev.c +++ b/drivers/crypto/cnxk/cn10k_cryptodev.c @@ -80,6 +80,8 @@ cn10k_cpt_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, dev->dev_ops = &cn10k_cpt_ops; dev->driver_id = cn10k_cryptodev_driver_id; + cn10k_cpt_set_enqdeq_fns(dev); + return 0; dev_fini: diff --git a/drivers/crypto/cnxk/cn10k_cryptodev_ops.c b/drivers/crypto/cnxk/cn10k_cryptodev_ops.c index 34dc107..5dd2cd2 100644 --- a/drivers/crypto/cnxk/cn10k_cryptodev_ops.c +++ b/drivers/crypto/cnxk/cn10k_cryptodev_ops.c @@ -7,7 +7,196 @@ #include "cn10k_cryptodev.h" #include "cn10k_cryptodev_ops.h" +#include "cnxk_cryptodev.h" #include "cnxk_cryptodev_ops.h" +#include "cnxk_se.h" + +static inline struct cnxk_se_sess * +cn10k_cpt_sym_temp_sess_create(struct cnxk_cpt_qp *qp, struct rte_crypto_op *op) +{ + const int driver_id = cn10k_cryptodev_driver_id; + struct rte_crypto_sym_op *sym_op = op->sym; + struct rte_cryptodev_sym_session *sess; + struct cnxk_se_sess *priv; + int ret; + + /* Create temporary session */ + sess = rte_cryptodev_sym_session_create(qp->sess_mp); + if (sess == NULL) + return NULL; + + ret = sym_session_configure(qp->lf.roc_cpt, driver_id, sym_op->xform, + sess, qp->sess_mp_priv); + if (ret) + goto sess_put; + + priv = get_sym_session_private_data(sess, driver_id); + + sym_op->session = sess; + + return priv; + +sess_put: + rte_mempool_put(qp->sess_mp, sess); + return NULL; +} + +static __rte_always_inline int __rte_hot +cpt_sym_inst_fill(struct cnxk_cpt_qp *qp, struct rte_crypto_op *op, + struct cnxk_se_sess *sess, struct cpt_inflight_req *infl_req, + struct cpt_inst_s *inst) +{ + RTE_SET_USED(qp); + RTE_SET_USED(op); + RTE_SET_USED(sess); + RTE_SET_USED(infl_req); + RTE_SET_USED(inst); + + return -ENOTSUP; +} + +static inline int +cn10k_cpt_fill_inst(struct cnxk_cpt_qp *qp, struct rte_crypto_op *ops[], + struct cpt_inst_s inst[], struct cpt_inflight_req *infl_req) +{ + struct rte_crypto_sym_op *sym_op; + struct cnxk_se_sess *sess; + struct rte_crypto_op *op; + uint64_t w7; + int ret; + + op = ops[0]; + + inst[0].w0.u64 = 0; + inst[0].w2.u64 = 0; + inst[0].w3.u64 = 0; + + sym_op = op->sym; + + if (op->type == RTE_CRYPTO_OP_TYPE_SYMMETRIC) { + if (op->sess_type == RTE_CRYPTO_OP_WITH_SESSION) { + sess = get_sym_session_private_data( + sym_op->session, cn10k_cryptodev_driver_id); + ret = cpt_sym_inst_fill(qp, op, sess, infl_req, + &inst[0]); + if (unlikely(ret)) + return 0; + w7 = sess->cpt_inst_w7; + } else { + sess = cn10k_cpt_sym_temp_sess_create(qp, op); + if (unlikely(sess == NULL)) { + plt_dp_err("Could not create temp session"); + return 0; + } + + ret = cpt_sym_inst_fill(qp, op, sess, infl_req, + &inst[0]); + if (unlikely(ret)) { + sym_session_clear(cn10k_cryptodev_driver_id, + op->sym->session); + rte_mempool_put(qp->sess_mp, op->sym->session); + return 0; + } + w7 = sess->cpt_inst_w7; + } + } else { + plt_dp_err("Unsupported op type"); + return 0; + } + + inst[0].res_addr = (uint64_t)&infl_req->res; + infl_req->res.cn10k.compcode = CPT_COMP_NOT_DONE; + infl_req->cop = op; + + inst[0].w7.u64 = w7; + + return 1; +} + +#define PKTS_PER_LOOP 32 +#define PKTS_PER_STEORL 16 + +static uint16_t +cn10k_cpt_enqueue_burst(void *qptr, struct rte_crypto_op **ops, uint16_t nb_ops) +{ + uint64_t lmt_base, lmt_arg, io_addr; + struct cpt_inflight_req *infl_req; + uint16_t nb_allowed, count = 0; + struct cnxk_cpt_qp *qp = qptr; + struct pending_queue *pend_q; + struct cpt_inst_s *inst; + uint16_t lmt_id; + int ret, i; + + pend_q = &qp->pend_q; + + nb_allowed = qp->lf.nb_desc - pend_q->pending_count; + nb_ops = RTE_MIN(nb_ops, nb_allowed); + + if (unlikely(nb_ops == 0)) + return 0; + + lmt_base = qp->lmtline.lmt_base; + io_addr = qp->lmtline.io_addr; + + ROC_LMT_BASE_ID_GET(lmt_base, lmt_id); + inst = (struct cpt_inst_s *)lmt_base; + +again: + for (i = 0; i < RTE_MIN(PKTS_PER_LOOP, nb_ops); i++) { + infl_req = &pend_q->req_queue[pend_q->enq_tail]; + infl_req->op_flags = 0; + + ret = cn10k_cpt_fill_inst(qp, ops + i, &inst[2 * i], infl_req); + if (unlikely(ret != 1)) { + plt_dp_err("Could not process op: %p", ops + i); + if (i == 0) + goto update_pending; + break; + } + + MOD_INC(pend_q->enq_tail, qp->lf.nb_desc); + } + + if (i > PKTS_PER_STEORL) { + lmt_arg = ROC_CN10K_CPT_LMT_ARG | (PKTS_PER_STEORL - 1) << 12 | + (uint64_t)lmt_id; + roc_lmt_submit_steorl(lmt_arg, io_addr); + lmt_arg = ROC_CN10K_CPT_LMT_ARG | + (i - PKTS_PER_STEORL - 1) << 12 | + (uint64_t)(lmt_id + PKTS_PER_STEORL); + roc_lmt_submit_steorl(lmt_arg, io_addr); + } else { + lmt_arg = ROC_CN10K_CPT_LMT_ARG | (i - 1) << 12 | + (uint64_t)lmt_id; + roc_lmt_submit_steorl(lmt_arg, io_addr); + } + + rte_io_wmb(); + + if (nb_ops - i > 0 && i == PKTS_PER_LOOP) { + nb_ops -= i; + ops += i; + count += i; + goto again; + } + +update_pending: + pend_q->pending_count += count + i; + + pend_q->time_out = rte_get_timer_cycles() + + DEFAULT_COMMAND_TIMEOUT * rte_get_timer_hz(); + + return count + i; +} + +void +cn10k_cpt_set_enqdeq_fns(struct rte_cryptodev *dev) +{ + dev->enqueue_burst = cn10k_cpt_enqueue_burst; + + rte_mb(); +} static void cn10k_cpt_dev_info_get(struct rte_cryptodev *dev, diff --git a/drivers/crypto/cnxk/cn10k_cryptodev_ops.h b/drivers/crypto/cnxk/cn10k_cryptodev_ops.h index 24611bf..d500b7d 100644 --- a/drivers/crypto/cnxk/cn10k_cryptodev_ops.h +++ b/drivers/crypto/cnxk/cn10k_cryptodev_ops.h @@ -10,4 +10,6 @@ extern struct rte_cryptodev_ops cn10k_cpt_ops; +void cn10k_cpt_set_enqdeq_fns(struct rte_cryptodev *dev); + #endif /* _CN10K_CRYPTODEV_OPS_H_ */ diff --git a/drivers/crypto/cnxk/cn9k_cryptodev.c b/drivers/crypto/cnxk/cn9k_cryptodev.c index 46ad33f..4dbb40d 100644 --- a/drivers/crypto/cnxk/cn9k_cryptodev.c +++ b/drivers/crypto/cnxk/cn9k_cryptodev.c @@ -78,6 +78,8 @@ cn9k_cpt_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, dev->dev_ops = &cn9k_cpt_ops; dev->driver_id = cn9k_cryptodev_driver_id; + cn9k_cpt_set_enqdeq_fns(dev); + return 0; dev_fini: diff --git a/drivers/crypto/cnxk/cn9k_cryptodev_ops.c b/drivers/crypto/cnxk/cn9k_cryptodev_ops.c index bef6159..f09f9ee 100644 --- a/drivers/crypto/cnxk/cn9k_cryptodev_ops.c +++ b/drivers/crypto/cnxk/cn9k_cryptodev_ops.c @@ -7,7 +7,161 @@ #include "cn9k_cryptodev.h" #include "cn9k_cryptodev_ops.h" +#include "cnxk_cryptodev.h" #include "cnxk_cryptodev_ops.h" +#include "cnxk_se.h" + +static __rte_always_inline int __rte_hot +cn9k_cpt_sym_inst_fill(struct cnxk_cpt_qp *qp, struct rte_crypto_op *op, + struct cnxk_se_sess *sess, + struct cpt_inflight_req *infl_req, + struct cpt_inst_s *inst) +{ + RTE_SET_USED(qp); + RTE_SET_USED(op); + RTE_SET_USED(sess); + RTE_SET_USED(infl_req); + RTE_SET_USED(inst); + + return -ENOTSUP; +} + +static inline struct cnxk_se_sess * +cn9k_cpt_sym_temp_sess_create(struct cnxk_cpt_qp *qp, struct rte_crypto_op *op) +{ + const int driver_id = cn9k_cryptodev_driver_id; + struct rte_crypto_sym_op *sym_op = op->sym; + struct rte_cryptodev_sym_session *sess; + struct cnxk_se_sess *priv; + int ret; + + /* Create temporary session */ + sess = rte_cryptodev_sym_session_create(qp->sess_mp); + if (sess == NULL) + return NULL; + + ret = sym_session_configure(qp->lf.roc_cpt, driver_id, sym_op->xform, + sess, qp->sess_mp_priv); + if (ret) + goto sess_put; + + priv = get_sym_session_private_data(sess, driver_id); + + sym_op->session = sess; + + return priv; + +sess_put: + rte_mempool_put(qp->sess_mp, sess); + return NULL; +} + +static uint16_t +cn9k_cpt_enqueue_burst(void *qptr, struct rte_crypto_op **ops, uint16_t nb_ops) +{ + struct cpt_inflight_req *infl_req; + struct rte_crypto_sym_op *sym_op; + uint16_t nb_allowed, count = 0; + struct cnxk_cpt_qp *qp = qptr; + struct pending_queue *pend_q; + struct cnxk_se_sess *sess; + struct rte_crypto_op *op; + struct cpt_inst_s inst; + uint64_t lmt_status; + uint64_t lmtline; + uint64_t io_addr; + int ret; + + pend_q = &qp->pend_q; + + lmtline = qp->lmtline.lmt_base; + io_addr = qp->lmtline.io_addr; + + inst.w0.u64 = 0; + inst.w2.u64 = 0; + inst.w3.u64 = 0; + + nb_allowed = qp->lf.nb_desc - pend_q->pending_count; + nb_ops = RTE_MIN(nb_ops, nb_allowed); + + for (count = 0; count < nb_ops; count++) { + op = ops[count]; + infl_req = &pend_q->req_queue[pend_q->enq_tail]; + infl_req->op_flags = 0; + + if (op->type == RTE_CRYPTO_OP_TYPE_SYMMETRIC) { + if (op->sess_type == RTE_CRYPTO_OP_WITH_SESSION) { + sym_op = op->sym; + sess = get_sym_session_private_data( + sym_op->session, + cn9k_cryptodev_driver_id); + ret = cn9k_cpt_sym_inst_fill(qp, op, sess, + infl_req, &inst); + } else { + sess = cn9k_cpt_sym_temp_sess_create(qp, op); + if (unlikely(sess == NULL)) { + plt_dp_err( + "Could not create temp session"); + break; + } + + ret = cn9k_cpt_sym_inst_fill(qp, op, sess, + infl_req, &inst); + if (unlikely(ret)) { + sym_session_clear( + cn9k_cryptodev_driver_id, + op->sym->session); + rte_mempool_put(qp->sess_mp, + op->sym->session); + } + } + } else { + plt_dp_err("Unsupported op type"); + break; + } + + if (unlikely(ret)) { + plt_dp_err("Could not process op: %p", op); + break; + } + + infl_req->cop = op; + + infl_req->res.cn9k.compcode = CPT_COMP_NOT_DONE; + inst.res_addr = (uint64_t)&infl_req->res; + inst.w7.u64 = sess->cpt_inst_w7; + + do { + /* Copy CPT command to LMTLINE */ + memcpy((void *)lmtline, &inst, sizeof(inst)); + + /* + * Make sure compiler does not reorder memcpy and ldeor. + * LMTST transactions are always flushed from the write + * buffer immediately, a DMB is not required to push out + * LMTSTs. + */ + rte_io_wmb(); + lmt_status = roc_lmt_submit_ldeor(io_addr); + } while (lmt_status == 0); + + MOD_INC(pend_q->enq_tail, qp->lf.nb_desc); + } + + pend_q->pending_count += count; + pend_q->time_out = rte_get_timer_cycles() + + DEFAULT_COMMAND_TIMEOUT * rte_get_timer_hz(); + + return count; +} + +void +cn9k_cpt_set_enqdeq_fns(struct rte_cryptodev *dev) +{ + dev->enqueue_burst = cn9k_cpt_enqueue_burst; + + rte_mb(); +} static void cn9k_cpt_dev_info_get(struct rte_cryptodev *dev, diff --git a/drivers/crypto/cnxk/cn9k_cryptodev_ops.h b/drivers/crypto/cnxk/cn9k_cryptodev_ops.h index 72fc297..2277f6b 100644 --- a/drivers/crypto/cnxk/cn9k_cryptodev_ops.h +++ b/drivers/crypto/cnxk/cn9k_cryptodev_ops.h @@ -9,4 +9,6 @@ extern struct rte_cryptodev_ops cn9k_cpt_ops; +void cn9k_cpt_set_enqdeq_fns(struct rte_cryptodev *dev); + #endif /* _CN9K_CRYPTODEV_OPS_H_ */ diff --git a/drivers/crypto/cnxk/cnxk_cryptodev_ops.h b/drivers/crypto/cnxk/cnxk_cryptodev_ops.h index 0a3c705..7995959 100644 --- a/drivers/crypto/cnxk/cnxk_cryptodev_ops.h +++ b/drivers/crypto/cnxk/cnxk_cryptodev_ops.h @@ -11,6 +11,11 @@ #define CNXK_CPT_MIN_HEADROOM_REQ 24 +/* Default command timeout in seconds */ +#define DEFAULT_COMMAND_TIMEOUT 4 + +#define MOD_INC(i, l) ((i) == (l - 1) ? (i) = 0 : (i)++) + struct cpt_qp_meta_info { struct rte_mempool *pool; int mlen; @@ -26,6 +31,10 @@ enum sym_xform_type { CNXK_CPT_CIPHER_DEC_AUTH_VRFY }; +#define CPT_OP_FLAGS_METABUF (1 << 1) +#define CPT_OP_FLAGS_AUTH_VERIFY (1 << 0) +#define CPT_OP_FLAGS_IPSEC_DIR_INBOUND (1 << 2) + struct cpt_inflight_req { union cpt_res_s res; struct rte_crypto_op *cop; -- 2.7.4