* [PATCH rdma-core 0/7] Add mlx5 direct verbs @ 2017-01-26 15:56 Yishai Hadas [not found] ` <1485446182-5109-1-git-send-email-yishaih-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org> 0 siblings, 1 reply; 23+ messages in thread From: Yishai Hadas @ 2017-01-26 15:56 UTC (permalink / raw) To: dledford-H+wXaHxf7aLQT0dZR+AlfA Cc: linux-rdma-u79uwXL29TY76Z2rM5mHXA, yishaih-VPRAkNaXOzVWk0Htik3J/w, leonro-VPRAkNaXOzVWk0Htik3J/w, majd-VPRAkNaXOzVWk0Htik3J/w This patchset from Leon adds direct access to mlx5 devices. The libibverbs API is an abstract API. It is agnostic to any underlying provider specific implementation. While this abstraction has the advantage of user applications portability it has a performance penalty. For some applications optimizing performance is more important than portability. The mlx5 direct verbs API introduced in this patchset is intended for such applications. It exposes mlx5 specific low level data path (send/receive/completion) operations, allowing the application to bypass the libibverbs data path API. The proposed interface consists from small number of hardware specific headers with relevant inline functions and conversion logic from ibverbs structures to mlx5 related structures. New headers are going to be installed to the /usr/include/infiniband/ folder in such way that it will allow future extension and coexistence for other vendors. * /usr/inlcude/infiniband/mlx5/dv.h - shared code * /usr/inlcude/infiniband/mlx5/dv-cqe.h - CQE related functions (general implementation) * /usr/inlcude/infiniband/mlx5/dv-wqe.h - WQE related functions (general and SSE3 implementations) The direct include of dv.h together with linkage to mlx5 library will allow usage of this new interface. Note: all Mellanox NIC devices starting from Connect-IB (Connect-IB, ConnectX-4, ConnectX-4Lx, ConnectX-5 and more to come in the future) implement the mlx5 API, thus using the mlx5 direct verbs does not limit the applications to a single NIC HW device thus keeping some level of portability. Pull request was sent: https://github.com/linux-rdma/rdma-core/pull/63 Yishai Leon Romanovsky (7): mlx5: Add mlx5 direct verbs files mlx5: Add CQE fields mlx5: Add WQE segments implementation mlx5: Use macro for CQE version 1 mlx5: Convert to use predefined get CQE opcodes mlx5: Unify exported and internal CQE format mlx5: Export mlx5 direct verbs interface buildlib/provider.map | 13 +- providers/mlx5/CMakeLists.txt | 2 + providers/mlx5/cq.c | 120 +++---------- providers/mlx5/include/CMakeLists.txt | 5 + providers/mlx5/include/dv-cqe.h | 156 +++++++++++++++++ providers/mlx5/include/dv-wqe.h | 315 ++++++++++++++++++++++++++++++++++ providers/mlx5/include/dv.h | 181 +++++++++++++++++++ providers/mlx5/mlx5.c | 111 +++++++++++- providers/mlx5/mlx5.h | 62 +------ providers/mlx5/qp.c | 2 +- providers/mlx5/wqe.h | 97 +---------- 11 files changed, 806 insertions(+), 258 deletions(-) create mode 100644 providers/mlx5/include/CMakeLists.txt create mode 100644 providers/mlx5/include/dv-cqe.h create mode 100644 providers/mlx5/include/dv-wqe.h create mode 100644 providers/mlx5/include/dv.h -- 1.8.3.1 -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 23+ messages in thread
[parent not found: <1485446182-5109-1-git-send-email-yishaih-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>]
* [PATCH rdma-core 1/7] mlx5: Add mlx5 direct verbs files [not found] ` <1485446182-5109-1-git-send-email-yishaih-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org> @ 2017-01-26 15:56 ` Yishai Hadas [not found] ` <1485446182-5109-2-git-send-email-yishaih-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org> 2017-01-26 15:56 ` [PATCH rdma-core 2/7] mlx5: Add CQE fields Yishai Hadas ` (9 subsequent siblings) 10 siblings, 1 reply; 23+ messages in thread From: Yishai Hadas @ 2017-01-26 15:56 UTC (permalink / raw) To: dledford-H+wXaHxf7aLQT0dZR+AlfA Cc: linux-rdma-u79uwXL29TY76Z2rM5mHXA, yishaih-VPRAkNaXOzVWk0Htik3J/w, leonro-VPRAkNaXOzVWk0Htik3J/w, majd-VPRAkNaXOzVWk0Htik3J/w From: Leon Romanovsky <leonro-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org> Direct verbs feature provides fast data path execution by implementing verbs directly over hardware specification. It consists from the two parts: * Exported functions to translate internal mlx5 structures to be accessible by user applications. * Inline functions, enums and defines to simplify programmer's life. This patch adds following files: * mlx5/dv.h - shared code * mlx5/dv-cqe.h - CQE related functions (general implementation) * mlx5/dv-wqe.h - WQE related functions (general implementation) Signed-off-by: Leon Romanovsky <leonro-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org> Reviewed-by: Majd Dibbiny <majd-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org> Acked-by: Yishai Hadas <yishaih-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org> --- providers/mlx5/CMakeLists.txt | 2 + providers/mlx5/cq.c | 84 +--------------- providers/mlx5/include/CMakeLists.txt | 1 + providers/mlx5/include/dv-cqe.h | 125 +++++++++++++++++++++++ providers/mlx5/include/dv-wqe.h | 140 ++++++++++++++++++++++++++ providers/mlx5/include/dv.h | 181 ++++++++++++++++++++++++++++++++++ providers/mlx5/mlx5.c | 109 ++++++++++++++++++++ providers/mlx5/mlx5.h | 62 +----------- providers/mlx5/qp.c | 2 +- providers/mlx5/wqe.h | 97 +----------------- 10 files changed, 568 insertions(+), 235 deletions(-) create mode 100644 providers/mlx5/include/CMakeLists.txt create mode 100644 providers/mlx5/include/dv-cqe.h create mode 100644 providers/mlx5/include/dv-wqe.h create mode 100644 providers/mlx5/include/dv.h diff --git a/providers/mlx5/CMakeLists.txt b/providers/mlx5/CMakeLists.txt index 44df83b..252243c 100644 --- a/providers/mlx5/CMakeLists.txt +++ b/providers/mlx5/CMakeLists.txt @@ -19,3 +19,5 @@ rdma_provider(mlx5 srq.c verbs.c ) + +add_subdirectory(include) diff --git a/providers/mlx5/cq.c b/providers/mlx5/cq.c index 7ad27a9..e5b50a3 100644 --- a/providers/mlx5/cq.c +++ b/providers/mlx5/cq.c @@ -47,97 +47,17 @@ #include "doorbell.h" enum { - MLX5_CQ_DOORBELL = 0x20 -}; - -enum { CQ_OK = 0, CQ_EMPTY = -1, CQ_POLL_ERR = -2 }; -#define MLX5_CQ_DB_REQ_NOT_SOL (1 << 24) -#define MLX5_CQ_DB_REQ_NOT (0 << 24) - -enum { - MLX5_CQE_SYNDROME_LOCAL_LENGTH_ERR = 0x01, - MLX5_CQE_SYNDROME_LOCAL_QP_OP_ERR = 0x02, - MLX5_CQE_SYNDROME_LOCAL_PROT_ERR = 0x04, - MLX5_CQE_SYNDROME_WR_FLUSH_ERR = 0x05, - MLX5_CQE_SYNDROME_MW_BIND_ERR = 0x06, - MLX5_CQE_SYNDROME_BAD_RESP_ERR = 0x10, - MLX5_CQE_SYNDROME_LOCAL_ACCESS_ERR = 0x11, - MLX5_CQE_SYNDROME_REMOTE_INVAL_REQ_ERR = 0x12, - MLX5_CQE_SYNDROME_REMOTE_ACCESS_ERR = 0x13, - MLX5_CQE_SYNDROME_REMOTE_OP_ERR = 0x14, - MLX5_CQE_SYNDROME_TRANSPORT_RETRY_EXC_ERR = 0x15, - MLX5_CQE_SYNDROME_RNR_RETRY_EXC_ERR = 0x16, - MLX5_CQE_SYNDROME_REMOTE_ABORTED_ERR = 0x22, -}; - -enum { - MLX5_CQE_OWNER_MASK = 1, - MLX5_CQE_REQ = 0, - MLX5_CQE_RESP_WR_IMM = 1, - MLX5_CQE_RESP_SEND = 2, - MLX5_CQE_RESP_SEND_IMM = 3, - MLX5_CQE_RESP_SEND_INV = 4, - MLX5_CQE_RESIZE_CQ = 5, - MLX5_CQE_REQ_ERR = 13, - MLX5_CQE_RESP_ERR = 14, - MLX5_CQE_INVALID = 15, -}; - enum { MLX5_CQ_MODIFY_RESEIZE = 0, MLX5_CQ_MODIFY_MODER = 1, MLX5_CQ_MODIFY_MAPPING = 2, }; -enum { - MLX5_CQE_L2_OK = 1 << 0, - MLX5_CQE_L3_OK = 1 << 1, - MLX5_CQE_L4_OK = 1 << 2, -}; - -enum { - MLX5_CQE_L3_HDR_TYPE_NONE = 0x0, - MLX5_CQE_L3_HDR_TYPE_IPV6 = 0x1, - MLX5_CQE_L3_HDR_TYPE_IPV4 = 0x2, -}; - -struct mlx5_err_cqe { - uint8_t rsvd0[32]; - uint32_t srqn; - uint8_t rsvd1[18]; - uint8_t vendor_err_synd; - uint8_t syndrome; - uint32_t s_wqe_opcode_qpn; - uint16_t wqe_counter; - uint8_t signature; - uint8_t op_own; -}; - -struct mlx5_cqe64 { - uint8_t rsvd0[17]; - uint8_t ml_path; - uint8_t rsvd20[4]; - uint16_t slid; - uint32_t flags_rqpn; - uint8_t hds_ip_ext; - uint8_t l4_hdr_type_etc; - uint16_t vlan_info; - uint32_t srqn_uidx; - uint32_t imm_inval_pkey; - uint8_t rsvd40[4]; - uint32_t byte_cnt; - __be64 timestamp; - uint32_t sop_drop_qpn; - uint16_t wqe_counter; - uint8_t signature; - uint8_t op_own; -}; - int mlx5_stall_num_loop = 60; int mlx5_stall_cq_poll_min = 60; int mlx5_stall_cq_poll_max = 100000; @@ -255,7 +175,7 @@ static inline int handle_responder_lazy(struct mlx5_cq *cq, struct mlx5_cqe64 *c else if (cqe->op_own & MLX5_INLINE_SCATTER_64) err = mlx5_copy_to_recv_wqe(qp, wqe_ctr, cqe - 1, ntohl(cqe->byte_cnt)); - } +} return err; } @@ -1432,7 +1352,7 @@ void __mlx5_cq_clean(struct mlx5_cq *cq, uint32_t rsn, struct mlx5_srq *srq) uint8_t owner_bit; int cqe_version; - if (!cq) + if (!cq || cq->flags & MLX5_CQ_FLAGS_DV_OWNED) return; /* diff --git a/providers/mlx5/include/CMakeLists.txt b/providers/mlx5/include/CMakeLists.txt new file mode 100644 index 0000000..11a2839 --- /dev/null +++ b/providers/mlx5/include/CMakeLists.txt @@ -0,0 +1 @@ +# Empty CMakelists.txt to avoid compilation error diff --git a/providers/mlx5/include/dv-cqe.h b/providers/mlx5/include/dv-cqe.h new file mode 100644 index 0000000..1ad33ad --- /dev/null +++ b/providers/mlx5/include/dv-cqe.h @@ -0,0 +1,125 @@ +/* + * Copyright (c) 2017 Mellanox Technologies, Inc. All rights reserved. + * + * This software is available to you under a choice of one of two + * licenses. You may choose to be licensed under the terms of the GNU + * General Public License (GPL) Version 2, available from the file + * COPYING in the main directory of this source tree, or the + * OpenIB.org BSD license below: + * + * Redistribution and use in source and binary forms, with or + * without modification, are permitted provided that the following + * conditions are met: + * + * - Redistributions of source code must retain the above + * copyright notice, this list of conditions and the following + * disclaimer. + * + * - Redistributions in binary form must reproduce the above + * copyright notice, this list of conditions and the following + * disclaimer in the documentation and/or other materials + * provided with the distribution. + * + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF + * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS + * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN + * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN + * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE + * SOFTWARE. + */ + +#ifndef _MLX5DV_CQE_H_ +#define _MLX5DV_CQE_H_ + +/* For __be64 type */ +#include <linux/types.h> + +enum { + MLX5_INLINE_SCATTER_32 = 0x4, + MLX5_INLINE_SCATTER_64 = 0x8, +}; + +enum { + MLX5_CQE_SYNDROME_LOCAL_LENGTH_ERR = 0x01, + MLX5_CQE_SYNDROME_LOCAL_QP_OP_ERR = 0x02, + MLX5_CQE_SYNDROME_LOCAL_PROT_ERR = 0x04, + MLX5_CQE_SYNDROME_WR_FLUSH_ERR = 0x05, + MLX5_CQE_SYNDROME_MW_BIND_ERR = 0x06, + MLX5_CQE_SYNDROME_BAD_RESP_ERR = 0x10, + MLX5_CQE_SYNDROME_LOCAL_ACCESS_ERR = 0x11, + MLX5_CQE_SYNDROME_REMOTE_INVAL_REQ_ERR = 0x12, + MLX5_CQE_SYNDROME_REMOTE_ACCESS_ERR = 0x13, + MLX5_CQE_SYNDROME_REMOTE_OP_ERR = 0x14, + MLX5_CQE_SYNDROME_TRANSPORT_RETRY_EXC_ERR = 0x15, + MLX5_CQE_SYNDROME_RNR_RETRY_EXC_ERR = 0x16, + MLX5_CQE_SYNDROME_REMOTE_ABORTED_ERR = 0x22, +}; + +enum { + MLX5_CQE_L2_OK = 1 << 0, + MLX5_CQE_L3_OK = 1 << 1, + MLX5_CQE_L4_OK = 1 << 2, +}; + +enum { + MLX5_CQE_L3_HDR_TYPE_NONE = 0x0, + MLX5_CQE_L3_HDR_TYPE_IPV6 = 0x1, + MLX5_CQE_L3_HDR_TYPE_IPV4 = 0x2, +}; + +enum { + MLX5_CQE_OWNER_MASK = 1, + MLX5_CQE_REQ = 0, + MLX5_CQE_RESP_WR_IMM = 1, + MLX5_CQE_RESP_SEND = 2, + MLX5_CQE_RESP_SEND_IMM = 3, + MLX5_CQE_RESP_SEND_INV = 4, + MLX5_CQE_RESIZE_CQ = 5, + MLX5_CQE_REQ_ERR = 13, + MLX5_CQE_RESP_ERR = 14, + MLX5_CQE_INVALID = 15, +}; + +enum { + MLX5_CQ_DOORBELL = 0x20 +}; + +enum { + MLX5_CQ_DB_REQ_NOT_SOL = 1 << 24, + MLX5_CQ_DB_REQ_NOT = 0 << 24, +}; + +struct mlx5_err_cqe { + uint8_t rsvd0[32]; + uint32_t srqn; + uint8_t rsvd1[18]; + uint8_t vendor_err_synd; + uint8_t syndrome; + uint32_t s_wqe_opcode_qpn; + uint16_t wqe_counter; + uint8_t signature; + uint8_t op_own; +}; + +struct mlx5_cqe64 { + uint8_t rsvd0[17]; + uint8_t ml_path; + uint8_t rsvd20[4]; + uint16_t slid; + uint32_t flags_rqpn; + uint8_t hds_ip_ext; + uint8_t l4_hdr_type_etc; + uint16_t vlan_info; + uint32_t srqn_uidx; + uint32_t imm_inval_pkey; + uint8_t rsvd40[4]; + uint32_t byte_cnt; + __be64 timestamp; + uint32_t sop_drop_qpn; + uint16_t wqe_counter; + uint8_t signature; + uint8_t op_own; +}; +#endif /* _MLX5DV_CQE_H_ */ diff --git a/providers/mlx5/include/dv-wqe.h b/providers/mlx5/include/dv-wqe.h new file mode 100644 index 0000000..ce9bd86 --- /dev/null +++ b/providers/mlx5/include/dv-wqe.h @@ -0,0 +1,140 @@ +/* + * Copyright (c) 2017 Mellanox Technologies, Inc. All rights reserved. + * + * This software is available to you under a choice of one of two + * licenses. You may choose to be licensed under the terms of the GNU + * General Public License (GPL) Version 2, available from the file + * COPYING in the main directory of this source tree, or the + * OpenIB.org BSD license below: + * + * Redistribution and use in source and binary forms, with or + * without modification, are permitted provided that the following + * conditions are met: + * + * - Redistributions of source code must retain the above + * copyright notice, this list of conditions and the following + * disclaimer. + * + * - Redistributions in binary form must reproduce the above + * copyright notice, this list of conditions and the following + * disclaimer in the documentation and/or other materials + * provided with the distribution. + * + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF + * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS + * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN + * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN + * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE + * SOFTWARE. + */ + +#ifndef _MLX5DV_WQE_H_ +#define _MLX5DV_WQE_H_ + +#include <arpa/inet.h> + +enum { + MLX5_INVALID_LKEY = 0x100, +}; + +enum { + MLX5_EXTENDED_UD_AV = 0x80000000, +}; + +enum { + MLX5_WQE_CTRL_CQ_UPDATE = 2 << 2, + MLX5_WQE_CTRL_SOLICITED = 1 << 1, + MLX5_WQE_CTRL_FENCE = 4 << 5, + MLX5_WQE_CTRL_INITIATOR_SMALL_FENCE = 1 << 5, +}; + +enum { + MLX5_SEND_WQE_BB = 64, + MLX5_SEND_WQE_SHIFT = 6, +}; + +enum { + MLX5_INLINE_SEG = 0x80000000, +}; + +enum { + MLX5_ETH_WQE_L3_CSUM = (1 << 6), + MLX5_ETH_WQE_L4_CSUM = (1 << 7), +}; + +struct mlx5_wqe_srq_next_seg { + uint8_t rsvd0[2]; + uint16_t next_wqe_index; + uint8_t signature; + uint8_t rsvd1[11]; +}; + +struct mlx5_wqe_data_seg { + uint32_t byte_count; + uint32_t lkey; + uint64_t addr; +}; + +struct mlx5_wqe_ctrl_seg { + uint32_t opmod_idx_opcode; + uint32_t qpn_ds; + uint8_t signature; + uint8_t rsvd[2]; + uint8_t fm_ce_se; + uint32_t imm; +}; + +struct mlx5_wqe_av { + union { + struct { + uint32_t qkey; + uint32_t reserved; + } qkey; + uint64_t dc_key; + } key; + uint32_t dqp_dct; + uint8_t stat_rate_sl; + uint8_t fl_mlid; + uint16_t rlid; + uint8_t reserved0[4]; + uint8_t rmac[6]; + uint8_t tclass; + uint8_t hop_limit; + uint32_t grh_gid_fl; + uint8_t rgid[16]; +}; + +struct mlx5_wqe_datagram_seg { + struct mlx5_wqe_av av; +}; + +struct mlx5_wqe_raddr_seg { + uint64_t raddr; + uint32_t rkey; + uint32_t reserved; +}; + +struct mlx5_wqe_atomic_seg { + uint64_t swap_add; + uint64_t compare; +}; + +struct mlx5_wqe_inl_data_seg { + uint32_t byte_count; +}; + +struct mlx5_wqe_eth_seg { + uint32_t rsvd0; + uint8_t cs_flags; + uint8_t rsvd1; + uint16_t mss; + uint32_t rsvd2; + uint16_t inline_hdr_sz; + uint8_t inline_hdr_start[2]; + uint8_t inline_hdr[16]; +}; + + +#endif /* _MLX5DV_WQE_H_ */ diff --git a/providers/mlx5/include/dv.h b/providers/mlx5/include/dv.h new file mode 100644 index 0000000..ce4af76 --- /dev/null +++ b/providers/mlx5/include/dv.h @@ -0,0 +1,181 @@ +/* + * Copyright (c) 2017 Mellanox Technologies, Inc. All rights reserved. + * + * This software is available to you under a choice of one of two + * licenses. You may choose to be licensed under the terms of the GNU + * General Public License (GPL) Version 2, available from the file + * COPYING in the main directory of this source tree, or the + * OpenIB.org BSD license below: + * + * Redistribution and use in source and binary forms, with or + * without modification, are permitted provided that the following + * conditions are met: + * + * - Redistributions of source code must retain the above + * copyright notice, this list of conditions and the following + * disclaimer. + * + * - Redistributions in binary form must reproduce the above + * copyright notice, this list of conditions and the following + * disclaimer in the documentation and/or other materials + * provided with the distribution. + * + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF + * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS + * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN + * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN + * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE + * SOFTWARE. + */ + +#ifndef _MLX5DV_H_ +#define _MLX5DV_H_ + +/* Always inline the function */ +#ifdef __GNUC__ +#define MLX5DV_ALWAYS_INLINE inline __attribute__((always_inline)) +#else +#define MLX5DV_ALWAYS_INLINE inline +#endif + +#include "dv-cqe.h" +#include "dv-wqe.h" + +enum { + MLX5_RCV_DBR = 0, + MLX5_SND_DBR = 1, +}; + +/* + * Direct verbs device-specific attributes + */ +struct mlx5dv_context { + uint8_t version; + uint64_t flags; + uint64_t comp_mask; +}; + +enum mlx5dv_context_flags { + /* + * This flag indicates if CQE version 0 or 1 is needed. + */ + MLX5DV_CONTEXT_FLAGS_CQE_V1 = (1 << 0), +}; + +/* + * Most device capabilities are exported by ibv_query_device(...), + * but there is HW device-specific information which is important + * for data-path, but isn't provided. + * + * Return 0 on success. + */ +int mlx5dv_query_device(struct ibv_context *ctx_in, + struct mlx5dv_context *attrs_out); + +struct mlx5dv_qp { + uint32_t *dbrec; + struct { + void *buf; + uint32_t wqe_cnt; + uint32_t stride; + } sq; + struct { + void *buf; + uint32_t wqe_cnt; + uint32_t stride; + } rq; + struct { + void *reg; + uint32_t size; + } bf; + uint64_t comp_mask; +}; + +struct mlx5dv_cq { + void *buf; + uint32_t *dbrec; + uint32_t cqe_cnt; + uint32_t cqe_size; + void *uar; + uint32_t cqn; + uint64_t comp_mask; +}; + +struct mlx5dv_srq { + void *buf; + uint32_t *dbrec; + uint32_t stride; + uint32_t head; + uint32_t tail; + uint64_t comp_mask; +}; + +struct mlx5dv_rwq { + void *buf; + uint32_t *dbrec; + uint32_t wqe_cnt; + uint32_t stride; + uint64_t comp_mask; +}; + +struct mlx5dv_obj { + struct { + struct ibv_qp *in; + struct mlx5dv_qp *out; + } qp; + struct { + struct ibv_cq *in; + struct mlx5dv_cq *out; + } cq; + struct { + struct ibv_srq *in; + struct mlx5dv_srq *out; + } srq; + struct { + struct ibv_wq *in; + struct mlx5dv_rwq *out; + } rwq; +}; + +enum mlx5dv_obj_type { + MLX5DV_OBJ_QP = 1 << 0, + MLX5DV_OBJ_CQ = 1 << 1, + MLX5DV_OBJ_SRQ = 1 << 2, + MLX5DV_OBJ_RWQ = 1 << 3, +}; + +/* + * This function will initialize mlx5dv_xxx structs based on supplied type. + * The information for initialization is taken from ibv_xx structs supplied + * as part of input. + * + * Request information of CQ marks its owned by DV for all consumer index + * related actions. + * + * The initialization type can be combination of several types together. + * + * Return: 0 in case of success. + */ +int mlx5dv_init_obj(struct mlx5dv_obj *obj, uint64_t obj_type); + +enum { + MLX5_OPCODE_NOP = 0x00, + MLX5_OPCODE_SEND_INVAL = 0x01, + MLX5_OPCODE_RDMA_WRITE = 0x08, + MLX5_OPCODE_RDMA_WRITE_IMM = 0x09, + MLX5_OPCODE_SEND = 0x0a, + MLX5_OPCODE_SEND_IMM = 0x0b, + MLX5_OPCODE_TSO = 0x0e, + MLX5_OPCODE_RDMA_READ = 0x10, + MLX5_OPCODE_ATOMIC_CS = 0x11, + MLX5_OPCODE_ATOMIC_FA = 0x12, + MLX5_OPCODE_ATOMIC_MASKED_CS = 0x14, + MLX5_OPCODE_ATOMIC_MASKED_FA = 0x15, + MLX5_OPCODE_FMR = 0x19, + MLX5_OPCODE_LOCAL_INVAL = 0x1b, + MLX5_OPCODE_CONFIG_CMD = 0x1f, + MLX5_OPCODE_UMR = 0x25, +}; +#endif /* _MLX5DV_H_ */ diff --git a/providers/mlx5/mlx5.c b/providers/mlx5/mlx5.c index 747e242..1cf5a5f 100644 --- a/providers/mlx5/mlx5.c +++ b/providers/mlx5/mlx5.c @@ -578,6 +578,115 @@ static int mlx5_map_internal_clock(struct mlx5_device *mdev, return 0; } +int mlx5dv_query_device(struct ibv_context *ctx_in, + struct mlx5dv_context *attrs_out) +{ + attrs_out->comp_mask = 0; + attrs_out->version = 0; + attrs_out->flags = 0; + + if (to_mctx(ctx_in)->cqe_version == MLX5_CQE_VERSION_V1) + attrs_out->flags |= MLX5DV_CONTEXT_FLAGS_CQE_V1; + + return 0; +} + +static int mlx5dv_get_qp(struct ibv_qp *qp_in, + struct mlx5dv_qp *qp_out) +{ + struct mlx5_qp *mqp = to_mqp(qp_in); + + qp_out->comp_mask = 0; + qp_out->dbrec = mqp->db; + + if (mqp->sq_buf_size) + /* IBV_QPT_RAW_PACKET */ + qp_out->sq.buf = (void *)((uintptr_t)mqp->sq_buf.buf); + else + qp_out->sq.buf = (void *)((uintptr_t)mqp->buf.buf + mqp->sq.offset); + qp_out->sq.wqe_cnt = mqp->sq.wqe_cnt; + qp_out->sq.stride = 1 << mqp->sq.wqe_shift; + + qp_out->rq.buf = (void *)((uintptr_t)mqp->buf.buf + mqp->rq.offset); + qp_out->rq.wqe_cnt = mqp->rq.wqe_cnt; + qp_out->rq.stride = 1 << mqp->rq.wqe_shift; + + qp_out->bf.reg = mqp->bf->reg; + + if (mqp->bf->uuarn > 0) + qp_out->bf.size = mqp->bf->buf_size; + else + qp_out->bf.size = 0; + + return 0; +} + +static int mlx5dv_get_cq(struct ibv_cq *cq_in, + struct mlx5dv_cq *cq_out) +{ + struct mlx5_cq *mcq = to_mcq(cq_in); + struct mlx5_context *mctx = to_mctx(cq_in->context); + + cq_out->comp_mask = 0; + cq_out->cqn = mcq->cqn; + cq_out->cqe_cnt = mcq->ibv_cq.cqe + 1; + cq_out->cqe_size = mcq->cqe_sz; + cq_out->buf = mcq->active_buf->buf; + cq_out->dbrec = mcq->dbrec; + cq_out->uar = mctx->uar; + + mcq->flags |= MLX5_CQ_FLAGS_DV_OWNED; + + return 0; +} + +static int mlx5dv_get_rwq(struct ibv_wq *wq_in, + struct mlx5dv_rwq *rwq_out) +{ + struct mlx5_rwq *mrwq = to_mrwq(wq_in); + + rwq_out->comp_mask = 0; + rwq_out->buf = mrwq->pbuff; + rwq_out->dbrec = mrwq->recv_db; + rwq_out->wqe_cnt = mrwq->rq.wqe_cnt; + rwq_out->stride = 1 << mrwq->rq.wqe_shift; + + return 0; +} + +static int mlx5dv_get_srq(struct ibv_srq *srq_in, + struct mlx5dv_srq *srq_out) +{ + struct mlx5_srq *msrq; + + msrq = container_of(srq_in, struct mlx5_srq, vsrq.srq); + + srq_out->comp_mask = 0; + srq_out->buf = msrq->buf.buf; + srq_out->dbrec = msrq->db; + srq_out->stride = 1 << msrq->wqe_shift; + srq_out->head = msrq->head; + srq_out->tail = msrq->tail; + + return 0; +} + +int mlx5dv_init_obj(struct mlx5dv_obj *obj, uint64_t obj_type) +{ + int ret = 0; + + if (obj_type & MLX5DV_OBJ_QP) + ret = mlx5dv_get_qp(obj->qp.in, obj->qp.out); + if (!ret && (obj_type & MLX5DV_OBJ_CQ)) + ret = mlx5dv_get_cq(obj->cq.in, obj->cq.out); + if (!ret && (obj_type & MLX5DV_OBJ_SRQ)) + ret = mlx5dv_get_srq(obj->srq.in, obj->srq.out); + if (!ret && (obj_type & MLX5DV_OBJ_RWQ)) + ret = mlx5dv_get_rwq(obj->rwq.in, obj->rwq.out); + + return ret; +} + static int mlx5_init_context(struct verbs_device *vdev, struct ibv_context *ctx, int cmd_fd) { diff --git a/providers/mlx5/mlx5.h b/providers/mlx5/mlx5.h index cf314fc..49ddc1d 100644 --- a/providers/mlx5/mlx5.h +++ b/providers/mlx5/mlx5.h @@ -42,6 +42,7 @@ #include <ccan/list.h> #include "bitmap.h" #include <ccan/minmax.h> +#include "include/dv.h" #ifdef __GNUC__ #define likely(x) __builtin_expect((x), 1) @@ -110,11 +111,6 @@ static inline void mlx5_dbg(FILE *fp, uint32_t mask, const char *fmt, ...) #endif enum { - MLX5_RCV_DBR = 0, - MLX5_SND_DBR = 1, -}; - -enum { MLX5_STAT_RATE_OFFSET = 5 }; @@ -137,36 +133,10 @@ enum { }; enum { - MLX5_SEND_WQE_BB = 64, - MLX5_SEND_WQE_SHIFT = 6, -}; - -enum { MLX5_BF_OFFSET = 0x800 }; enum { - MLX5_INLINE_SCATTER_32 = 0x4, - MLX5_INLINE_SCATTER_64 = 0x8, -}; - -enum { - MLX5_OPCODE_NOP = 0x00, - MLX5_OPCODE_SEND_INVAL = 0x01, - MLX5_OPCODE_RDMA_WRITE = 0x08, - MLX5_OPCODE_RDMA_WRITE_IMM = 0x09, - MLX5_OPCODE_SEND = 0x0a, - MLX5_OPCODE_SEND_IMM = 0x0b, - MLX5_OPCODE_TSO = 0x0e, - MLX5_OPCODE_RDMA_READ = 0x10, - MLX5_OPCODE_ATOMIC_CS = 0x11, - MLX5_OPCODE_ATOMIC_FA = 0x12, - MLX5_OPCODE_ATOMIC_MASKED_CS = 0x14, - MLX5_OPCODE_ATOMIC_MASKED_FA = 0x15, - MLX5_OPCODE_FMR = 0x19, - MLX5_OPCODE_LOCAL_INVAL = 0x1b, - MLX5_OPCODE_CONFIG_CMD = 0x1f, - MLX5_RECV_OPCODE_RDMA_WRITE_IMM = 0x00, MLX5_RECV_OPCODE_SEND = 0x01, MLX5_RECV_OPCODE_SEND_IMM = 0x02, @@ -174,7 +144,6 @@ enum { MLX5_CQE_OPCODE_ERROR = 0x1e, MLX5_CQE_OPCODE_RESIZE = 0x16, - MLX5_OPCODE_UMR = 0x25, }; enum { @@ -182,10 +151,6 @@ enum { }; enum { - MLX5_INLINE_SEG = 0x80000000, -}; - -enum { MLX5_MAX_PORTS_NUM = 2, }; @@ -249,7 +214,7 @@ struct mlx5_context { int prefer_bf; int shut_up_bf; struct { - struct mlx5_qp **table; + struct mlx5_qp **table; int refcnt; } qp_table[MLX5_QP_TABLE_SIZE]; pthread_mutex_t qp_table_mutex; @@ -338,6 +303,7 @@ enum { MLX5_CQ_FLAGS_FOUND_CQES = 1 << 2, MLX5_CQ_FLAGS_EXTENDED = 1 << 3, MLX5_CQ_FLAGS_SINGLE_THREADED = 1 << 4, + MLX5_CQ_FLAGS_DV_OWNED = 1 << 5, }; struct mlx5_cq { @@ -448,29 +414,9 @@ struct mlx5_qp { int rss_qp; }; -struct mlx5_av { - union { - struct { - uint32_t qkey; - uint32_t reserved; - } qkey; - uint64_t dc_key; - } key; - uint32_t dqp_dct; - uint8_t stat_rate_sl; - uint8_t fl_mlid; - uint16_t rlid; - uint8_t reserved0[4]; - uint8_t rmac[6]; - uint8_t tclass; - uint8_t hop_limit; - uint32_t grh_gid_fl; - uint8_t rgid[16]; -}; - struct mlx5_ah { struct ibv_ah ibv_ah; - struct mlx5_av av; + struct mlx5_wqe_av av; bool kern_ah; }; diff --git a/providers/mlx5/qp.c b/providers/mlx5/qp.c index e82b1a0..177b9ae 100644 --- a/providers/mlx5/qp.c +++ b/providers/mlx5/qp.c @@ -212,7 +212,7 @@ static void set_datagram_seg(struct mlx5_wqe_datagram_seg *dseg, struct ibv_send_wr *wr) { memcpy(&dseg->av, &to_mah(wr->wr.ud.ah)->av, sizeof dseg->av); - dseg->av.dqp_dct = htonl(wr->wr.ud.remote_qpn | MLX5_EXTENED_UD_AV); + dseg->av.dqp_dct = htonl(wr->wr.ud.remote_qpn | MLX5_EXTENDED_UD_AV); dseg->av.key.qkey.qkey = htonl(wr->wr.ud.remote_qkey); } diff --git a/providers/mlx5/wqe.h b/providers/mlx5/wqe.h index f097b77..13597bc 100644 --- a/providers/mlx5/wqe.h +++ b/providers/mlx5/wqe.h @@ -33,34 +33,6 @@ #ifndef WQE_H #define WQE_H -enum { - MLX5_WQE_CTRL_CQ_UPDATE = 2 << 2, - MLX5_WQE_CTRL_SOLICITED = 1 << 1, - MLX5_WQE_CTRL_FENCE = 4 << 5, - MLX5_WQE_CTRL_INITIATOR_SMALL_FENCE = 1 << 5, -}; - -enum { - MLX5_INVALID_LKEY = 0x100, -}; - -enum { - MLX5_EXTENED_UD_AV = 0x80000000, -}; - -struct mlx5_wqe_srq_next_seg { - uint8_t rsvd0[2]; - uint16_t next_wqe_index; - uint8_t signature; - uint8_t rsvd1[11]; -}; - -struct mlx5_wqe_data_seg { - uint32_t byte_count; - uint32_t lkey; - uint64_t addr; -}; - struct mlx5_sg_copy_ptr { int index; int offset; @@ -76,36 +48,6 @@ struct mlx5_eqe_qp_srq { uint32_t qp_srq_n; }; -enum { - MLX5_ETH_L2_INLINE_HEADER_SIZE = 18, - MLX5_ETH_L2_MIN_HEADER_SIZE = 14, -}; - -enum { - MLX5_ETH_WQE_L3_CSUM = (1 << 6), - MLX5_ETH_WQE_L4_CSUM = (1 << 7), -}; - -struct mlx5_wqe_eth_seg { - uint32_t rsvd0; - uint8_t cs_flags; - uint8_t rsvd1; - uint16_t mss; - uint32_t rsvd2; - uint16_t inline_hdr_sz; - uint8_t inline_hdr_start[2]; - uint8_t inline_hdr[16]; -}; - -struct mlx5_wqe_ctrl_seg { - uint32_t opmod_idx_opcode; - uint32_t qpn_ds; - uint8_t signature; - uint8_t rsvd[2]; - uint8_t fm_ce_se; - uint32_t imm; -}; - struct mlx5_wqe_xrc_seg { uint32_t xrc_srqn; uint8_t rsvd[12]; @@ -118,42 +60,9 @@ struct mlx5_wqe_masked_atomic_seg { uint64_t compare_mask; }; -struct mlx5_wqe_av { - union { - struct { - uint32_t qkey; - uint32_t reserved; - } qkey; - uint64_t dc_key; - } key; - uint32_t dqp_dct; - uint8_t stat_rate_sl; - uint8_t fl_mlid; - uint16_t rlid; - uint8_t reserved0[10]; - uint8_t tclass; - uint8_t hop_limit; - uint32_t grh_gid_fl; - uint8_t rgid[16]; -}; - -struct mlx5_wqe_datagram_seg { - struct mlx5_wqe_av av; -}; - -struct mlx5_wqe_raddr_seg { - uint64_t raddr; - uint32_t rkey; - uint32_t reserved; -}; - -struct mlx5_wqe_atomic_seg { - uint64_t swap_add; - uint64_t compare; -}; - -struct mlx5_wqe_inl_data_seg { - uint32_t byte_count; +enum { + MLX5_ETH_L2_INLINE_HEADER_SIZE = 18, + MLX5_ETH_L2_MIN_HEADER_SIZE = 14, }; enum { -- 1.8.3.1 -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply related [flat|nested] 23+ messages in thread
[parent not found: <1485446182-5109-2-git-send-email-yishaih-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>]
* Re: [PATCH rdma-core 1/7] mlx5: Add mlx5 direct verbs files [not found] ` <1485446182-5109-2-git-send-email-yishaih-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org> @ 2017-01-26 17:27 ` Jason Gunthorpe 0 siblings, 0 replies; 23+ messages in thread From: Jason Gunthorpe @ 2017-01-26 17:27 UTC (permalink / raw) To: Yishai Hadas Cc: dledford-H+wXaHxf7aLQT0dZR+AlfA, linux-rdma-u79uwXL29TY76Z2rM5mHXA, leonro-VPRAkNaXOzVWk0Htik3J/w, majd-VPRAkNaXOzVWk0Htik3J/w On Thu, Jan 26, 2017 at 05:56:16PM +0200, Yishai Hadas wrote: > +add_subdirectory(include) No need for a subdirectory. I've been flattening things for a while > +/* Always inline the function */ > +#ifdef __GNUC__ > +#define MLX5DV_ALWAYS_INLINE inline __attribute__((always_inline)) > +#else > +#define MLX5DV_ALWAYS_INLINE inline > +#endif I doubt this is necessary... static inline is fine these days. Jason -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 23+ messages in thread
* [PATCH rdma-core 2/7] mlx5: Add CQE fields [not found] ` <1485446182-5109-1-git-send-email-yishaih-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org> 2017-01-26 15:56 ` [PATCH rdma-core 1/7] mlx5: Add mlx5 direct verbs files Yishai Hadas @ 2017-01-26 15:56 ` Yishai Hadas 2017-01-26 15:56 ` [PATCH rdma-core 3/7] mlx5: Add WQE segments implementation Yishai Hadas ` (8 subsequent siblings) 10 siblings, 0 replies; 23+ messages in thread From: Yishai Hadas @ 2017-01-26 15:56 UTC (permalink / raw) To: dledford-H+wXaHxf7aLQT0dZR+AlfA Cc: linux-rdma-u79uwXL29TY76Z2rM5mHXA, yishaih-VPRAkNaXOzVWk0Htik3J/w, leonro-VPRAkNaXOzVWk0Htik3J/w, majd-VPRAkNaXOzVWk0Htik3J/w From: Leon Romanovsky <leonro-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org> Introduce inline calls to get/set for most commonly used CQE data path fields. Signed-off-by: Leon Romanovsky <leonro-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org> Reviewed-by: Majd Dibbiny <majd-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org> Acked-by: Yishai Hadas <yishaih-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org> --- providers/mlx5/include/dv-cqe.h | 31 +++++++++++++++++++++++++++++++ 1 file changed, 31 insertions(+) diff --git a/providers/mlx5/include/dv-cqe.h b/providers/mlx5/include/dv-cqe.h index 1ad33ad..c36b7e0 100644 --- a/providers/mlx5/include/dv-cqe.h +++ b/providers/mlx5/include/dv-cqe.h @@ -122,4 +122,35 @@ struct mlx5_cqe64 { uint8_t signature; uint8_t op_own; }; + +static MLX5DV_ALWAYS_INLINE +uint8_t mlx5dv_get_cqe_owner(struct mlx5_cqe64 *cqe) +{ + return cqe->op_own & 0x1; +} + +static MLX5DV_ALWAYS_INLINE +void mlx5dv_set_cqe_owner(struct mlx5_cqe64 *cqe, uint8_t val) +{ + cqe->op_own = (val & 0x1) | (cqe->op_own & ~0x1); +} + +/* Solicited event */ +static MLX5DV_ALWAYS_INLINE +uint8_t mlx5dv_get_cqe_se(struct mlx5_cqe64 *cqe) +{ + return (cqe->op_own >> 1) & 0x1; +} + +static MLX5DV_ALWAYS_INLINE +uint8_t mlx5dv_get_cqe_format(struct mlx5_cqe64 *cqe) +{ + return (cqe->op_own >> 2) & 0x3; +} + +static MLX5DV_ALWAYS_INLINE +uint8_t mlx5dv_get_cqe_opcode(struct mlx5_cqe64 *cqe) +{ + return cqe->op_own >> 4; +} #endif /* _MLX5DV_CQE_H_ */ -- 1.8.3.1 -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply related [flat|nested] 23+ messages in thread
* [PATCH rdma-core 3/7] mlx5: Add WQE segments implementation [not found] ` <1485446182-5109-1-git-send-email-yishaih-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org> 2017-01-26 15:56 ` [PATCH rdma-core 1/7] mlx5: Add mlx5 direct verbs files Yishai Hadas 2017-01-26 15:56 ` [PATCH rdma-core 2/7] mlx5: Add CQE fields Yishai Hadas @ 2017-01-26 15:56 ` Yishai Hadas 2017-01-26 15:56 ` [PATCH rdma-core 4/7] mlx5: Use macro for CQE version 1 Yishai Hadas ` (7 subsequent siblings) 10 siblings, 0 replies; 23+ messages in thread From: Yishai Hadas @ 2017-01-26 15:56 UTC (permalink / raw) To: dledford-H+wXaHxf7aLQT0dZR+AlfA Cc: linux-rdma-u79uwXL29TY76Z2rM5mHXA, yishaih-VPRAkNaXOzVWk0Htik3J/w, leonro-VPRAkNaXOzVWk0Htik3J/w, majd-VPRAkNaXOzVWk0Htik3J/w From: Leon Romanovsky <leonro-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org> Add generic and x86 SSE3 versions of WQE segments implementations. Signed-off-by: Leon Romanovsky <leonro-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org> Reviewed-by: Majd Dibbiny <majd-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org> Acked-by: Yishai Hadas <yishaih-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org> --- providers/mlx5/include/dv-wqe.h | 175 ++++++++++++++++++++++++++++++++++++++++ 1 file changed, 175 insertions(+) diff --git a/providers/mlx5/include/dv-wqe.h b/providers/mlx5/include/dv-wqe.h index ce9bd86..a819a6f 100644 --- a/providers/mlx5/include/dv-wqe.h +++ b/providers/mlx5/include/dv-wqe.h @@ -34,6 +34,10 @@ #define _MLX5DV_WQE_H_ #include <arpa/inet.h> +#if defined(__SSE3__) +#include <emmintrin.h> +#include <tmmintrin.h> +#endif /* defined(__SSE3__) */ enum { MLX5_INVALID_LKEY = 0x100, @@ -136,5 +140,176 @@ struct mlx5_wqe_eth_seg { uint8_t inline_hdr[16]; }; +/* + * Control segment - contains some control information for the current WQE. + * + * Output: + * seg - control segment to be filled + * Input: + * pi - WQEBB number of the first block of this WQE. + * This number should wrap at 0xffff, regardless of + * size of the WQ. + * opcode - Opcode of this WQE. Encodes the type of operation + * to be executed on the QP. + * opmod - Opcode modifier. + * qp_num - QP/SQ number this WQE is posted to. + * fm_ce_se - FM (fence mode), CE (completion and event mode) + * and SE (solicited event). + * ds - WQE size in octowords (16-byte units). DS accounts for all + * the segments in the WQE as summarized in WQE construction. + * signature - WQE signature. + * imm - Immediate data/Invalidation key/UMR mkey. + */ +static MLX5DV_ALWAYS_INLINE +void mlx5dv_set_ctrl_seg(struct mlx5_wqe_ctrl_seg *seg, uint16_t pi, + uint8_t opcode, uint8_t opmod, uint32_t qp_num, + uint8_t fm_ce_se, uint8_t ds, + uint8_t signature, uint32_t imm) +{ + seg->opmod_idx_opcode = htonl(((uint32_t)opmod << 24) | ((uint32_t)pi << 8) | opcode); + seg->qpn_ds = htonl((qp_num << 8) | ds); + seg->fm_ce_se = fm_ce_se; + seg->signature = signature; + /* + * The caller should prepare "imm" in advance based on WR opcode. + * For IBV_WR_SEND_WITH_IMM and IBV_WR_RDMA_WRITE_WITH_IMM, + * the "imm" should be assigned as is. + * For the IBV_WR_SEND_WITH_INV, it should be htonl(imm). + */ + seg->imm = imm; +} + +/* x86 optimized version of mlx5dv_set_ctrl_seg() + * + * This is useful when doing calculations on large data sets + * for parallel calculations. + * + * It doesn't suit for serialized algorithms. + */ +#if defined(__SSE3__) +static MLX5DV_ALWAYS_INLINE +void mlx5dv_x86_set_ctrl_seg(struct mlx5_wqe_ctrl_seg *seg, uint16_t pi, + uint8_t opcode, uint8_t opmod, uint32_t qp_num, + uint8_t fm_ce_se, uint8_t ds, + uint8_t signature, uint32_t imm) +{ + __m128i val = _mm_set_epi32(imm, qp_num, (ds << 16) | pi, + (signature << 24) | (opcode << 16) | (opmod << 8) | fm_ce_se); + __m128i mask = _mm_set_epi8(12, 13, 14, 15, /* immediate */ + 0, /* signal/fence_mode */ + 0x80, 0x80, /* reserved */ + 3, /* signature */ + 6, /* data size */ + 8, 9, 10, /* QP num */ + 2, /* opcode */ + 4, 5, /* sw_pi in BE */ + 1 /* opmod */ + ); + *(__m128i *) seg = _mm_shuffle_epi8(val, mask); +} +#endif /* defined(__SSE3__) */ + +/* + * Datagram Segment - contains address information required in order + * to form a datagram message. + * + * Output: + * seg - datagram segment to be filled. + * Input: + * key - Q_key/access key. + * dqp_dct - Destination QP number for UD and DCT for DC. + * ext - Address vector extension. + * stat_rate_sl - Maximum static rate control, SL/ethernet priority. + * fl_mlid - Force loopback and source LID for IB. + * rlid - Remote LID + * rmac - Remote MAC + * tclass - GRH tclass/IPv6 tclass/IPv4 ToS + * hop_limit - GRH hop limit/IPv6 hop limit/IPv4 TTL + * grh_gid_fi - GRH, source GID address and IPv6 flow label. + * rgid - Remote GID/IP address. + */ +static MLX5DV_ALWAYS_INLINE +void mlx5dv_set_dgram_seg(struct mlx5_wqe_datagram_seg *seg, + uint64_t key, uint32_t dqp_dct, + uint8_t ext, uint8_t stat_rate_sl, + uint8_t fl_mlid, uint16_t rlid, + uint8_t *rmac, uint8_t tclass, + uint8_t hop_limit, uint32_t grh_gid_fi, + uint8_t *rgid) +{ + /* Always put 64 bits, in q_key, the reserved part will be 0 */ + seg->av.key.dc_key = htonll(key); + seg->av.dqp_dct = htonl(((uint32_t)ext << 31) | dqp_dct); + seg->av.stat_rate_sl = stat_rate_sl; + seg->av.fl_mlid = fl_mlid; + seg->av.rlid = htons(rlid); + memcpy(seg->av.rmac, rmac, 6); + seg->av.tclass = tclass; + seg->av.hop_limit = hop_limit; + seg->av.grh_gid_fl = htonl(grh_gid_fi); + memcpy(seg->av.rgid, rgid, 16); +} + +/* + * Data Segments - contain pointers and a byte count for the scatter/gather list. + * They can optionally contain data, which will save a memory read access for + * gather Work Requests. + */ +static MLX5DV_ALWAYS_INLINE +void mlx5dv_set_data_seg(struct mlx5_wqe_data_seg *seg, + uint32_t length, uint32_t lkey, + uintptr_t address) +{ + seg->byte_count = htonl(length); + seg->lkey = htonl(lkey); + seg->addr = htonll(address); +} +/* + * x86 optimized version of mlx5dv_set_data_seg() + * + * This is useful when doing calculations on large data sets + * for parallel calculations. + * + * It doesn't suit for serialized algorithms. + */ +#if defined(__SSE3__) +static MLX5DV_ALWAYS_INLINE +void mlx5dv_x86_set_data_seg(struct mlx5_wqe_data_seg *seg, + uint32_t length, uint32_t lkey, + uintptr_t address) +{ + __m128i val = _mm_set_epi32((uint32_t)address, (uint32_t)(address >> 32), lkey, length); + __m128i mask = _mm_set_epi8(12, 13, 14, 15, /* local address low */ + 8, 9, 10, 11, /* local address high */ + 4, 5, 6, 7, /* l_key */ + 0, 1, 2, 3 /* byte count */ + ); + *(__m128i *) seg = _mm_shuffle_epi8(val, mask); +} +#endif /* defined(__SSE3__) */ + +/* + * Eth Segment - contains packet headers and information for stateless L2, L3, L4 offloading. + * + * Output: + * seg - Eth segment to be filled. + * Input: + * cs_flags - l3cs/l3cs_inner/l4cs/l4cs_inner. + * mss - Maximum segment size. For TSO WQEs, the number of bytes + * in the TCP payload to be transmitted in each packet. Must + * be 0 on non TSO WQEs. + * inline_hdr_sz - Length of the inlined packet headers. + * inline_hdr_start - Inlined packet header. + */ +static MLX5DV_ALWAYS_INLINE +void mlx5dv_set_eth_seg(struct mlx5_wqe_eth_seg *seg, uint8_t cs_flags, + uint16_t mss, uint16_t inline_hdr_sz, + uint8_t *inline_hdr_start) +{ + seg->cs_flags = cs_flags; + seg->mss = htons(mss); + seg->inline_hdr_sz = htons(inline_hdr_sz); + memcpy(seg->inline_hdr_start, inline_hdr_start, inline_hdr_sz); +} #endif /* _MLX5DV_WQE_H_ */ -- 1.8.3.1 -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply related [flat|nested] 23+ messages in thread
* [PATCH rdma-core 4/7] mlx5: Use macro for CQE version 1 [not found] ` <1485446182-5109-1-git-send-email-yishaih-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org> ` (2 preceding siblings ...) 2017-01-26 15:56 ` [PATCH rdma-core 3/7] mlx5: Add WQE segments implementation Yishai Hadas @ 2017-01-26 15:56 ` Yishai Hadas 2017-01-26 15:56 ` [PATCH rdma-core 5/7] mlx5: Convert to use predefined get CQE opcodes Yishai Hadas ` (6 subsequent siblings) 10 siblings, 0 replies; 23+ messages in thread From: Yishai Hadas @ 2017-01-26 15:56 UTC (permalink / raw) To: dledford-H+wXaHxf7aLQT0dZR+AlfA Cc: linux-rdma-u79uwXL29TY76Z2rM5mHXA, yishaih-VPRAkNaXOzVWk0Htik3J/w, leonro-VPRAkNaXOzVWk0Htik3J/w, majd-VPRAkNaXOzVWk0Htik3J/w From: Leon Romanovsky <leonro-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org> There is macro defined to mark CQE version 1, so let's use it. Signed-off-by: Leon Romanovsky <leonro-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org> Reviewed-by: Majd Dibbiny <majd-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org> Acked-by: Yishai Hadas <yishaih-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org> --- providers/mlx5/mlx5.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/providers/mlx5/mlx5.c b/providers/mlx5/mlx5.c index 1cf5a5f..980e4df 100644 --- a/providers/mlx5/mlx5.c +++ b/providers/mlx5/mlx5.c @@ -771,7 +771,7 @@ static int mlx5_init_context(struct verbs_device *vdev, context->cqe_version = resp.cqe_version; if (context->cqe_version) { - if (context->cqe_version == 1) + if (context->cqe_version == MLX5_CQE_VERSION_V1) mlx5_ctx_ops.poll_cq = mlx5_poll_cq_v1; else goto err_free_bf; -- 1.8.3.1 -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply related [flat|nested] 23+ messages in thread
* [PATCH rdma-core 5/7] mlx5: Convert to use predefined get CQE opcodes [not found] ` <1485446182-5109-1-git-send-email-yishaih-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org> ` (3 preceding siblings ...) 2017-01-26 15:56 ` [PATCH rdma-core 4/7] mlx5: Use macro for CQE version 1 Yishai Hadas @ 2017-01-26 15:56 ` Yishai Hadas 2017-01-26 15:56 ` [PATCH rdma-core 6/7] mlx5: Unify exported and internal CQE format Yishai Hadas ` (5 subsequent siblings) 10 siblings, 0 replies; 23+ messages in thread From: Yishai Hadas @ 2017-01-26 15:56 UTC (permalink / raw) To: dledford-H+wXaHxf7aLQT0dZR+AlfA Cc: linux-rdma-u79uwXL29TY76Z2rM5mHXA, yishaih-VPRAkNaXOzVWk0Htik3J/w, leonro-VPRAkNaXOzVWk0Htik3J/w, majd-VPRAkNaXOzVWk0Htik3J/w From: Leon Romanovsky <leonro-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org> Signed-off-by: Leon Romanovsky <leonro-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org> Reviewed-by: Majd Dibbiny <majd-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org> Acked-by: Yishai Hadas <yishaih-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org> --- providers/mlx5/cq.c | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/providers/mlx5/cq.c b/providers/mlx5/cq.c index e5b50a3..fe7379c 100644 --- a/providers/mlx5/cq.c +++ b/providers/mlx5/cq.c @@ -86,7 +86,7 @@ static void *get_sw_cqe(struct mlx5_cq *cq, int n) cqe64 = (cq->cqe_sz == 64) ? cqe : cqe + 64; - if (likely((cqe64->op_own) >> 4 != MLX5_CQE_INVALID) && + if (likely(mlx5dv_get_cqe_opcode(cqe64) != MLX5_CQE_INVALID) && !((cqe64->op_own & MLX5_CQE_OWNER_MASK) ^ !!(n & (cq->ibv_cq.cqe + 1)))) { return cqe; } else { @@ -544,7 +544,7 @@ static inline int mlx5_parse_cqe(struct mlx5_cq *cq, wc->qp_num = qpn; } - opcode = cqe64->op_own >> 4; + opcode = mlx5dv_get_cqe_opcode(cqe64); switch (opcode) { case MLX5_CQE_REQ: { @@ -1053,7 +1053,7 @@ static inline enum ibv_wc_opcode mlx5_cq_read_wc_opcode(struct ibv_cq_ex *ibcq) { struct mlx5_cq *cq = to_mcq(ibv_cq_ex_to_cq(ibcq)); - switch (cq->cqe64->op_own >> 4) { + switch (mlx5dv_get_cqe_opcode(cq->cqe64)) { case MLX5_CQE_RESP_WR_IMM: return IBV_WC_RECV_RDMA_WITH_IMM; case MLX5_CQE_RESP_SEND: @@ -1111,7 +1111,7 @@ static inline int mlx5_cq_read_wc_flags(struct ibv_cq_ex *ibcq) MLX5_CQE_L3_HDR_TYPE_IPV4)) << IBV_WC_IP_CSUM_OK_SHIFT; - switch (cq->cqe64->op_own >> 4) { + switch (mlx5dv_get_cqe_opcode(cq->cqe64)) { case MLX5_CQE_RESP_WR_IMM: case MLX5_CQE_RESP_SEND_IMM: wc_flags |= IBV_WC_WITH_IMM; @@ -1144,7 +1144,7 @@ static inline uint32_t mlx5_cq_read_wc_imm_data(struct ibv_cq_ex *ibcq) { struct mlx5_cq *cq = to_mcq(ibv_cq_ex_to_cq(ibcq)); - switch (cq->cqe64->op_own >> 4) { + switch (mlx5dv_get_cqe_opcode(cq->cqe64)) { case MLX5_CQE_RESP_SEND_INV: return ntohl(cq->cqe64->imm_inval_pkey); default: @@ -1326,7 +1326,7 @@ static inline int free_res_cqe(struct mlx5_cqe64 *cqe64, uint32_t rsn, { if (cqe_version) { if (is_equal_uidx(cqe64, rsn)) { - if (srq && is_responder(cqe64->op_own >> 4)) + if (srq && is_responder(mlx5dv_get_cqe_opcode(cqe64))) mlx5_free_srq_wqe(srq, ntohs(cqe64->wqe_counter)); return 1; -- 1.8.3.1 -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply related [flat|nested] 23+ messages in thread
* [PATCH rdma-core 6/7] mlx5: Unify exported and internal CQE format [not found] ` <1485446182-5109-1-git-send-email-yishaih-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org> ` (4 preceding siblings ...) 2017-01-26 15:56 ` [PATCH rdma-core 5/7] mlx5: Convert to use predefined get CQE opcodes Yishai Hadas @ 2017-01-26 15:56 ` Yishai Hadas 2017-01-26 15:56 ` [PATCH rdma-core 7/7] mlx5: Export mlx5 direct verbs interface Yishai Hadas ` (4 subsequent siblings) 10 siblings, 0 replies; 23+ messages in thread From: Yishai Hadas @ 2017-01-26 15:56 UTC (permalink / raw) To: dledford-H+wXaHxf7aLQT0dZR+AlfA Cc: linux-rdma-u79uwXL29TY76Z2rM5mHXA, yishaih-VPRAkNaXOzVWk0Htik3J/w, leonro-VPRAkNaXOzVWk0Htik3J/w, majd-VPRAkNaXOzVWk0Htik3J/w From: Leon Romanovsky <leonro-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org> Direct verbs are exporting CQE format, which is mapped to enum MLX5_INLINE_SCATTER_*. The proposed change brings in sync the internal code, exported enum and returned CQE format. Signed-off-by: Leon Romanovsky <leonro-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org> Reviewed-by: Majd Dibbiny <majd-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org> Acked-by: Yishai Hadas <yishaih-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org> --- providers/mlx5/cq.c | 24 ++++++++++++------------ providers/mlx5/include/dv-cqe.h | 4 ++-- 2 files changed, 14 insertions(+), 14 deletions(-) diff --git a/providers/mlx5/cq.c b/providers/mlx5/cq.c index fe7379c..0ae6a19 100644 --- a/providers/mlx5/cq.c +++ b/providers/mlx5/cq.c @@ -151,10 +151,10 @@ static inline int handle_responder_lazy(struct mlx5_cq *cq, struct mlx5_cqe64 *c wqe_ctr = ntohs(cqe->wqe_counter); cq->ibv_cq.wr_id = srq->wrid[wqe_ctr]; mlx5_free_srq_wqe(srq, wqe_ctr); - if (cqe->op_own & MLX5_INLINE_SCATTER_32) + if (mlx5dv_get_cqe_format(cqe) == MLX5_INLINE_SCATTER_32) err = mlx5_copy_to_recv_srq(srq, wqe_ctr, cqe, ntohl(cqe->byte_cnt)); - else if (cqe->op_own & MLX5_INLINE_SCATTER_64) + else if (mlx5dv_get_cqe_format(cqe) == MLX5_INLINE_SCATTER_64) err = mlx5_copy_to_recv_srq(srq, wqe_ctr, cqe - 1, ntohl(cqe->byte_cnt)); } else { @@ -169,10 +169,10 @@ static inline int handle_responder_lazy(struct mlx5_cq *cq, struct mlx5_cqe64 *c wqe_ctr = wq->tail & (wq->wqe_cnt - 1); cq->ibv_cq.wr_id = wq->wrid[wqe_ctr]; ++wq->tail; - if (cqe->op_own & MLX5_INLINE_SCATTER_32) + if (mlx5dv_get_cqe_format(cqe) == MLX5_INLINE_SCATTER_32) err = mlx5_copy_to_recv_wqe(qp, wqe_ctr, cqe, ntohl(cqe->byte_cnt)); - else if (cqe->op_own & MLX5_INLINE_SCATTER_64) + else if (mlx5dv_get_cqe_format(cqe) == MLX5_INLINE_SCATTER_64) err = mlx5_copy_to_recv_wqe(qp, wqe_ctr, cqe - 1, ntohl(cqe->byte_cnt)); } @@ -194,10 +194,10 @@ static inline int handle_responder(struct ibv_wc *wc, struct mlx5_cqe64 *cqe, wqe_ctr = ntohs(cqe->wqe_counter); wc->wr_id = srq->wrid[wqe_ctr]; mlx5_free_srq_wqe(srq, wqe_ctr); - if (cqe->op_own & MLX5_INLINE_SCATTER_32) + if (mlx5dv_get_cqe_format(cqe) == MLX5_INLINE_SCATTER_32) err = mlx5_copy_to_recv_srq(srq, wqe_ctr, cqe, wc->byte_len); - else if (cqe->op_own & MLX5_INLINE_SCATTER_64) + else if (mlx5dv_get_cqe_format(cqe) == MLX5_INLINE_SCATTER_64) err = mlx5_copy_to_recv_srq(srq, wqe_ctr, cqe - 1, wc->byte_len); } else { @@ -216,10 +216,10 @@ static inline int handle_responder(struct ibv_wc *wc, struct mlx5_cqe64 *cqe, wqe_ctr = wq->tail & (wq->wqe_cnt - 1); wc->wr_id = wq->wrid[wqe_ctr]; ++wq->tail; - if (cqe->op_own & MLX5_INLINE_SCATTER_32) + if (mlx5dv_get_cqe_format(cqe) == MLX5_INLINE_SCATTER_32) err = mlx5_copy_to_recv_wqe(qp, wqe_ctr, cqe, wc->byte_len); - else if (cqe->op_own & MLX5_INLINE_SCATTER_64) + else if (mlx5dv_get_cqe_format(cqe) == MLX5_INLINE_SCATTER_64) err = mlx5_copy_to_recv_wqe(qp, wqe_ctr, cqe - 1, wc->byte_len); } @@ -572,10 +572,10 @@ static inline int mlx5_parse_cqe(struct mlx5_cq *cq, wc_byte_len = 8; scatter_out: - if (cqe64->op_own & MLX5_INLINE_SCATTER_32) + if (mlx5dv_get_cqe_format(cqe64) == MLX5_INLINE_SCATTER_32) err = mlx5_copy_to_send_wqe( mqp, wqe_ctr, cqe, wc_byte_len); - else if (cqe64->op_own & MLX5_INLINE_SCATTER_64) + else if (mlx5dv_get_cqe_format(cqe64) == MLX5_INLINE_SCATTER_64) err = mlx5_copy_to_send_wqe( mqp, wqe_ctr, cqe - 1, wc_byte_len); break; @@ -586,10 +586,10 @@ static inline int mlx5_parse_cqe(struct mlx5_cq *cq, } else { handle_good_req(wc, cqe64, wq, idx); - if (cqe64->op_own & MLX5_INLINE_SCATTER_32) + if (mlx5dv_get_cqe_format(cqe64) == MLX5_INLINE_SCATTER_32) err = mlx5_copy_to_send_wqe(mqp, wqe_ctr, cqe, wc->byte_len); - else if (cqe64->op_own & MLX5_INLINE_SCATTER_64) + else if (mlx5dv_get_cqe_format(cqe64) == MLX5_INLINE_SCATTER_64) err = mlx5_copy_to_send_wqe( mqp, wqe_ctr, cqe - 1, wc->byte_len); diff --git a/providers/mlx5/include/dv-cqe.h b/providers/mlx5/include/dv-cqe.h index c36b7e0..d7d3dba 100644 --- a/providers/mlx5/include/dv-cqe.h +++ b/providers/mlx5/include/dv-cqe.h @@ -37,8 +37,8 @@ #include <linux/types.h> enum { - MLX5_INLINE_SCATTER_32 = 0x4, - MLX5_INLINE_SCATTER_64 = 0x8, + MLX5_INLINE_SCATTER_32 = 0x1, + MLX5_INLINE_SCATTER_64 = 0x2, }; enum { -- 1.8.3.1 -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply related [flat|nested] 23+ messages in thread
* [PATCH rdma-core 7/7] mlx5: Export mlx5 direct verbs interface [not found] ` <1485446182-5109-1-git-send-email-yishaih-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org> ` (5 preceding siblings ...) 2017-01-26 15:56 ` [PATCH rdma-core 6/7] mlx5: Unify exported and internal CQE format Yishai Hadas @ 2017-01-26 15:56 ` Yishai Hadas [not found] ` <1485446182-5109-8-git-send-email-yishaih-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org> 2017-01-26 17:25 ` [PATCH rdma-core 0/7] Add mlx5 direct verbs Jason Gunthorpe ` (3 subsequent siblings) 10 siblings, 1 reply; 23+ messages in thread From: Yishai Hadas @ 2017-01-26 15:56 UTC (permalink / raw) To: dledford-H+wXaHxf7aLQT0dZR+AlfA Cc: linux-rdma-u79uwXL29TY76Z2rM5mHXA, yishaih-VPRAkNaXOzVWk0Htik3J/w, leonro-VPRAkNaXOzVWk0Htik3J/w, majd-VPRAkNaXOzVWk0Htik3J/w From: Leon Romanovsky <leonro-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org> Install direct verbs related files into /usr/include/infiniband/mlx5 folder and allow for possible users to explicitly include them into their applications. Signed-off-by: Leon Romanovsky <leonro-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org> Reviewed-by: Majd Dibbiny <majd-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org> Acked-by: Yishai Hadas <yishaih-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org> --- buildlib/provider.map | 13 +++++++++---- providers/mlx5/include/CMakeLists.txt | 6 +++++- 2 files changed, 14 insertions(+), 5 deletions(-) diff --git a/buildlib/provider.map b/buildlib/provider.map index e985a6f..8c7875f 100644 --- a/buildlib/provider.map +++ b/buildlib/provider.map @@ -1,6 +1,11 @@ -/* The providers do not export any symbols at all. Instead they rely on - attribute(constructor) to cause their init function to run at dlopen - time. */ -{ +/* The providers usually do not export any symbols at all. Instead + they rely on attribute(constructor) to cause their init function to + run at dlopen time. + Providers that wish to export symbols should add them below according to + Documentation/versioning.md document. */ +MLX5_1.0 { + global: + mlx5dv_query_device; + mlx5dv_init_obj; local: *; }; diff --git a/providers/mlx5/include/CMakeLists.txt b/providers/mlx5/include/CMakeLists.txt index 11a2839..8e2dc4a 100644 --- a/providers/mlx5/include/CMakeLists.txt +++ b/providers/mlx5/include/CMakeLists.txt @@ -1 +1,5 @@ -# Empty CMakelists.txt to avoid compilation error +publish_headers(infiniband/mlx5 + dv.h + dv-wqe.h + dv-cqe.h +) -- 1.8.3.1 -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply related [flat|nested] 23+ messages in thread
[parent not found: <1485446182-5109-8-git-send-email-yishaih-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>]
* Re: [PATCH rdma-core 7/7] mlx5: Export mlx5 direct verbs interface [not found] ` <1485446182-5109-8-git-send-email-yishaih-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org> @ 2017-01-26 17:30 ` Jason Gunthorpe [not found] ` <20170126173005.GD16562-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org> 0 siblings, 1 reply; 23+ messages in thread From: Jason Gunthorpe @ 2017-01-26 17:30 UTC (permalink / raw) To: Yishai Hadas Cc: dledford-H+wXaHxf7aLQT0dZR+AlfA, linux-rdma-u79uwXL29TY76Z2rM5mHXA, leonro-VPRAkNaXOzVWk0Htik3J/w, majd-VPRAkNaXOzVWk0Htik3J/w On Thu, Jan 26, 2017 at 05:56:22PM +0200, Yishai Hadas wrote: > From: Leon Romanovsky <leonro-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org> > > Install direct verbs related files into > /usr/include/infiniband/mlx5 folder and allow > for possible users to explicitly include them > into their applications. > > Signed-off-by: Leon Romanovsky <leonro-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org> > Reviewed-by: Majd Dibbiny <majd-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org> > Acked-by: Yishai Hadas <yishaih-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org> > buildlib/provider.map | 13 +++++++++---- > providers/mlx5/include/CMakeLists.txt | 6 +++++- > 2 files changed, 14 insertions(+), 5 deletions(-) > > diff --git a/buildlib/provider.map b/buildlib/provider.map > index e985a6f..8c7875f 100644 > +++ b/buildlib/provider.map > @@ -1,6 +1,11 @@ > -/* The providers do not export any symbols at all. Instead they rely on > - attribute(constructor) to cause their init function to run at dlopen > - time. */ > -{ > +/* The providers usually do not export any symbols at all. Instead > + they rely on attribute(constructor) to cause their init function to > + run at dlopen time. > + Providers that wish to export symbols should add them below according to > + Documentation/versioning.md document. */ > +MLX5_1.0 { > + global: > + mlx5dv_query_device; > + mlx5dv_init_obj; > local: *; No, this is no longer just a provider so it doesn't get to use provider.map mlx5 will need its own map file and the various magic I described to become a proper library and a plugin at once. > +publish_headers(infiniband/mlx5 > + dv.h > + dv-wqe.h > + dv-cqe.h > +) not ever dir needs a cmakefile, this is fine in the providers/mlx5 even with the include/ dir. It is easier to understand if things are not spread out so much. Jason -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 23+ messages in thread
[parent not found: <20170126173005.GD16562-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org>]
* Re: [PATCH rdma-core 7/7] mlx5: Export mlx5 direct verbs interface [not found] ` <20170126173005.GD16562-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org> @ 2017-02-03 16:04 ` Christoph Hellwig [not found] ` <20170203160419.GA28953-wEGCiKHe2LqWVfeAwA7xHQ@public.gmane.org> 0 siblings, 1 reply; 23+ messages in thread From: Christoph Hellwig @ 2017-02-03 16:04 UTC (permalink / raw) To: Jason Gunthorpe Cc: Yishai Hadas, dledford-H+wXaHxf7aLQT0dZR+AlfA, linux-rdma-u79uwXL29TY76Z2rM5mHXA, leonro-VPRAkNaXOzVWk0Htik3J/w, majd-VPRAkNaXOzVWk0Htik3J/w On Thu, Jan 26, 2017 at 10:30:05AM -0700, Jason Gunthorpe wrote: > No, this is no longer just a provider so it doesn't get to use > provider.map > > mlx5 will need its own map file and the various magic I described to > become a proper library and a plugin at once. Honestly, I don't think it should be both at once. The library for direct access needs to be in /usr/lib and the provider hidden in the providers directory, and not export anything not in the providers API. I think what needs to happen is a new library for the direct access, and the provider should be layered on top of it. -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 23+ messages in thread
[parent not found: <20170203160419.GA28953-wEGCiKHe2LqWVfeAwA7xHQ@public.gmane.org>]
* Re: [PATCH rdma-core 7/7] mlx5: Export mlx5 direct verbs interface [not found] ` <20170203160419.GA28953-wEGCiKHe2LqWVfeAwA7xHQ@public.gmane.org> @ 2017-02-03 16:48 ` Jason Gunthorpe 0 siblings, 0 replies; 23+ messages in thread From: Jason Gunthorpe @ 2017-02-03 16:48 UTC (permalink / raw) To: Christoph Hellwig Cc: Yishai Hadas, dledford-H+wXaHxf7aLQT0dZR+AlfA, linux-rdma-u79uwXL29TY76Z2rM5mHXA, leonro-VPRAkNaXOzVWk0Htik3J/w, majd-VPRAkNaXOzVWk0Htik3J/w On Fri, Feb 03, 2017 at 08:04:19AM -0800, Christoph Hellwig wrote: > On Thu, Jan 26, 2017 at 10:30:05AM -0700, Jason Gunthorpe wrote: > > No, this is no longer just a provider so it doesn't get to use > > provider.map > > > > mlx5 will need its own map file and the various magic I described to > > become a proper library and a plugin at once. > > Honestly, I don't think it should be both at once. The library for > direct access needs to be in /usr/lib and the provider hidden in the > providers directory, and not export anything not in the providers API. Provider libraries don't export any symbols at all. That is the main reason having a library be both could potentially work out OK.. But it would have to be a proper /usr/lib/../ library first, and be symlinked to a provider lib name second... And they will have to make sure that dlopen figures this out and only loads one copy.. > I think what needs to happen is a new library for the direct access, > and the provider should be layered on top of it. I suspect that might just make a bigger mess, it looks like there is some sharing of internal statics and what not - so the two libraries would end up having their own private symbol interface, which is looking like be a worse situation then the above... Jason -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [PATCH rdma-core 0/7] Add mlx5 direct verbs [not found] ` <1485446182-5109-1-git-send-email-yishaih-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org> ` (6 preceding siblings ...) 2017-01-26 15:56 ` [PATCH rdma-core 7/7] mlx5: Export mlx5 direct verbs interface Yishai Hadas @ 2017-01-26 17:25 ` Jason Gunthorpe [not found] ` <20170126172511.GB16562-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org> 2017-02-01 15:56 ` Christoph Lameter ` (2 subsequent siblings) 10 siblings, 1 reply; 23+ messages in thread From: Jason Gunthorpe @ 2017-01-26 17:25 UTC (permalink / raw) To: Yishai Hadas Cc: dledford-H+wXaHxf7aLQT0dZR+AlfA, linux-rdma-u79uwXL29TY76Z2rM5mHXA, leonro-VPRAkNaXOzVWk0Htik3J/w, majd-VPRAkNaXOzVWk0Htik3J/w On Thu, Jan 26, 2017 at 05:56:15PM +0200, Yishai Hadas wrote: > This patchset from Leon adds direct access to mlx5 devices. > > The libibverbs API is an abstract API. It is agnostic to any underlying > provider specific implementation. While this abstraction has the advantage of > user applications portability it has a performance penalty. For some > applications optimizing performance is more important than portability. > > The mlx5 direct verbs API introduced in this patchset is intended for such > applications. It exposes mlx5 specific low level data path > (send/receive/completion) operations, allowing the application to bypass the > libibverbs data path API. Applications need to link directly to the libraries that provide the symbols they use - this approach you have where linking happens implicitly at runtime is not OK. Somehow libmlx needs to be in the standard ld.so search path, and available for linking via gcc, as well as continuing to be a plugin.. The best choice might be to put it in /usr/lib/../libmlx5.so.1 and symlink that file to /usr/lib/../libibverbs/libmlx5-rdmav2.so This series also needs patches to the included packaging files to bundle the new files. (buildlib/cbuild exists for a reason, be sure to test the packaging) These probably need a man page as well. It wasn't instantly obvious to me how these were supposed to work. Jason -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 23+ messages in thread
[parent not found: <20170126172511.GB16562-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org>]
* Re: [PATCH rdma-core 0/7] Add mlx5 direct verbs [not found] ` <20170126172511.GB16562-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org> @ 2017-01-26 21:54 ` Yishai Hadas [not found] ` <ac21fe37-d617-e5a8-cd67-e8fdfbd69d55-LDSdmyG8hGV8YrgS2mwiifqBs+8SCbDb@public.gmane.org> 0 siblings, 1 reply; 23+ messages in thread From: Yishai Hadas @ 2017-01-26 21:54 UTC (permalink / raw) To: Jason Gunthorpe Cc: Yishai Hadas, dledford-H+wXaHxf7aLQT0dZR+AlfA, linux-rdma-u79uwXL29TY76Z2rM5mHXA, leonro-VPRAkNaXOzVWk0Htik3J/w, majd-VPRAkNaXOzVWk0Htik3J/w On 1/26/2017 7:25 PM, Jason Gunthorpe wrote: > On Thu, Jan 26, 2017 at 05:56:15PM +0200, Yishai Hadas wrote: >> This patchset from Leon adds direct access to mlx5 devices. >> >> The libibverbs API is an abstract API. It is agnostic to any underlying >> provider specific implementation. While this abstraction has the advantage of >> user applications portability it has a performance penalty. For some >> applications optimizing performance is more important than portability. >> >> The mlx5 direct verbs API introduced in this patchset is intended for such >> applications. It exposes mlx5 specific low level data path >> (send/receive/completion) operations, allowing the application to bypass the >> libibverbs data path API. > > Applications need to link directly to the libraries that provide the > symbols they use - this approach you have where linking happens > implicitly at runtime is not OK. Correct, the link will be explicitly by the application at compile time so that the exported symbols will be resolved. The below note from the cover letter planned to point on that: "The direct include of dv.h together with linkage to mlx5 library will allow usage of this new interface" > Somehow libmlx needs to be in the standard ld.so search path, and > available for linking via gcc, as well as continuing to be a plugin.. > The best choice might be to put it in /usr/lib/../libmlx5.so.1 and > symlink that file to /usr/lib/../libibverbs/libmlx5-rdmav2.so You are pointing on same usage as done for libibverbs which has the symlink, correct ? Alternatively, application can use the linker flags as part of its Makefile to find the shared library location, isn't it ? > This series also needs patches to the included packaging files to > bundle the new files. (buildlib/cbuild exists for a reason, be sure to > test the packaging) Thanks for pointing on, we'll add and test the packaging. > These probably need a man page as well. It wasn't instantly obvious to > me how these were supposed to work. Sure, we can add some man page here. -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 23+ messages in thread
[parent not found: <ac21fe37-d617-e5a8-cd67-e8fdfbd69d55-LDSdmyG8hGV8YrgS2mwiifqBs+8SCbDb@public.gmane.org>]
* Re: [PATCH rdma-core 0/7] Add mlx5 direct verbs [not found] ` <ac21fe37-d617-e5a8-cd67-e8fdfbd69d55-LDSdmyG8hGV8YrgS2mwiifqBs+8SCbDb@public.gmane.org> @ 2017-01-26 22:02 ` Jason Gunthorpe 0 siblings, 0 replies; 23+ messages in thread From: Jason Gunthorpe @ 2017-01-26 22:02 UTC (permalink / raw) To: Yishai Hadas Cc: Yishai Hadas, dledford-H+wXaHxf7aLQT0dZR+AlfA, linux-rdma-u79uwXL29TY76Z2rM5mHXA, leonro-VPRAkNaXOzVWk0Htik3J/w, majd-VPRAkNaXOzVWk0Htik3J/w On Thu, Jan 26, 2017 at 11:54:53PM +0200, Yishai Hadas wrote: > Correct, the link will be explicitly by the application at compile time so > that the exported symbols will be resolved. Okay, then you just need to make that possible. > >Somehow libmlx needs to be in the standard ld.so search path, and > >available for linking via gcc, as well as continuing to be a plugin.. > >The best choice might be to put it in /usr/lib/../libmlx5.so.1 and > >symlink that file to /usr/lib/../libibverbs/libmlx5-rdmav2.so > > You are pointing on same usage as done for libibverbs which has the symlink, > correct ? libibverbs has a symlink for a different purpose. > Alternatively, application can use the linker flags as part of its > Makefile to find the shared library location, isn't it ? No, that would require applications to hardwire /usr/lib/.../libibverbs/libmlx5-rdmav2.so as a path. Do not want that. Make it so gcc -libverbs_dv_mlx5 foo.c Is all that is required. Jason -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [PATCH rdma-core 0/7] Add mlx5 direct verbs [not found] ` <1485446182-5109-1-git-send-email-yishaih-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org> ` (7 preceding siblings ...) 2017-01-26 17:25 ` [PATCH rdma-core 0/7] Add mlx5 direct verbs Jason Gunthorpe @ 2017-02-01 15:56 ` Christoph Lameter [not found] ` <alpine.DEB.2.20.1702010956080.12203-wcBtFHqTun5QOdAKl3ChDw@public.gmane.org> 2017-02-02 8:52 ` Sagi Grimberg 2017-02-06 10:56 ` Amrani, Ram 10 siblings, 1 reply; 23+ messages in thread From: Christoph Lameter @ 2017-02-01 15:56 UTC (permalink / raw) To: Yishai Hadas Cc: dledford-H+wXaHxf7aLQT0dZR+AlfA, linux-rdma-u79uwXL29TY76Z2rM5mHXA, leonro-VPRAkNaXOzVWk0Htik3J/w, majd-VPRAkNaXOzVWk0Htik3J/w Looks fairly straightforward from a user point of view. I hope you iterate this patchset fast so that we can get it soon? -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 23+ messages in thread
[parent not found: <alpine.DEB.2.20.1702010956080.12203-wcBtFHqTun5QOdAKl3ChDw@public.gmane.org>]
* Re: [PATCH rdma-core 0/7] Add mlx5 direct verbs [not found] ` <alpine.DEB.2.20.1702010956080.12203-wcBtFHqTun5QOdAKl3ChDw@public.gmane.org> @ 2017-02-01 16:17 ` Leon Romanovsky 0 siblings, 0 replies; 23+ messages in thread From: Leon Romanovsky @ 2017-02-01 16:17 UTC (permalink / raw) To: Christoph Lameter Cc: Yishai Hadas, dledford-H+wXaHxf7aLQT0dZR+AlfA, linux-rdma-u79uwXL29TY76Z2rM5mHXA, majd-VPRAkNaXOzVWk0Htik3J/w [-- Attachment #1: Type: text/plain, Size: 467 bytes --] On Wed, Feb 01, 2017 at 09:56:56AM -0600, Christoph Lameter wrote: > > Looks fairly straightforward from a user point of view. I hope you iterate > this patchset fast so that we can get it soon? Yes, we have no plans to delay it. Thanks > > -- > To unsubscribe from this list: send the line "unsubscribe linux-rdma" in > the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org > More majordomo info at http://vger.kernel.org/majordomo-info.html [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 833 bytes --] ^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [PATCH rdma-core 0/7] Add mlx5 direct verbs [not found] ` <1485446182-5109-1-git-send-email-yishaih-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org> ` (8 preceding siblings ...) 2017-02-01 15:56 ` Christoph Lameter @ 2017-02-02 8:52 ` Sagi Grimberg [not found] ` <782c31cb-e247-1094-955f-2766f9c66cbd-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org> 2017-02-06 10:56 ` Amrani, Ram 10 siblings, 1 reply; 23+ messages in thread From: Sagi Grimberg @ 2017-02-02 8:52 UTC (permalink / raw) To: Yishai Hadas, dledford-H+wXaHxf7aLQT0dZR+AlfA Cc: linux-rdma-u79uwXL29TY76Z2rM5mHXA, leonro-VPRAkNaXOzVWk0Htik3J/w, majd-VPRAkNaXOzVWk0Htik3J/w > This patchset from Leon adds direct access to mlx5 devices. > > The libibverbs API is an abstract API. It is agnostic to any underlying > provider specific implementation. While this abstraction has the advantage of > user applications portability it has a performance penalty. For some > applications optimizing performance is more important than portability. > > The mlx5 direct verbs API introduced in this patchset is intended for such > applications. It exposes mlx5 specific low level data path > (send/receive/completion) operations, allowing the application to bypass the > libibverbs data path API. Yishai and Leon this is good stuff !! Couple of comments: - The user of this is pretty intimate with the driver, which makes a somewhat strange combo of having both ibv_ and mlx5dv_ API. I understand that it can inflate the the implementation, but perhaps for completeness we can have also the control plane API as well (even as a trampoline to the generic equivalent where it's useless). The rational here is that extended features that needs allocation flags can be contained to mlx5 and not pollute/inflate the generic verbs implementation. - This brings me to the next step of this. What is still missing is how we allow the user to have more control and information about how his db records (namely UARs) are allocated. How does the user know if he needs to protect the DB write? Is it BF? How can the user ask for more UARs than originally allocated to it by the kernel? The user of this can really use an API that allows more control of this area. Hopefully we can come up with something that is abstract enough to not expose how exactly UARs are strided on pages and the exact interface with the kernel, but still let the user choose if to use a dedicated db_rec and let the driver manage all the internal details. Thoughts? -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 23+ messages in thread
[parent not found: <782c31cb-e247-1094-955f-2766f9c66cbd-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org>]
* Re: [PATCH rdma-core 0/7] Add mlx5 direct verbs [not found] ` <782c31cb-e247-1094-955f-2766f9c66cbd-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org> @ 2017-02-02 13:15 ` Yishai Hadas 0 siblings, 0 replies; 23+ messages in thread From: Yishai Hadas @ 2017-02-02 13:15 UTC (permalink / raw) To: Sagi Grimberg Cc: Yishai Hadas, dledford-H+wXaHxf7aLQT0dZR+AlfA, linux-rdma-u79uwXL29TY76Z2rM5mHXA, leonro-VPRAkNaXOzVWk0Htik3J/w, majd-VPRAkNaXOzVWk0Htik3J/w On 2/2/2017 10:52 AM, Sagi Grimberg wrote: > >> This patchset from Leon adds direct access to mlx5 devices. >> >> The libibverbs API is an abstract API. It is agnostic to any underlying >> provider specific implementation. While this abstraction has the >> advantage of >> user applications portability it has a performance penalty. For some >> applications optimizing performance is more important than portability. >> >> The mlx5 direct verbs API introduced in this patchset is intended for >> such >> applications. It exposes mlx5 specific low level data path >> (send/receive/completion) operations, allowing the application to >> bypass the >> libibverbs data path API. > > Yishai and Leon this is good stuff !! Thanks Sagi for pointing on, it's really appreciated. > Couple of comments: > > - The user of this is pretty intimate with the driver, which makes a > somewhat strange combo of having both ibv_ and mlx5dv_ API. I > understand that it can inflate the the implementation, but perhaps > for completeness we can have also the control plane API as well (even > as a trampoline to the generic equivalent where it's useless). > > The rational here is that extended features that needs allocation > flags can be contained to mlx5 and not pollute/inflate the generic > verbs implementation. This series opens the door for optimizing the data path flows by exposing the internal driver stuff that is required in the general data path flow. It enables an application to optimize its own specific flow without considering the general case which involves branches that are not relevant to its use case. Re control path: Extending control path for driver specific configuration is really something that exists in our plans. It may be achieved by extending the above direct channel in a second phase with some extra APIs or by introducing some vendor channel mechanism to enable that on top of current control verbs. I believe that triggering that will be some feature request or a specific need that will come from the field/customers that will require this functionality. > - This brings me to the next step of this. What is still missing is > how we allow the user to have more control and information about how > his db records (namely UARs) are allocated Are you referring to some specific information that exists in mlx5 driver which was not exposed by the direct API ? please have a look in dv.h for mlx5dv_init_obj and in mlx5 driver code and let us know if we missed something useful, we can add it. Please consider that the above API was introduced with some extension mechanism (i.e. comp_mask) so that we can extend it in the future without breaking compatibility with applications that will use it. >. How does the user know if he needs to protect the DB write? Once an application uses the direct flow the locking scheme is fully managed by itself. We don't expect mixed flows in the data path for both direct/non-direct access by same application. > Is it BF? This information is based on mlx5dv_qp->bf->size, in case of 0 it's not a BF. Please note that as part of V1 we plan to introduce a man page for mlx5 direct verbs which may point on that, in addition to other notes that I pointed on above. > How can the user ask for more UARs than originally allocated to it by the kernel? > The user of this can really use an API that allows more control > of this area. This is some functionality that currently doesn't exist even in the *non* direct mode. It should be considered as some feature request regardless the above series. Please note that moving forward with this series and concept opens the door to give such an API which is some vendor specific more quickly and easily. -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 23+ messages in thread
* RE: [PATCH rdma-core 0/7] Add mlx5 direct verbs [not found] ` <1485446182-5109-1-git-send-email-yishaih-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org> ` (9 preceding siblings ...) 2017-02-02 8:52 ` Sagi Grimberg @ 2017-02-06 10:56 ` Amrani, Ram [not found] ` <SN1PR07MB22070381E7566FA819E5F8F8F8400-mikhvbZlbf8TSoR2DauN2+FPX92sqiQdvxpqHgZTriW3zl9H0oFU5g@public.gmane.org> 10 siblings, 1 reply; 23+ messages in thread From: Amrani, Ram @ 2017-02-06 10:56 UTC (permalink / raw) To: Yishai Hadas, dledford-H+wXaHxf7aLQT0dZR+AlfA Cc: linux-rdma-u79uwXL29TY76Z2rM5mHXA, leonro-VPRAkNaXOzVWk0Htik3J/w, majd-VPRAkNaXOzVWk0Htik3J/w > This patchset from Leon adds direct access to mlx5 devices. > > The libibverbs API is an abstract API. It is agnostic to any underlying > provider specific implementation. While this abstraction has the advantage of > user applications portability it has a performance penalty. For some > applications optimizing performance is more important than portability. > > The mlx5 direct verbs API introduced in this patchset is intended for such > applications. It exposes mlx5 specific low level data path > (send/receive/completion) operations, allowing the application to bypass the > libibverbs data path API. > > The proposed interface consists from small number of hardware specific headers > with relevant inline functions and conversion logic from ibverbs structures to > mlx5 related structures. > Hi Yishay, While I'm always for better performance, I'm not sure driver specific APIs is the way. This will hurt portability for users. Thanks, Ram -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 23+ messages in thread
[parent not found: <SN1PR07MB22070381E7566FA819E5F8F8F8400-mikhvbZlbf8TSoR2DauN2+FPX92sqiQdvxpqHgZTriW3zl9H0oFU5g@public.gmane.org>]
* Re: [PATCH rdma-core 0/7] Add mlx5 direct verbs [not found] ` <SN1PR07MB22070381E7566FA819E5F8F8F8400-mikhvbZlbf8TSoR2DauN2+FPX92sqiQdvxpqHgZTriW3zl9H0oFU5g@public.gmane.org> @ 2017-02-06 11:46 ` Leon Romanovsky [not found] ` <20170206114609.GK6005-U/DQcQFIOTAAJjI8aNfphQ@public.gmane.org> 0 siblings, 1 reply; 23+ messages in thread From: Leon Romanovsky @ 2017-02-06 11:46 UTC (permalink / raw) To: Amrani, Ram Cc: Yishai Hadas, dledford-H+wXaHxf7aLQT0dZR+AlfA, linux-rdma-u79uwXL29TY76Z2rM5mHXA, majd-VPRAkNaXOzVWk0Htik3J/w [-- Attachment #1: Type: text/plain, Size: 1458 bytes --] On Mon, Feb 06, 2017 at 10:56:56AM +0000, Amrani, Ram wrote: > > This patchset from Leon adds direct access to mlx5 devices. > > > > The libibverbs API is an abstract API. It is agnostic to any underlying > > provider specific implementation. While this abstraction has the advantage of > > user applications portability it has a performance penalty. For some > > applications optimizing performance is more important than portability. > > > > The mlx5 direct verbs API introduced in this patchset is intended for such > > applications. It exposes mlx5 specific low level data path > > (send/receive/completion) operations, allowing the application to bypass the > > libibverbs data path API. > > > > The proposed interface consists from small number of hardware specific headers > > with relevant inline functions and conversion logic from ibverbs structures to > > mlx5 related structures. > > > > Hi Yishay, > While I'm always for better performance, I'm not sure driver specific APIs is the way. > This will hurt portability for users. Hi Ram, As it was expressed in cover letter, this feature is intended for users who want performance and understand the disadvantages of such direct approach. Thanks > > Thanks, > Ram > > > -- > To unsubscribe from this list: send the line "unsubscribe linux-rdma" in > the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org > More majordomo info at http://vger.kernel.org/majordomo-info.html [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 833 bytes --] ^ permalink raw reply [flat|nested] 23+ messages in thread
[parent not found: <20170206114609.GK6005-U/DQcQFIOTAAJjI8aNfphQ@public.gmane.org>]
* Re: [PATCH rdma-core 0/7] Add mlx5 direct verbs [not found] ` <20170206114609.GK6005-U/DQcQFIOTAAJjI8aNfphQ@public.gmane.org> @ 2017-02-06 16:39 ` Jason Gunthorpe [not found] ` <20170206163937.GB14714-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org> 0 siblings, 1 reply; 23+ messages in thread From: Jason Gunthorpe @ 2017-02-06 16:39 UTC (permalink / raw) To: Leon Romanovsky Cc: Amrani, Ram, Yishai Hadas, dledford-H+wXaHxf7aLQT0dZR+AlfA, linux-rdma-u79uwXL29TY76Z2rM5mHXA, majd-VPRAkNaXOzVWk0Htik3J/w On Mon, Feb 06, 2017 at 01:46:09PM +0200, Leon Romanovsky wrote: > On Mon, Feb 06, 2017 at 10:56:56AM +0000, Amrani, Ram wrote: > > > This patchset from Leon adds direct access to mlx5 devices. > > > > > > The libibverbs API is an abstract API. It is agnostic to any underlying > > > provider specific implementation. While this abstraction has the advantage of > > > user applications portability it has a performance penalty. For some > > > applications optimizing performance is more important than portability. > > > > > > The mlx5 direct verbs API introduced in this patchset is intended for such > > > applications. It exposes mlx5 specific low level data path > > > (send/receive/completion) operations, allowing the application to bypass the > > > libibverbs data path API. > > > > > > The proposed interface consists from small number of hardware specific headers > > > with relevant inline functions and conversion logic from ibverbs structures to > > > mlx5 related structures. > > > > > While I'm always for better performance, I'm not sure driver specific APIs is the way. > > This will hurt portability for users. > > As it was expressed in cover letter, this feature is intended for > users who want performance and understand the disadvantages of such > direct approach. To expound on that - the trade off we discussed in-person was we either do something like this or continually take endless vendor-specific patches to add 'common' core code for unstandardized chip-specific features. This seemed like the least bad trade off, if another vendor wants the same things then they could work together and make an actual multi-chip common API. I was expecting this interface to be used in only a few places, like the next level of middleware libraries (libfabric, mxm, etc) not for many end-users. I'm still not completely sure what this API is intended to do, if it is really just performance then I am less supportive and would rather see an approach like libfabric where a user can build a single provider verbs with the same source API, but optimized.... Jason -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 23+ messages in thread
[parent not found: <20170206163937.GB14714-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org>]
* Re: [PATCH rdma-core 0/7] Add mlx5 direct verbs [not found] ` <20170206163937.GB14714-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org> @ 2017-02-08 18:45 ` Sagi Grimberg 0 siblings, 0 replies; 23+ messages in thread From: Sagi Grimberg @ 2017-02-08 18:45 UTC (permalink / raw) To: Jason Gunthorpe, Leon Romanovsky Cc: Amrani, Ram, Yishai Hadas, dledford-H+wXaHxf7aLQT0dZR+AlfA, linux-rdma-u79uwXL29TY76Z2rM5mHXA, majd-VPRAkNaXOzVWk0Htik3J/w Hey Jason, > To expound on that - the trade off we discussed in-person was we > either do something like this or continually take endless > vendor-specific patches to add 'common' core code for unstandardized > chip-specific features. > > This seemed like the least bad trade off, if another vendor wants the > same things then they could work together and make an actual > multi-chip common API. I was expecting this interface to be used in > only a few places, like the next level of middleware libraries > (libfabric, mxm, etc) not for many end-users. I think this is exactly the use-case. Something that looks like a user-space driver on the back-side of a common API. > I'm still not completely sure what this API is intended to do, if it is > really just performance then I am less supportive and would rather see > an approach like libfabric where a user can build a single provider verbs > with the same source API, but optimized.... I'm also not in favor for having this code-path solely for performance reasons, if the common stack is not good enough, let's fix it. What I think this is trying to solve is the continuous inflation of non-standard extensions to the common API. These features raised constant complaints in code review before, which in turn caused the vendors to place their extensions in an internal packages (e.g. MLNX_OFED) and also educate their customers to use their packaged drivers (which usually broke the rest of the world). So I can understand Mellanox wanting to expose vendor specific capabilities and move away from their in-house distribution of user-space packages, and I think we should all encourage it. And, this is also why I think that another set of a common API is a good choice. -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 23+ messages in thread
end of thread, other threads:[~2017-02-08 18:45 UTC | newest] Thread overview: 23+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2017-01-26 15:56 [PATCH rdma-core 0/7] Add mlx5 direct verbs Yishai Hadas [not found] ` <1485446182-5109-1-git-send-email-yishaih-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org> 2017-01-26 15:56 ` [PATCH rdma-core 1/7] mlx5: Add mlx5 direct verbs files Yishai Hadas [not found] ` <1485446182-5109-2-git-send-email-yishaih-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org> 2017-01-26 17:27 ` Jason Gunthorpe 2017-01-26 15:56 ` [PATCH rdma-core 2/7] mlx5: Add CQE fields Yishai Hadas 2017-01-26 15:56 ` [PATCH rdma-core 3/7] mlx5: Add WQE segments implementation Yishai Hadas 2017-01-26 15:56 ` [PATCH rdma-core 4/7] mlx5: Use macro for CQE version 1 Yishai Hadas 2017-01-26 15:56 ` [PATCH rdma-core 5/7] mlx5: Convert to use predefined get CQE opcodes Yishai Hadas 2017-01-26 15:56 ` [PATCH rdma-core 6/7] mlx5: Unify exported and internal CQE format Yishai Hadas 2017-01-26 15:56 ` [PATCH rdma-core 7/7] mlx5: Export mlx5 direct verbs interface Yishai Hadas [not found] ` <1485446182-5109-8-git-send-email-yishaih-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org> 2017-01-26 17:30 ` Jason Gunthorpe [not found] ` <20170126173005.GD16562-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org> 2017-02-03 16:04 ` Christoph Hellwig [not found] ` <20170203160419.GA28953-wEGCiKHe2LqWVfeAwA7xHQ@public.gmane.org> 2017-02-03 16:48 ` Jason Gunthorpe 2017-01-26 17:25 ` [PATCH rdma-core 0/7] Add mlx5 direct verbs Jason Gunthorpe [not found] ` <20170126172511.GB16562-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org> 2017-01-26 21:54 ` Yishai Hadas [not found] ` <ac21fe37-d617-e5a8-cd67-e8fdfbd69d55-LDSdmyG8hGV8YrgS2mwiifqBs+8SCbDb@public.gmane.org> 2017-01-26 22:02 ` Jason Gunthorpe 2017-02-01 15:56 ` Christoph Lameter [not found] ` <alpine.DEB.2.20.1702010956080.12203-wcBtFHqTun5QOdAKl3ChDw@public.gmane.org> 2017-02-01 16:17 ` Leon Romanovsky 2017-02-02 8:52 ` Sagi Grimberg [not found] ` <782c31cb-e247-1094-955f-2766f9c66cbd-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org> 2017-02-02 13:15 ` Yishai Hadas 2017-02-06 10:56 ` Amrani, Ram [not found] ` <SN1PR07MB22070381E7566FA819E5F8F8F8400-mikhvbZlbf8TSoR2DauN2+FPX92sqiQdvxpqHgZTriW3zl9H0oFU5g@public.gmane.org> 2017-02-06 11:46 ` Leon Romanovsky [not found] ` <20170206114609.GK6005-U/DQcQFIOTAAJjI8aNfphQ@public.gmane.org> 2017-02-06 16:39 ` Jason Gunthorpe [not found] ` <20170206163937.GB14714-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org> 2017-02-08 18:45 ` Sagi Grimberg
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.