* [PATCH rdma-next 0/4] Introduce dynamic UAR allocation mode @ 2020-03-18 12:43 Leon Romanovsky 2020-03-18 12:43 ` [PATCH mlx5-next 4/4] IB/mlx5: Move to fully dynamic UAR mode once user space supports it Leon Romanovsky 2020-03-18 12:54 ` [PATCH rdma-next 0/4] Introduce dynamic UAR allocation mode Jason Gunthorpe 0 siblings, 2 replies; 14+ messages in thread From: Leon Romanovsky @ 2020-03-18 12:43 UTC (permalink / raw) To: Doug Ledford, Jason Gunthorpe Cc: Leon Romanovsky, linux-rdma, Michael Guralnik, netdev, Saeed Mahameed, Yishai Hadas From: Leon Romanovsky <leonro@mellanox.com> From Yishai, This series exposes API to enable a dynamic allocation and management of a UAR which now becomes to be a regular uobject. Moving to that mode enables allocating a UAR only upon demand and drop the redundant static allocation of UARs upon context creation. In addition, it allows master and secondary processes that own the same command FD to allocate and manage UARs according to their needs, this can’t be achieved today. As part of this option, QP & CQ creation flows were adapted to support this dynamic UAR mode once asked by user space. Once this mode is asked by mlx5 user space driver on a given context, it will be mutual exclusive, means both the static and legacy dynamic modes for using UARs will be blocked. The legacy modes are supported for backward compatible reasons, looking forward we expect this new mode to be the default. Thanks Yishai Hadas (4): IB/mlx5: Expose UAR object and its alloc/destroy commands IB/mlx5: Extend CQ creation to get uar page index from user space IB/mlx5: Extend QP creation to get uar page index from user space IB/mlx5: Move to fully dynamic UAR mode once user space supports it drivers/infiniband/hw/mlx5/cq.c | 21 ++- drivers/infiniband/hw/mlx5/main.c | 185 ++++++++++++++++++++-- drivers/infiniband/hw/mlx5/mlx5_ib.h | 2 + drivers/infiniband/hw/mlx5/qp.c | 33 ++-- include/linux/mlx5/driver.h | 1 + include/rdma/uverbs_ioctl.h | 2 +- include/uapi/rdma/mlx5-abi.h | 6 + include/uapi/rdma/mlx5_user_ioctl_cmds.h | 18 +++ include/uapi/rdma/mlx5_user_ioctl_verbs.h | 5 + 9 files changed, 246 insertions(+), 27 deletions(-) -- 2.24.1 ^ permalink raw reply [flat|nested] 14+ messages in thread
* [PATCH mlx5-next 4/4] IB/mlx5: Move to fully dynamic UAR mode once user space supports it 2020-03-18 12:43 [PATCH rdma-next 0/4] Introduce dynamic UAR allocation mode Leon Romanovsky @ 2020-03-18 12:43 ` Leon Romanovsky 2020-03-18 23:38 ` Saeed Mahameed 2020-03-18 12:54 ` [PATCH rdma-next 0/4] Introduce dynamic UAR allocation mode Jason Gunthorpe 1 sibling, 1 reply; 14+ messages in thread From: Leon Romanovsky @ 2020-03-18 12:43 UTC (permalink / raw) To: Doug Ledford, Jason Gunthorpe Cc: Yishai Hadas, linux-rdma, Michael Guralnik, netdev, Saeed Mahameed From: Yishai Hadas <yishaih@mellanox.com> Move to fully dynamic UAR mode once user space supports it. In this case we prevent any legacy mode of UARs on the allocated context and prevent redundant allocation of the static ones. Signed-off-by: Yishai Hadas <yishaih@mellanox.com> Reviewed-by: Michael Guralnik <michaelgur@mellanox.com> Signed-off-by: Leon Romanovsky <leonro@mellanox.com> --- drivers/infiniband/hw/mlx5/cq.c | 8 ++++++-- drivers/infiniband/hw/mlx5/main.c | 13 ++++++++++++- drivers/infiniband/hw/mlx5/qp.c | 6 ++++++ include/linux/mlx5/driver.h | 1 + include/uapi/rdma/mlx5-abi.h | 1 + 5 files changed, 26 insertions(+), 3 deletions(-) diff --git a/drivers/infiniband/hw/mlx5/cq.c b/drivers/infiniband/hw/mlx5/cq.c index eafedc2f697b..146ba2966744 100644 --- a/drivers/infiniband/hw/mlx5/cq.c +++ b/drivers/infiniband/hw/mlx5/cq.c @@ -764,10 +764,14 @@ static int create_cq_user(struct mlx5_ib_dev *dev, struct ib_udata *udata, MLX5_SET(cqc, cqc, log_page_size, page_shift - MLX5_ADAPTER_PAGE_SHIFT); - if (ucmd.flags & MLX5_IB_CREATE_CQ_FLAGS_UAR_PAGE_INDEX) + if (ucmd.flags & MLX5_IB_CREATE_CQ_FLAGS_UAR_PAGE_INDEX) { *index = ucmd.uar_page_index; - else + } else if (context->bfregi.lib_uar_dyn) { + err = -EINVAL; + goto err_cqb; + } else { *index = context->bfregi.sys_pages[0]; + } if (ucmd.cqe_comp_en == 1) { int mini_cqe_format; diff --git a/drivers/infiniband/hw/mlx5/main.c b/drivers/infiniband/hw/mlx5/main.c index e8787af2d74d..e355e06bf3ac 100644 --- a/drivers/infiniband/hw/mlx5/main.c +++ b/drivers/infiniband/hw/mlx5/main.c @@ -1787,6 +1787,7 @@ static int mlx5_ib_alloc_ucontext(struct ib_ucontext *uctx, max_cqe_version); u32 dump_fill_mkey; bool lib_uar_4k; + bool lib_uar_dyn; if (!dev->ib_active) return -EAGAIN; @@ -1845,8 +1846,14 @@ static int mlx5_ib_alloc_ucontext(struct ib_ucontext *uctx, } lib_uar_4k = req.lib_caps & MLX5_LIB_CAP_4K_UAR; + lib_uar_dyn = req.lib_caps & MLX5_LIB_CAP_DYN_UAR; bfregi = &context->bfregi; + if (lib_uar_dyn) { + bfregi->lib_uar_dyn = lib_uar_dyn; + goto uar_done; + } + /* updates req->total_num_bfregs */ err = calc_total_bfregs(dev, lib_uar_4k, &req, bfregi); if (err) @@ -1873,6 +1880,7 @@ static int mlx5_ib_alloc_ucontext(struct ib_ucontext *uctx, if (err) goto out_sys_pages; +uar_done: if (req.flags & MLX5_IB_ALLOC_UCTX_DEVX) { err = mlx5_ib_devx_create(dev, true); if (err < 0) @@ -1894,7 +1902,7 @@ static int mlx5_ib_alloc_ucontext(struct ib_ucontext *uctx, INIT_LIST_HEAD(&context->db_page_list); mutex_init(&context->db_page_mutex); - resp.tot_bfregs = req.total_num_bfregs; + resp.tot_bfregs = lib_uar_dyn ? 0 : req.total_num_bfregs; resp.num_ports = dev->num_ports; if (offsetofend(typeof(resp), cqe_version) <= udata->outlen) @@ -2142,6 +2150,9 @@ static int uar_mmap(struct mlx5_ib_dev *dev, enum mlx5_ib_mmap_cmd cmd, int max_valid_idx = dyn_uar ? bfregi->num_sys_pages : bfregi->num_static_sys_pages; + if (bfregi->lib_uar_dyn) + return -EINVAL; + if (vma->vm_end - vma->vm_start != PAGE_SIZE) return -EINVAL; diff --git a/drivers/infiniband/hw/mlx5/qp.c b/drivers/infiniband/hw/mlx5/qp.c index 380ba3321851..319d514a2223 100644 --- a/drivers/infiniband/hw/mlx5/qp.c +++ b/drivers/infiniband/hw/mlx5/qp.c @@ -697,6 +697,9 @@ static int alloc_bfreg(struct mlx5_ib_dev *dev, { int bfregn = -ENOMEM; + if (bfregi->lib_uar_dyn) + return -EINVAL; + mutex_lock(&bfregi->lock); if (bfregi->ver >= 2) { bfregn = alloc_high_class_bfreg(dev, bfregi); @@ -768,6 +771,9 @@ int bfregn_to_uar_index(struct mlx5_ib_dev *dev, u32 index_of_sys_page; u32 offset; + if (bfregi->lib_uar_dyn) + return -EINVAL; + bfregs_per_sys_page = get_uars_per_sys_page(dev, bfregi->lib_uar_4k) * MLX5_NON_FP_BFREGS_PER_UAR; index_of_sys_page = bfregn / bfregs_per_sys_page; diff --git a/include/linux/mlx5/driver.h b/include/linux/mlx5/driver.h index 3f10a9633012..e4ab0eb9d202 100644 --- a/include/linux/mlx5/driver.h +++ b/include/linux/mlx5/driver.h @@ -224,6 +224,7 @@ struct mlx5_bfreg_info { struct mutex lock; u32 ver; bool lib_uar_4k; + u8 lib_uar_dyn : 1; u32 num_sys_pages; u32 num_static_sys_pages; u32 total_num_bfregs; diff --git a/include/uapi/rdma/mlx5-abi.h b/include/uapi/rdma/mlx5-abi.h index a65d60b44829..df1cc3641bda 100644 --- a/include/uapi/rdma/mlx5-abi.h +++ b/include/uapi/rdma/mlx5-abi.h @@ -79,6 +79,7 @@ struct mlx5_ib_alloc_ucontext_req { enum mlx5_lib_caps { MLX5_LIB_CAP_4K_UAR = (__u64)1 << 0, + MLX5_LIB_CAP_DYN_UAR = (__u64)1 << 1, }; enum mlx5_ib_alloc_uctx_v2_flags { -- 2.24.1 ^ permalink raw reply related [flat|nested] 14+ messages in thread
* Re: [PATCH mlx5-next 4/4] IB/mlx5: Move to fully dynamic UAR mode once user space supports it 2020-03-18 12:43 ` [PATCH mlx5-next 4/4] IB/mlx5: Move to fully dynamic UAR mode once user space supports it Leon Romanovsky @ 2020-03-18 23:38 ` Saeed Mahameed 2020-03-19 5:55 ` Leon Romanovsky 0 siblings, 1 reply; 14+ messages in thread From: Saeed Mahameed @ 2020-03-18 23:38 UTC (permalink / raw) To: Jason Gunthorpe, leon, dledford Cc: Michael Guralnik, Yishai Hadas, netdev, linux-rdma On Wed, 2020-03-18 at 14:43 +0200, Leon Romanovsky wrote: > From: Yishai Hadas <yishaih@mellanox.com> > > Move to fully dynamic UAR mode once user space supports it. > In this case we prevent any legacy mode of UARs on the allocated > context > and prevent redundant allocation of the static ones. > > Signed-off-by: Yishai Hadas <yishaih@mellanox.com> > Reviewed-by: Michael Guralnik <michaelgur@mellanox.com> > Signed-off-by: Leon Romanovsky <leonro@mellanox.com> > --- > drivers/infiniband/hw/mlx5/cq.c | 8 ++++++-- > drivers/infiniband/hw/mlx5/main.c | 13 ++++++++++++- > drivers/infiniband/hw/mlx5/qp.c | 6 ++++++ > include/linux/mlx5/driver.h | 1 + > include/uapi/rdma/mlx5-abi.h | 1 + > 5 files changed, 26 insertions(+), 3 deletions(-) > > diff --git a/drivers/infiniband/hw/mlx5/cq.c > b/drivers/infiniband/hw/mlx5/cq.c > index eafedc2f697b..146ba2966744 100644 > --- a/drivers/infiniband/hw/mlx5/cq.c > +++ b/drivers/infiniband/hw/mlx5/cq.c > @@ -764,10 +764,14 @@ static int create_cq_user(struct mlx5_ib_dev > *dev, struct ib_udata *udata, > MLX5_SET(cqc, cqc, log_page_size, > page_shift - MLX5_ADAPTER_PAGE_SHIFT); > > - if (ucmd.flags & MLX5_IB_CREATE_CQ_FLAGS_UAR_PAGE_INDEX) > + if (ucmd.flags & MLX5_IB_CREATE_CQ_FLAGS_UAR_PAGE_INDEX) { > *index = ucmd.uar_page_index; > - else > + } else if (context->bfregi.lib_uar_dyn) { > + err = -EINVAL; > + goto err_cqb; > + } else { > *index = context->bfregi.sys_pages[0]; > + } > > if (ucmd.cqe_comp_en == 1) { > int mini_cqe_format; > diff --git a/drivers/infiniband/hw/mlx5/main.c > b/drivers/infiniband/hw/mlx5/main.c > index e8787af2d74d..e355e06bf3ac 100644 > --- a/drivers/infiniband/hw/mlx5/main.c > +++ b/drivers/infiniband/hw/mlx5/main.c > @@ -1787,6 +1787,7 @@ static int mlx5_ib_alloc_ucontext(struct > ib_ucontext *uctx, > max_cqe_version); > u32 dump_fill_mkey; > bool lib_uar_4k; > + bool lib_uar_dyn; > > if (!dev->ib_active) > return -EAGAIN; > @@ -1845,8 +1846,14 @@ static int mlx5_ib_alloc_ucontext(struct > ib_ucontext *uctx, > } > > lib_uar_4k = req.lib_caps & MLX5_LIB_CAP_4K_UAR; > + lib_uar_dyn = req.lib_caps & MLX5_LIB_CAP_DYN_UAR; > bfregi = &context->bfregi; > > + if (lib_uar_dyn) { > + bfregi->lib_uar_dyn = lib_uar_dyn; > + goto uar_done; > + } > + > /* updates req->total_num_bfregs */ > err = calc_total_bfregs(dev, lib_uar_4k, &req, bfregi); > if (err) > @@ -1873,6 +1880,7 @@ static int mlx5_ib_alloc_ucontext(struct > ib_ucontext *uctx, > if (err) > goto out_sys_pages; > > +uar_done: > if (req.flags & MLX5_IB_ALLOC_UCTX_DEVX) { > err = mlx5_ib_devx_create(dev, true); > if (err < 0) > @@ -1894,7 +1902,7 @@ static int mlx5_ib_alloc_ucontext(struct > ib_ucontext *uctx, > INIT_LIST_HEAD(&context->db_page_list); > mutex_init(&context->db_page_mutex); > > - resp.tot_bfregs = req.total_num_bfregs; > + resp.tot_bfregs = lib_uar_dyn ? 0 : req.total_num_bfregs; > resp.num_ports = dev->num_ports; > > if (offsetofend(typeof(resp), cqe_version) <= udata->outlen) > @@ -2142,6 +2150,9 @@ static int uar_mmap(struct mlx5_ib_dev *dev, > enum mlx5_ib_mmap_cmd cmd, > int max_valid_idx = dyn_uar ? bfregi->num_sys_pages : > bfregi->num_static_sys_pages; > > + if (bfregi->lib_uar_dyn) > + return -EINVAL; > + > if (vma->vm_end - vma->vm_start != PAGE_SIZE) > return -EINVAL; > > diff --git a/drivers/infiniband/hw/mlx5/qp.c > b/drivers/infiniband/hw/mlx5/qp.c > index 380ba3321851..319d514a2223 100644 > --- a/drivers/infiniband/hw/mlx5/qp.c > +++ b/drivers/infiniband/hw/mlx5/qp.c > @@ -697,6 +697,9 @@ static int alloc_bfreg(struct mlx5_ib_dev *dev, > { > int bfregn = -ENOMEM; > > + if (bfregi->lib_uar_dyn) > + return -EINVAL; > + > mutex_lock(&bfregi->lock); > if (bfregi->ver >= 2) { > bfregn = alloc_high_class_bfreg(dev, bfregi); > @@ -768,6 +771,9 @@ int bfregn_to_uar_index(struct mlx5_ib_dev *dev, > u32 index_of_sys_page; > u32 offset; > > + if (bfregi->lib_uar_dyn) > + return -EINVAL; > + > bfregs_per_sys_page = get_uars_per_sys_page(dev, bfregi- > >lib_uar_4k) * > MLX5_NON_FP_BFREGS_PER_UAR; > index_of_sys_page = bfregn / bfregs_per_sys_page; > diff --git a/include/linux/mlx5/driver.h > b/include/linux/mlx5/driver.h > index 3f10a9633012..e4ab0eb9d202 100644 > --- a/include/linux/mlx5/driver.h > +++ b/include/linux/mlx5/driver.h > @@ -224,6 +224,7 @@ struct mlx5_bfreg_info { > struct mutex lock; > u32 ver; > bool lib_uar_4k; > + u8 lib_uar_dyn : 1; > u32 num_sys_pages; > u32 num_static_sys_pages > u32 total_num_bfregs; this struct is not used in mlx5_core, shall we move it to mlx5_ib as part of this patch? so next time you need to update it you don't need to bother netdev busy people about it ;-) > diff --git a/include/uapi/rdma/mlx5-abi.h b/include/uapi/rdma/mlx5- > abi.h > index a65d60b44829..df1cc3641bda 100644 > --- a/include/uapi/rdma/mlx5-abi.h > +++ b/include/uapi/rdma/mlx5-abi.h > @@ -79,6 +79,7 @@ struct mlx5_ib_alloc_ucontext_req { > > enum mlx5_lib_caps { > MLX5_LIB_CAP_4K_UAR = (__u64)1 << 0, > + MLX5_LIB_CAP_DYN_UAR = (__u64)1 << 1, > }; > > enum mlx5_ib_alloc_uctx_v2_flags { ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [PATCH mlx5-next 4/4] IB/mlx5: Move to fully dynamic UAR mode once user space supports it 2020-03-18 23:38 ` Saeed Mahameed @ 2020-03-19 5:55 ` Leon Romanovsky 0 siblings, 0 replies; 14+ messages in thread From: Leon Romanovsky @ 2020-03-19 5:55 UTC (permalink / raw) To: Saeed Mahameed Cc: Jason Gunthorpe, dledford, Michael Guralnik, Yishai Hadas, netdev, linux-rdma On Wed, Mar 18, 2020 at 11:38:32PM +0000, Saeed Mahameed wrote: > On Wed, 2020-03-18 at 14:43 +0200, Leon Romanovsky wrote: > > From: Yishai Hadas <yishaih@mellanox.com> > > > > Move to fully dynamic UAR mode once user space supports it. > > In this case we prevent any legacy mode of UARs on the allocated > > context > > and prevent redundant allocation of the static ones. > > > > Signed-off-by: Yishai Hadas <yishaih@mellanox.com> > > Reviewed-by: Michael Guralnik <michaelgur@mellanox.com> > > Signed-off-by: Leon Romanovsky <leonro@mellanox.com> > > --- > > drivers/infiniband/hw/mlx5/cq.c | 8 ++++++-- > > drivers/infiniband/hw/mlx5/main.c | 13 ++++++++++++- > > drivers/infiniband/hw/mlx5/qp.c | 6 ++++++ > > include/linux/mlx5/driver.h | 1 + > > include/uapi/rdma/mlx5-abi.h | 1 + > > 5 files changed, 26 insertions(+), 3 deletions(-) > > > > diff --git a/drivers/infiniband/hw/mlx5/cq.c > > b/drivers/infiniband/hw/mlx5/cq.c > > index eafedc2f697b..146ba2966744 100644 > > --- a/drivers/infiniband/hw/mlx5/cq.c > > +++ b/drivers/infiniband/hw/mlx5/cq.c > > @@ -764,10 +764,14 @@ static int create_cq_user(struct mlx5_ib_dev > > *dev, struct ib_udata *udata, > > MLX5_SET(cqc, cqc, log_page_size, > > page_shift - MLX5_ADAPTER_PAGE_SHIFT); > > > > - if (ucmd.flags & MLX5_IB_CREATE_CQ_FLAGS_UAR_PAGE_INDEX) > > + if (ucmd.flags & MLX5_IB_CREATE_CQ_FLAGS_UAR_PAGE_INDEX) { > > *index = ucmd.uar_page_index; > > - else > > + } else if (context->bfregi.lib_uar_dyn) { > > + err = -EINVAL; > > + goto err_cqb; > > + } else { > > *index = context->bfregi.sys_pages[0]; > > + } > > > > if (ucmd.cqe_comp_en == 1) { > > int mini_cqe_format; > > diff --git a/drivers/infiniband/hw/mlx5/main.c > > b/drivers/infiniband/hw/mlx5/main.c > > index e8787af2d74d..e355e06bf3ac 100644 > > --- a/drivers/infiniband/hw/mlx5/main.c > > +++ b/drivers/infiniband/hw/mlx5/main.c > > @@ -1787,6 +1787,7 @@ static int mlx5_ib_alloc_ucontext(struct > > ib_ucontext *uctx, > > max_cqe_version); > > u32 dump_fill_mkey; > > bool lib_uar_4k; > > + bool lib_uar_dyn; > > > > if (!dev->ib_active) > > return -EAGAIN; > > @@ -1845,8 +1846,14 @@ static int mlx5_ib_alloc_ucontext(struct > > ib_ucontext *uctx, > > } > > > > lib_uar_4k = req.lib_caps & MLX5_LIB_CAP_4K_UAR; > > + lib_uar_dyn = req.lib_caps & MLX5_LIB_CAP_DYN_UAR; > > bfregi = &context->bfregi; > > > > + if (lib_uar_dyn) { > > + bfregi->lib_uar_dyn = lib_uar_dyn; > > + goto uar_done; > > + } > > + > > /* updates req->total_num_bfregs */ > > err = calc_total_bfregs(dev, lib_uar_4k, &req, bfregi); > > if (err) > > @@ -1873,6 +1880,7 @@ static int mlx5_ib_alloc_ucontext(struct > > ib_ucontext *uctx, > > if (err) > > goto out_sys_pages; > > > > +uar_done: > > if (req.flags & MLX5_IB_ALLOC_UCTX_DEVX) { > > err = mlx5_ib_devx_create(dev, true); > > if (err < 0) > > @@ -1894,7 +1902,7 @@ static int mlx5_ib_alloc_ucontext(struct > > ib_ucontext *uctx, > > INIT_LIST_HEAD(&context->db_page_list); > > mutex_init(&context->db_page_mutex); > > > > - resp.tot_bfregs = req.total_num_bfregs; > > + resp.tot_bfregs = lib_uar_dyn ? 0 : req.total_num_bfregs; > > resp.num_ports = dev->num_ports; > > > > if (offsetofend(typeof(resp), cqe_version) <= udata->outlen) > > @@ -2142,6 +2150,9 @@ static int uar_mmap(struct mlx5_ib_dev *dev, > > enum mlx5_ib_mmap_cmd cmd, > > int max_valid_idx = dyn_uar ? bfregi->num_sys_pages : > > bfregi->num_static_sys_pages; > > > > + if (bfregi->lib_uar_dyn) > > + return -EINVAL; > > + > > if (vma->vm_end - vma->vm_start != PAGE_SIZE) > > return -EINVAL; > > > > diff --git a/drivers/infiniband/hw/mlx5/qp.c > > b/drivers/infiniband/hw/mlx5/qp.c > > index 380ba3321851..319d514a2223 100644 > > --- a/drivers/infiniband/hw/mlx5/qp.c > > +++ b/drivers/infiniband/hw/mlx5/qp.c > > @@ -697,6 +697,9 @@ static int alloc_bfreg(struct mlx5_ib_dev *dev, > > { > > int bfregn = -ENOMEM; > > > > + if (bfregi->lib_uar_dyn) > > + return -EINVAL; > > + > > mutex_lock(&bfregi->lock); > > if (bfregi->ver >= 2) { > > bfregn = alloc_high_class_bfreg(dev, bfregi); > > @@ -768,6 +771,9 @@ int bfregn_to_uar_index(struct mlx5_ib_dev *dev, > > u32 index_of_sys_page; > > u32 offset; > > > > + if (bfregi->lib_uar_dyn) > > + return -EINVAL; > > + > > bfregs_per_sys_page = get_uars_per_sys_page(dev, bfregi- > > >lib_uar_4k) * > > MLX5_NON_FP_BFREGS_PER_UAR; > > index_of_sys_page = bfregn / bfregs_per_sys_page; > > diff --git a/include/linux/mlx5/driver.h > > b/include/linux/mlx5/driver.h > > index 3f10a9633012..e4ab0eb9d202 100644 > > --- a/include/linux/mlx5/driver.h > > +++ b/include/linux/mlx5/driver.h > > @@ -224,6 +224,7 @@ struct mlx5_bfreg_info { > > struct mutex lock; > > u32 ver; > > bool lib_uar_4k; > > + u8 lib_uar_dyn : 1; > > u32 num_sys_pages; > > u32 num_static_sys_pages > > u32 total_num_bfregs; > > this struct is not used in mlx5_core, shall we move it to mlx5_ib as > part of this patch? > so next time you need to update it you don't need to bother netdev busy > people about it ;-) Thanks, completely agree. I'll do it now. ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [PATCH rdma-next 0/4] Introduce dynamic UAR allocation mode 2020-03-18 12:43 [PATCH rdma-next 0/4] Introduce dynamic UAR allocation mode Leon Romanovsky 2020-03-18 12:43 ` [PATCH mlx5-next 4/4] IB/mlx5: Move to fully dynamic UAR mode once user space supports it Leon Romanovsky @ 2020-03-18 12:54 ` Jason Gunthorpe 2020-03-18 13:14 ` Leon Romanovsky 1 sibling, 1 reply; 14+ messages in thread From: Jason Gunthorpe @ 2020-03-18 12:54 UTC (permalink / raw) To: Leon Romanovsky Cc: Doug Ledford, Leon Romanovsky, linux-rdma, Michael Guralnik, netdev, Saeed Mahameed, Yishai Hadas On Wed, Mar 18, 2020 at 02:43:25PM +0200, Leon Romanovsky wrote: > From: Leon Romanovsky <leonro@mellanox.com> > > From Yishai, > > This series exposes API to enable a dynamic allocation and management of a > UAR which now becomes to be a regular uobject. > > Moving to that mode enables allocating a UAR only upon demand and drop the > redundant static allocation of UARs upon context creation. > > In addition, it allows master and secondary processes that own the same command > FD to allocate and manage UARs according to their needs, this can’t be achieved > today. > > As part of this option, QP & CQ creation flows were adapted to support this > dynamic UAR mode once asked by user space. > > Once this mode is asked by mlx5 user space driver on a given context, it will > be mutual exclusive, means both the static and legacy dynamic modes for using > UARs will be blocked. > > The legacy modes are supported for backward compatible reasons, looking > forward we expect this new mode to be the default. We are starting to accumulate a lot of code that is now old-rdma-core only. I have been wondering if we should add something like #if CONFIG_INFINIBAND_MIN_RDMA_CORE_VERSION < 21 #endif So we can keep track of what is actually a used code flow and what is now hard to test legacy code. eg this config would also disable the write interface(), turn off compat write interfaces as they are switched to use ioctl, etc, etc. Jason ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [PATCH rdma-next 0/4] Introduce dynamic UAR allocation mode 2020-03-18 12:54 ` [PATCH rdma-next 0/4] Introduce dynamic UAR allocation mode Jason Gunthorpe @ 2020-03-18 13:14 ` Leon Romanovsky 2020-03-18 13:21 ` Jason Gunthorpe 0 siblings, 1 reply; 14+ messages in thread From: Leon Romanovsky @ 2020-03-18 13:14 UTC (permalink / raw) To: Jason Gunthorpe Cc: Doug Ledford, linux-rdma, Michael Guralnik, netdev, Saeed Mahameed, Yishai Hadas On Wed, Mar 18, 2020 at 09:54:59AM -0300, Jason Gunthorpe wrote: > On Wed, Mar 18, 2020 at 02:43:25PM +0200, Leon Romanovsky wrote: > > From: Leon Romanovsky <leonro@mellanox.com> > > > > From Yishai, > > > > This series exposes API to enable a dynamic allocation and management of a > > UAR which now becomes to be a regular uobject. > > > > Moving to that mode enables allocating a UAR only upon demand and drop the > > redundant static allocation of UARs upon context creation. > > > > In addition, it allows master and secondary processes that own the same command > > FD to allocate and manage UARs according to their needs, this can’t be achieved > > today. > > > > As part of this option, QP & CQ creation flows were adapted to support this > > dynamic UAR mode once asked by user space. > > > > Once this mode is asked by mlx5 user space driver on a given context, it will > > be mutual exclusive, means both the static and legacy dynamic modes for using > > UARs will be blocked. > > > > The legacy modes are supported for backward compatible reasons, looking > > forward we expect this new mode to be the default. > > We are starting to accumulate a lot of code that is now old-rdma-core > only. Agree > > I have been wondering if we should add something like > > #if CONFIG_INFINIBAND_MIN_RDMA_CORE_VERSION < 21 > #endif From one side it will definitely help to see old code, but from another it will create many ifdef inside of the code with a very little chance of testing. Also we will continue to have the same problem to decide when we can delete this code. > > So we can keep track of what is actually a used code flow and what is > now hard to test legacy code. > > eg this config would also disable the write interface(), turn off > compat write interfaces as they are switched to use ioctl, etc, etc. What about if we introduce one ifdef, let's say CONFIG_INFINIBAND_LEGACY and put everything that will be declared as legacy to that bucket? And once every 5 (???) years delete everything from that bucket. > > Jason ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [PATCH rdma-next 0/4] Introduce dynamic UAR allocation mode 2020-03-18 13:14 ` Leon Romanovsky @ 2020-03-18 13:21 ` Jason Gunthorpe 2020-03-18 13:56 ` Leon Romanovsky 0 siblings, 1 reply; 14+ messages in thread From: Jason Gunthorpe @ 2020-03-18 13:21 UTC (permalink / raw) To: Leon Romanovsky Cc: Doug Ledford, linux-rdma, Michael Guralnik, netdev, Saeed Mahameed, Yishai Hadas On Wed, Mar 18, 2020 at 03:14:50PM +0200, Leon Romanovsky wrote: > On Wed, Mar 18, 2020 at 09:54:59AM -0300, Jason Gunthorpe wrote: > > On Wed, Mar 18, 2020 at 02:43:25PM +0200, Leon Romanovsky wrote: > > > From: Leon Romanovsky <leonro@mellanox.com> > > > > > > From Yishai, > > > > > > This series exposes API to enable a dynamic allocation and management of a > > > UAR which now becomes to be a regular uobject. > > > > > > Moving to that mode enables allocating a UAR only upon demand and drop the > > > redundant static allocation of UARs upon context creation. > > > > > > In addition, it allows master and secondary processes that own the same command > > > FD to allocate and manage UARs according to their needs, this can’t be achieved > > > today. > > > > > > As part of this option, QP & CQ creation flows were adapted to support this > > > dynamic UAR mode once asked by user space. > > > > > > Once this mode is asked by mlx5 user space driver on a given context, it will > > > be mutual exclusive, means both the static and legacy dynamic modes for using > > > UARs will be blocked. > > > > > > The legacy modes are supported for backward compatible reasons, looking > > > forward we expect this new mode to be the default. > > > > We are starting to accumulate a lot of code that is now old-rdma-core > > only. > > Agree > > > > > I have been wondering if we should add something like > > > > #if CONFIG_INFINIBAND_MIN_RDMA_CORE_VERSION < 21 > > #endif > > From one side it will definitely help to see old code, but from another > it will create many ifdef inside of the code with a very little chance > of testing. Also we will continue to have the same problem to decide when > we can delete this code. Well, it doesn't have to be an #ifdef, eg just sticking if (CONFIG_INFINIBAND_MIN_RDMA_CORE_VERSION >= 21) return -ENOPROTOOPT; at the top of obsolete functions would go a long way > > So we can keep track of what is actually a used code flow and what is > > now hard to test legacy code. > > > > eg this config would also disable the write interface(), turn off > > compat write interfaces as they are switched to use ioctl, etc, etc. > > What about if we introduce one ifdef, let's say CONFIG_INFINIBAND_LEGACY > and put everything that will be declared as legacy to that bucket? And > once every 5 (???) years delete everything from that bucket. It is much harder to see what is really old vs only a little old I'm not sure we can ever completely delete any of this, but at least the distros can make an informed choice to either do more detailed test of old libraries or disable those code paths. Jason ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [PATCH rdma-next 0/4] Introduce dynamic UAR allocation mode 2020-03-18 13:21 ` Jason Gunthorpe @ 2020-03-18 13:56 ` Leon Romanovsky 2020-03-18 14:00 ` Jason Gunthorpe 0 siblings, 1 reply; 14+ messages in thread From: Leon Romanovsky @ 2020-03-18 13:56 UTC (permalink / raw) To: Jason Gunthorpe Cc: Doug Ledford, linux-rdma, Michael Guralnik, netdev, Saeed Mahameed, Yishai Hadas On Wed, Mar 18, 2020 at 10:21:00AM -0300, Jason Gunthorpe wrote: > On Wed, Mar 18, 2020 at 03:14:50PM +0200, Leon Romanovsky wrote: > > On Wed, Mar 18, 2020 at 09:54:59AM -0300, Jason Gunthorpe wrote: > > > On Wed, Mar 18, 2020 at 02:43:25PM +0200, Leon Romanovsky wrote: > > > > From: Leon Romanovsky <leonro@mellanox.com> > > > > > > > > From Yishai, > > > > > > > > This series exposes API to enable a dynamic allocation and management of a > > > > UAR which now becomes to be a regular uobject. > > > > > > > > Moving to that mode enables allocating a UAR only upon demand and drop the > > > > redundant static allocation of UARs upon context creation. > > > > > > > > In addition, it allows master and secondary processes that own the same command > > > > FD to allocate and manage UARs according to their needs, this can’t be achieved > > > > today. > > > > > > > > As part of this option, QP & CQ creation flows were adapted to support this > > > > dynamic UAR mode once asked by user space. > > > > > > > > Once this mode is asked by mlx5 user space driver on a given context, it will > > > > be mutual exclusive, means both the static and legacy dynamic modes for using > > > > UARs will be blocked. > > > > > > > > The legacy modes are supported for backward compatible reasons, looking > > > > forward we expect this new mode to be the default. > > > > > > We are starting to accumulate a lot of code that is now old-rdma-core > > > only. > > > > Agree > > > > > > > > I have been wondering if we should add something like > > > > > > #if CONFIG_INFINIBAND_MIN_RDMA_CORE_VERSION < 21 > > > #endif > > > > From one side it will definitely help to see old code, but from another > > it will create many ifdef inside of the code with a very little chance > > of testing. Also we will continue to have the same problem to decide when > > we can delete this code. > > Well, it doesn't have to be an #ifdef, eg just sticking > > if (CONFIG_INFINIBAND_MIN_RDMA_CORE_VERSION >= 21) > return -ENOPROTOOPT; > > at the top of obsolete functions would go a long way First, how will you set this min_version? hordcoded in the kernel code? Second, it will work for simple flows, but can be extremely complex if your code looks like: if (old_version) do something if (new version) do something else You will need to add logic to handle this -ENOPROTOOPT error value. > > > > So we can keep track of what is actually a used code flow and what is > > > now hard to test legacy code. > > > > > > eg this config would also disable the write interface(), turn off > > > compat write interfaces as they are switched to use ioctl, etc, etc. > > > > What about if we introduce one ifdef, let's say CONFIG_INFINIBAND_LEGACY > > and put everything that will be declared as legacy to that bucket? And > > once every 5 (???) years delete everything from that bucket. > > It is much harder to see what is really old vs only a little old > > I'm not sure we can ever completely delete any of this, but at least > the distros can make an informed choice to either do more detailed > test of old libraries or disable those code paths. It will be nice to hear how distros decide to disable/drop the code. Thanks > > Jason ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [PATCH rdma-next 0/4] Introduce dynamic UAR allocation mode 2020-03-18 13:56 ` Leon Romanovsky @ 2020-03-18 14:00 ` Jason Gunthorpe 2020-03-18 14:09 ` Leon Romanovsky 0 siblings, 1 reply; 14+ messages in thread From: Jason Gunthorpe @ 2020-03-18 14:00 UTC (permalink / raw) To: Leon Romanovsky, Doug Ledford Cc: linux-rdma, Michael Guralnik, netdev, Saeed Mahameed, Yishai Hadas On Wed, Mar 18, 2020 at 03:56:31PM +0200, Leon Romanovsky wrote: > On Wed, Mar 18, 2020 at 10:21:00AM -0300, Jason Gunthorpe wrote: > > On Wed, Mar 18, 2020 at 03:14:50PM +0200, Leon Romanovsky wrote: > > > On Wed, Mar 18, 2020 at 09:54:59AM -0300, Jason Gunthorpe wrote: > > > > On Wed, Mar 18, 2020 at 02:43:25PM +0200, Leon Romanovsky wrote: > > > > > From: Leon Romanovsky <leonro@mellanox.com> > > > > > > > > > > From Yishai, > > > > > > > > > > This series exposes API to enable a dynamic allocation and management of a > > > > > UAR which now becomes to be a regular uobject. > > > > > > > > > > Moving to that mode enables allocating a UAR only upon demand and drop the > > > > > redundant static allocation of UARs upon context creation. > > > > > > > > > > In addition, it allows master and secondary processes that own the same command > > > > > FD to allocate and manage UARs according to their needs, this can’t be achieved > > > > > today. > > > > > > > > > > As part of this option, QP & CQ creation flows were adapted to support this > > > > > dynamic UAR mode once asked by user space. > > > > > > > > > > Once this mode is asked by mlx5 user space driver on a given context, it will > > > > > be mutual exclusive, means both the static and legacy dynamic modes for using > > > > > UARs will be blocked. > > > > > > > > > > The legacy modes are supported for backward compatible reasons, looking > > > > > forward we expect this new mode to be the default. > > > > > > > > We are starting to accumulate a lot of code that is now old-rdma-core > > > > only. > > > > > > Agree > > > > > > > > > > > I have been wondering if we should add something like > > > > > > > > #if CONFIG_INFINIBAND_MIN_RDMA_CORE_VERSION < 21 > > > > #endif > > > > > > From one side it will definitely help to see old code, but from another > > > it will create many ifdef inside of the code with a very little chance > > > of testing. Also we will continue to have the same problem to decide when > > > we can delete this code. > > > > Well, it doesn't have to be an #ifdef, eg just sticking > > > > if (CONFIG_INFINIBAND_MIN_RDMA_CORE_VERSION >= 21) > > return -ENOPROTOOPT; > > > > at the top of obsolete functions would go a long way > > First, how will you set this min_version? hordcoded in the kernel > code? Yes, when a rdma-core release obsoletes the code path then it can become annotated. > Second, it will work for simple flows, but can be extremely complex > if your code looks like: > if (old_version) > do something > if (new version) > do something else Well, we'd avoid making such complications, it would be something like if (flag & foo) { if (CONFIG_INFINIBAND_MIN_RDMA_CORE_VERSION >= 21) return -ENOPROTOOPT; [keep going as before] } At least we now know this conditional path isn't used / isn't covered by testing Doug? What does a distro think? Jason ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [PATCH rdma-next 0/4] Introduce dynamic UAR allocation mode 2020-03-18 14:00 ` Jason Gunthorpe @ 2020-03-18 14:09 ` Leon Romanovsky 2020-03-18 14:12 ` Jason Gunthorpe 0 siblings, 1 reply; 14+ messages in thread From: Leon Romanovsky @ 2020-03-18 14:09 UTC (permalink / raw) To: Jason Gunthorpe Cc: Doug Ledford, linux-rdma, Michael Guralnik, netdev, Saeed Mahameed, Yishai Hadas On Wed, Mar 18, 2020 at 11:00:01AM -0300, Jason Gunthorpe wrote: > On Wed, Mar 18, 2020 at 03:56:31PM +0200, Leon Romanovsky wrote: > > On Wed, Mar 18, 2020 at 10:21:00AM -0300, Jason Gunthorpe wrote: > > > On Wed, Mar 18, 2020 at 03:14:50PM +0200, Leon Romanovsky wrote: > > > > On Wed, Mar 18, 2020 at 09:54:59AM -0300, Jason Gunthorpe wrote: > > > > > On Wed, Mar 18, 2020 at 02:43:25PM +0200, Leon Romanovsky wrote: > > > > > > From: Leon Romanovsky <leonro@mellanox.com> > > > > > > > > > > > > From Yishai, > > > > > > > > > > > > This series exposes API to enable a dynamic allocation and management of a > > > > > > UAR which now becomes to be a regular uobject. > > > > > > > > > > > > Moving to that mode enables allocating a UAR only upon demand and drop the > > > > > > redundant static allocation of UARs upon context creation. > > > > > > > > > > > > In addition, it allows master and secondary processes that own the same command > > > > > > FD to allocate and manage UARs according to their needs, this can’t be achieved > > > > > > today. > > > > > > > > > > > > As part of this option, QP & CQ creation flows were adapted to support this > > > > > > dynamic UAR mode once asked by user space. > > > > > > > > > > > > Once this mode is asked by mlx5 user space driver on a given context, it will > > > > > > be mutual exclusive, means both the static and legacy dynamic modes for using > > > > > > UARs will be blocked. > > > > > > > > > > > > The legacy modes are supported for backward compatible reasons, looking > > > > > > forward we expect this new mode to be the default. > > > > > > > > > > We are starting to accumulate a lot of code that is now old-rdma-core > > > > > only. > > > > > > > > Agree > > > > > > > > > > > > > > I have been wondering if we should add something like > > > > > > > > > > #if CONFIG_INFINIBAND_MIN_RDMA_CORE_VERSION < 21 > > > > > #endif > > > > > > > > From one side it will definitely help to see old code, but from another > > > > it will create many ifdef inside of the code with a very little chance > > > > of testing. Also we will continue to have the same problem to decide when > > > > we can delete this code. > > > > > > Well, it doesn't have to be an #ifdef, eg just sticking > > > > > > if (CONFIG_INFINIBAND_MIN_RDMA_CORE_VERSION >= 21) > > > return -ENOPROTOOPT; > > > > > > at the top of obsolete functions would go a long way > > > > First, how will you set this min_version? hordcoded in the kernel > > code? > > Yes, when a rdma-core release obsoletes the code path then it can > become annotated. > > > Second, it will work for simple flows, but can be extremely complex > > if your code looks like: > > if (old_version) > > do something > > if (new version) > > do something else > > Well, we'd avoid making such complications, it would be something like > > if (flag & foo) { > if (CONFIG_INFINIBAND_MIN_RDMA_CORE_VERSION >= 21) > return -ENOPROTOOPT; > [keep going as before] > } > > At least we now know this conditional path isn't used / isn't covered > by testing I'm ok with this approach because it helps us to find those dead paths, but have last question, shouldn't this be achieved with proper documentation of every flag instead of adding CONFIG_..? > > Doug? What does a distro think? > > Jason ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [PATCH rdma-next 0/4] Introduce dynamic UAR allocation mode 2020-03-18 14:09 ` Leon Romanovsky @ 2020-03-18 14:12 ` Jason Gunthorpe 2020-03-18 14:24 ` Leon Romanovsky 0 siblings, 1 reply; 14+ messages in thread From: Jason Gunthorpe @ 2020-03-18 14:12 UTC (permalink / raw) To: Leon Romanovsky Cc: Doug Ledford, linux-rdma, Michael Guralnik, netdev, Saeed Mahameed, Yishai Hadas On Wed, Mar 18, 2020 at 04:09:32PM +0200, Leon Romanovsky wrote: > On Wed, Mar 18, 2020 at 11:00:01AM -0300, Jason Gunthorpe wrote: > > On Wed, Mar 18, 2020 at 03:56:31PM +0200, Leon Romanovsky wrote: > > > On Wed, Mar 18, 2020 at 10:21:00AM -0300, Jason Gunthorpe wrote: > > > > On Wed, Mar 18, 2020 at 03:14:50PM +0200, Leon Romanovsky wrote: > > > > > On Wed, Mar 18, 2020 at 09:54:59AM -0300, Jason Gunthorpe wrote: > > > > > > On Wed, Mar 18, 2020 at 02:43:25PM +0200, Leon Romanovsky wrote: > > > > > > > From: Leon Romanovsky <leonro@mellanox.com> > > > > > > > > > > > > > > From Yishai, > > > > > > > > > > > > > > This series exposes API to enable a dynamic allocation and management of a > > > > > > > UAR which now becomes to be a regular uobject. > > > > > > > > > > > > > > Moving to that mode enables allocating a UAR only upon demand and drop the > > > > > > > redundant static allocation of UARs upon context creation. > > > > > > > > > > > > > > In addition, it allows master and secondary processes that own the same command > > > > > > > FD to allocate and manage UARs according to their needs, this can’t be achieved > > > > > > > today. > > > > > > > > > > > > > > As part of this option, QP & CQ creation flows were adapted to support this > > > > > > > dynamic UAR mode once asked by user space. > > > > > > > > > > > > > > Once this mode is asked by mlx5 user space driver on a given context, it will > > > > > > > be mutual exclusive, means both the static and legacy dynamic modes for using > > > > > > > UARs will be blocked. > > > > > > > > > > > > > > The legacy modes are supported for backward compatible reasons, looking > > > > > > > forward we expect this new mode to be the default. > > > > > > > > > > > > We are starting to accumulate a lot of code that is now old-rdma-core > > > > > > only. > > > > > > > > > > Agree > > > > > > > > > > > > > > > > > I have been wondering if we should add something like > > > > > > > > > > > > #if CONFIG_INFINIBAND_MIN_RDMA_CORE_VERSION < 21 > > > > > > #endif > > > > > > > > > > From one side it will definitely help to see old code, but from another > > > > > it will create many ifdef inside of the code with a very little chance > > > > > of testing. Also we will continue to have the same problem to decide when > > > > > we can delete this code. > > > > > > > > Well, it doesn't have to be an #ifdef, eg just sticking > > > > > > > > if (CONFIG_INFINIBAND_MIN_RDMA_CORE_VERSION >= 21) > > > > return -ENOPROTOOPT; > > > > > > > > at the top of obsolete functions would go a long way > > > > > > First, how will you set this min_version? hordcoded in the kernel > > > code? > > > > Yes, when a rdma-core release obsoletes the code path then it can > > become annotated. > > > > > Second, it will work for simple flows, but can be extremely complex > > > if your code looks like: > > > if (old_version) > > > do something > > > if (new version) > > > do something else > > > > Well, we'd avoid making such complications, it would be something like > > > > if (flag & foo) { > > if (CONFIG_INFINIBAND_MIN_RDMA_CORE_VERSION >= 21) > > return -ENOPROTOOPT; > > [keep going as before] > > } > > > > At least we now know this conditional path isn't used / isn't covered > > by testing > > I'm ok with this approach because it helps us to find those dead > paths, but have last question, shouldn't this be achieved with > proper documentation of every flag instead of adding CONFIG_..? How do you mean? The other half of this idea is to disable obsolete un tested code to avoid potential bugs. Which requires CONFIG_? Jason ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [PATCH rdma-next 0/4] Introduce dynamic UAR allocation mode 2020-03-18 14:12 ` Jason Gunthorpe @ 2020-03-18 14:24 ` Leon Romanovsky 2020-03-18 14:39 ` Jason Gunthorpe 0 siblings, 1 reply; 14+ messages in thread From: Leon Romanovsky @ 2020-03-18 14:24 UTC (permalink / raw) To: Jason Gunthorpe Cc: Doug Ledford, linux-rdma, Michael Guralnik, netdev, Saeed Mahameed, Yishai Hadas On Wed, Mar 18, 2020 at 11:12:08AM -0300, Jason Gunthorpe wrote: > On Wed, Mar 18, 2020 at 04:09:32PM +0200, Leon Romanovsky wrote: > > On Wed, Mar 18, 2020 at 11:00:01AM -0300, Jason Gunthorpe wrote: > > > On Wed, Mar 18, 2020 at 03:56:31PM +0200, Leon Romanovsky wrote: > > > > On Wed, Mar 18, 2020 at 10:21:00AM -0300, Jason Gunthorpe wrote: > > > > > On Wed, Mar 18, 2020 at 03:14:50PM +0200, Leon Romanovsky wrote: > > > > > > On Wed, Mar 18, 2020 at 09:54:59AM -0300, Jason Gunthorpe wrote: > > > > > > > On Wed, Mar 18, 2020 at 02:43:25PM +0200, Leon Romanovsky wrote: > > > > > > > > From: Leon Romanovsky <leonro@mellanox.com> > > > > > > > > > > > > > > > > From Yishai, > > > > > > > > > > > > > > > > This series exposes API to enable a dynamic allocation and management of a > > > > > > > > UAR which now becomes to be a regular uobject. > > > > > > > > > > > > > > > > Moving to that mode enables allocating a UAR only upon demand and drop the > > > > > > > > redundant static allocation of UARs upon context creation. > > > > > > > > > > > > > > > > In addition, it allows master and secondary processes that own the same command > > > > > > > > FD to allocate and manage UARs according to their needs, this can’t be achieved > > > > > > > > today. > > > > > > > > > > > > > > > > As part of this option, QP & CQ creation flows were adapted to support this > > > > > > > > dynamic UAR mode once asked by user space. > > > > > > > > > > > > > > > > Once this mode is asked by mlx5 user space driver on a given context, it will > > > > > > > > be mutual exclusive, means both the static and legacy dynamic modes for using > > > > > > > > UARs will be blocked. > > > > > > > > > > > > > > > > The legacy modes are supported for backward compatible reasons, looking > > > > > > > > forward we expect this new mode to be the default. > > > > > > > > > > > > > > We are starting to accumulate a lot of code that is now old-rdma-core > > > > > > > only. > > > > > > > > > > > > Agree > > > > > > > > > > > > > > > > > > > > I have been wondering if we should add something like > > > > > > > > > > > > > > #if CONFIG_INFINIBAND_MIN_RDMA_CORE_VERSION < 21 > > > > > > > #endif > > > > > > > > > > > > From one side it will definitely help to see old code, but from another > > > > > > it will create many ifdef inside of the code with a very little chance > > > > > > of testing. Also we will continue to have the same problem to decide when > > > > > > we can delete this code. > > > > > > > > > > Well, it doesn't have to be an #ifdef, eg just sticking > > > > > > > > > > if (CONFIG_INFINIBAND_MIN_RDMA_CORE_VERSION >= 21) > > > > > return -ENOPROTOOPT; > > > > > > > > > > at the top of obsolete functions would go a long way > > > > > > > > First, how will you set this min_version? hordcoded in the kernel > > > > code? > > > > > > Yes, when a rdma-core release obsoletes the code path then it can > > > become annotated. > > > > > > > Second, it will work for simple flows, but can be extremely complex > > > > if your code looks like: > > > > if (old_version) > > > > do something > > > > if (new version) > > > > do something else > > > > > > Well, we'd avoid making such complications, it would be something like > > > > > > if (flag & foo) { > > > if (CONFIG_INFINIBAND_MIN_RDMA_CORE_VERSION >= 21) > > > return -ENOPROTOOPT; > > > [keep going as before] > > > } > > > > > > At least we now know this conditional path isn't used / isn't covered > > > by testing > > > > I'm ok with this approach because it helps us to find those dead > > paths, but have last question, shouldn't this be achieved with > > proper documentation of every flag instead of adding CONFIG_..? > > How do you mean? > > The other half of this idea is to disable obsolete un tested code to > avoid potential bugs. Which requires CONFIG_? The second part is achievable by distros when they will decide to support starting from version X. The same decision is not so easy to do in the upstream. Let's take as an example this feature. It will be set as default from rdma-core v29 and the legacy code will be guarded by "if (CONFIG_INFINIBAND_MIN_RDMA_CORE_VERSION >= 29)". When will change CONFIG_INFINIBAND_MIN_RDMA_CORE_VERSION to be above 29? So we will delete such legacy code. In the world where we are not breaking user space, it will never happen. It means that upstream doesn't get anything from those CONFIG_*s. Thanks > > Jason ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [PATCH rdma-next 0/4] Introduce dynamic UAR allocation mode 2020-03-18 14:24 ` Leon Romanovsky @ 2020-03-18 14:39 ` Jason Gunthorpe 2020-03-18 17:07 ` Leon Romanovsky 0 siblings, 1 reply; 14+ messages in thread From: Jason Gunthorpe @ 2020-03-18 14:39 UTC (permalink / raw) To: Leon Romanovsky Cc: Doug Ledford, linux-rdma, Michael Guralnik, netdev, Saeed Mahameed, Yishai Hadas On Wed, Mar 18, 2020 at 04:24:55PM +0200, Leon Romanovsky wrote: > > > I'm ok with this approach because it helps us to find those dead > > > paths, but have last question, shouldn't this be achieved with > > > proper documentation of every flag instead of adding CONFIG_..? > > > > How do you mean? > > > > The other half of this idea is to disable obsolete un tested code to > > avoid potential bugs. Which requires CONFIG_? > > The second part is achievable by distros when they will decide to > support starting from version X. The same decision is not so easy > to do in the upstream. Upstream will probably carry the code for a long, long time, that doesn't mean the distros don't get value by using a shorter time window > Let's take as an example this feature. It will be set as default from > rdma-core v29 and the legacy code will be guarded by > "if (CONFIG_INFINIBAND_MIN_RDMA_CORE_VERSION >= 29)". When will change > CONFIG_INFINIBAND_MIN_RDMA_CORE_VERSION to be above 29? So we will > delete such legacy code. First the distros will decide in their own kconfigs where they want to set the value. Then the upstream kernel will decide some default value Then maybe we could talk about lowest values when enough of the user community uses a higher value Jason ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [PATCH rdma-next 0/4] Introduce dynamic UAR allocation mode 2020-03-18 14:39 ` Jason Gunthorpe @ 2020-03-18 17:07 ` Leon Romanovsky 0 siblings, 0 replies; 14+ messages in thread From: Leon Romanovsky @ 2020-03-18 17:07 UTC (permalink / raw) To: Jason Gunthorpe Cc: Doug Ledford, linux-rdma, Michael Guralnik, netdev, Saeed Mahameed, Yishai Hadas On Wed, Mar 18, 2020 at 11:39:03AM -0300, Jason Gunthorpe wrote: > On Wed, Mar 18, 2020 at 04:24:55PM +0200, Leon Romanovsky wrote: > > > > I'm ok with this approach because it helps us to find those dead > > > > paths, but have last question, shouldn't this be achieved with > > > > proper documentation of every flag instead of adding CONFIG_..? > > > > > > How do you mean? > > > > > > The other half of this idea is to disable obsolete un tested code to > > > avoid potential bugs. Which requires CONFIG_? > > > > The second part is achievable by distros when they will decide to > > support starting from version X. The same decision is not so easy > > to do in the upstream. > > Upstream will probably carry the code for a long, long time, that > doesn't mean the distros don't get value by using a shorter time > window Sure > > > Let's take as an example this feature. It will be set as default from > > rdma-core v29 and the legacy code will be guarded by > > "if (CONFIG_INFINIBAND_MIN_RDMA_CORE_VERSION >= 29)". When will change > > CONFIG_INFINIBAND_MIN_RDMA_CORE_VERSION to be above 29? So we will > > delete such legacy code. > > First the distros will decide in their own kconfigs where they want to > set the value. > > Then the upstream kernel will decide some default value > > Then maybe we could talk about lowest values when enough of the user > community uses a higher value I think that you over-optimistic here, but let's hear other voices here. Thanks > > Jason ^ permalink raw reply [flat|nested] 14+ messages in thread
end of thread, other threads:[~2020-03-19 5:55 UTC | newest] Thread overview: 14+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2020-03-18 12:43 [PATCH rdma-next 0/4] Introduce dynamic UAR allocation mode Leon Romanovsky 2020-03-18 12:43 ` [PATCH mlx5-next 4/4] IB/mlx5: Move to fully dynamic UAR mode once user space supports it Leon Romanovsky 2020-03-18 23:38 ` Saeed Mahameed 2020-03-19 5:55 ` Leon Romanovsky 2020-03-18 12:54 ` [PATCH rdma-next 0/4] Introduce dynamic UAR allocation mode Jason Gunthorpe 2020-03-18 13:14 ` Leon Romanovsky 2020-03-18 13:21 ` Jason Gunthorpe 2020-03-18 13:56 ` Leon Romanovsky 2020-03-18 14:00 ` Jason Gunthorpe 2020-03-18 14:09 ` Leon Romanovsky 2020-03-18 14:12 ` Jason Gunthorpe 2020-03-18 14:24 ` Leon Romanovsky 2020-03-18 14:39 ` Jason Gunthorpe 2020-03-18 17:07 ` Leon Romanovsky
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).