Linux-RDMA Archive on lore.kernel.org
 help / color / Atom feed
From: Jinpu Wang <jinpu.wang@cloud.ionos.com>
To: Bart Van Assche <bvanassche@acm.org>
Cc: Jack Wang <jinpuwang@gmail.com>,
	linux-block@vger.kernel.org, linux-rdma@vger.kernel.org,
	Jens Axboe <axboe@kernel.dk>,
	Christoph Hellwig <hch@infradead.org>,
	Sagi Grimberg <sagi@grimberg.me>,
	Leon Romanovsky <leon@kernel.org>,
	Doug Ledford <dledford@redhat.com>,
	Danil Kipnis <danil.kipnis@cloud.ionos.com>,
	rpenyaev@suse.de
Subject: Re: [PATCH v6 04/25] rtrs: core: lib functions shared between client and server modules
Date: Tue, 7 Jan 2020 13:22:43 +0100
Message-ID: <CAMGffEkG3N7ESyneN6PObq3L0Ps5cGSYboYac=Ti6yt-BqC2xA@mail.gmail.com> (raw)
In-Reply-To: <d4288d41-b000-9a98-d12a-0738a0f647e8@acm.org>

On Mon, Dec 30, 2019 at 11:25 PM Bart Van Assche <bvanassche@acm.org> wrote:
>
> On 2019-12-30 02:29, Jack Wang wrote:
> > + * InfiniBand Transport Layer
>
> Is RTRS an InfiniBand or an RDMA transport layer?
will fix.
>
> > +MODULE_DESCRIPTION("RTRS Core");
>
> Please write out RTRS in full and consider changing the word "Core" into
> "client and server".
will do.
>
> > +     WARN_ON(!queue_size);
> > +     ius = kcalloc(queue_size, sizeof(*ius), gfp_mask);
> > +
> > +     if (unlikely(!ius))
> > +             return NULL;
>
> No blank line between the 'ius' assignment and the 'ius' check please.
ok.
>
> > +int rtrs_iu_post_recv(struct rtrs_con *con, struct rtrs_iu *iu)
> > +{
> > +     struct rtrs_sess *sess = con->sess;
> > +     struct ib_recv_wr wr;
> > +     const struct ib_recv_wr *bad_wr;
> > +     struct ib_sge list;
> > +
> > +     list.addr   = iu->dma_addr;
> > +     list.length = iu->size;
> > +     list.lkey   = sess->dev->ib_pd->local_dma_lkey;
> > +
> > +     if (WARN_ON(list.length == 0)) {
> > +             rtrs_wrn(con->sess,
> > +                       "Posting receive work request failed, sg list is empty\n");
> > +             return -EINVAL;
> > +     }
> > +
> > +     wr.next    = NULL;
> > +     wr.wr_cqe  = &iu->cqe;
> > +     wr.sg_list = &list;
> > +     wr.num_sge = 1;
> > +
> > +     return ib_post_recv(con->qp, &wr, &bad_wr);
> > +}
> > +EXPORT_SYMBOL_GPL(rtrs_iu_post_recv);
>
> The above code is fragile: although this is unlikely, if a member would
> be added in struct ib_sge or in struct ib_recv_wr then the above code
> will leave some member variables uninitialized. Has it been considered
> to initialize these structures using a single assignment statement, e.g.
> as follows:
>
>         wr = (struct ib_recv_wr) {
>                 .wr_cqe = ...,
>                 .sg_list = ...,
>                 .num_sge = 1,
>         };
Will do.
>
> > +int rtrs_post_recv_empty(struct rtrs_con *con, struct ib_cqe *cqe)
> > +{
> > +     struct ib_recv_wr wr;
> > +     const struct ib_recv_wr *bad_wr;
> > +
> > +     wr.next    = NULL;
> > +     wr.wr_cqe  = cqe;
> > +     wr.sg_list = NULL;
> > +     wr.num_sge = 0;
> > +
> > +     return ib_post_recv(con->qp, &wr, &bad_wr);
> > +}
> > +EXPORT_SYMBOL_GPL(rtrs_post_recv_empty);
>
> Same comment for this function.
dito.
>
> > +int rtrs_post_recv_empty_x2(struct rtrs_con *con, struct ib_cqe *cqe)
> > +{
> > +     struct ib_recv_wr wr_arr[2], *wr;
> > +     const struct ib_recv_wr *bad_wr;
> > +     int i;
> > +
> > +     memset(wr_arr, 0, sizeof(wr_arr));
> > +     for (i = 0; i < ARRAY_SIZE(wr_arr); i++) {
> > +             wr = &wr_arr[i];
> > +             wr->wr_cqe  = cqe;
> > +             if (i)
> > +                     /* Chain backwards */
> > +                     wr->next = &wr_arr[i - 1];
> > +     }
> > +
> > +     return ib_post_recv(con->qp, wr, &bad_wr);
> > +}
> > +EXPORT_SYMBOL_GPL(rtrs_post_recv_empty_x2);
>
> I have not yet seen any other RDMA code that is similar to the above
> function. A comment above this function that explains its purpose would
> be more than welcome.
Will add comment.
>
> > +int rtrs_iu_post_send(struct rtrs_con *con, struct rtrs_iu *iu, size_t size,
> > +                    struct ib_send_wr *head)
> > +{
> > +     struct rtrs_sess *sess = con->sess;
> > +     struct ib_send_wr wr;
> > +     const struct ib_send_wr *bad_wr;
> > +     struct ib_sge list;
> > +
> > +     if ((WARN_ON(size == 0)))
> > +             return -EINVAL;
>
> No superfluous parentheses please.
ok

>
> > +     list.addr   = iu->dma_addr;
> > +     list.length = size;
> > +     list.lkey   = sess->dev->ib_pd->local_dma_lkey;
> > +
> > +     memset(&wr, 0, sizeof(wr));
> > +     wr.next       = NULL;
> > +     wr.wr_cqe     = &iu->cqe;
> > +     wr.sg_list    = &list;
> > +     wr.num_sge    = 1;
> > +     wr.opcode     = IB_WR_SEND;
> > +     wr.send_flags = IB_SEND_SIGNALED;
>
> Has it been considered to use designated initializers instead of a
> memset() followed by multiple assignments? Same question for
> rtrs_iu_post_rdma_write_imm() and rtrs_post_rdma_write_imm_empty().
Sounds good, will do.

>
> > +static int create_qp(struct rtrs_con *con, struct ib_pd *pd,
> > +                  u16 wr_queue_size, u32 max_sge)
> > +{
> > +     struct ib_qp_init_attr init_attr = {NULL};
> > +     struct rdma_cm_id *cm_id = con->cm_id;
> > +     int ret;
> > +
> > +     init_attr.cap.max_send_wr = wr_queue_size;
> > +     init_attr.cap.max_recv_wr = wr_queue_size;
>
> What code is responsible for ensuring that neither max_send_wr nor
> max_recv_wr exceeds the device limits? Please document this in a comment
> above this function.
rtrs-clt/srv queries device limits for ensuring the settings will not
exceed the limits.
will add comment.

>
> > +     init_attr.cap.max_recv_sge = 1;
> > +     init_attr.event_handler = qp_event_handler;
> > +     init_attr.qp_context = con;
> > +#undef max_send_sge
> > +     init_attr.cap.max_send_sge = max_sge;
>
> Is the "undef max_send_sge" really necessary? If so, please add a
> comment that explains why it is necessary.
it's not, will remove.
>
> > +static int rtrs_str_gid_to_sockaddr(const char *addr, size_t len,
> > +                                  short port, struct sockaddr_storage *dst)
> > +{
> > +     struct sockaddr_ib *dst_ib = (struct sockaddr_ib *)dst;
> > +     int ret;
> > +
> > +     /*
> > +      * We can use some of the I6 functions since GID is a valid
> > +      * IPv6 address format
> > +      */
> > +     ret = in6_pton(addr, len, dst_ib->sib_addr.sib_raw, '\0', NULL);
> > +     if (ret == 0)
> > +             return -EINVAL;
>
> What is "I6"?
IPv6, will fix.
>
> Is the fourth argument to this function correct? From the comment above
> in6_pton(): "@delim: the delimiter of the IPv6 address in @src, -1 means
> no delimiter".
'\0' means end of the string here, seems correct to me.
>
> > +int sockaddr_to_str(const struct sockaddr *addr, char *buf, size_t len)
> > +{
> > +     int cnt;
> > +
> > +     switch (addr->sa_family) {
> > +     case AF_IB:
> > +             cnt = scnprintf(buf, len, "gid:%pI6",
> > +                     &((struct sockaddr_ib *)addr)->sib_addr.sib_raw);
> > +             return cnt;
> > +     case AF_INET:
> > +             cnt = scnprintf(buf, len, "ip:%pI4",
> > +                     &((struct sockaddr_in *)addr)->sin_addr);
> > +             return cnt;
> > +     case AF_INET6:
> > +             cnt = scnprintf(buf, len, "ip:%pI6c",
> > +                       &((struct sockaddr_in6 *)addr)->sin6_addr);
> > +             return cnt;
> > +     }
> > +     cnt = scnprintf(buf, len, "<invalid address family>");
> > +     pr_err("Invalid address family\n");
> > +     return cnt;
> > +}
> > +EXPORT_SYMBOL(sockaddr_to_str);
>
> Is the pr_err() statement in the above function useful? Will anyone be
> able to figure out what is going on if the "Invalid address family"
> string appears in the system log? Please consider changing that pr_err()
> statement into a WARN_ON_ONCE() statement.
I expect the caller should also print something in syslog, combine
them togather will help.
>
> > +     ret = rtrs_str_to_sockaddr(str, len, port, addr->dst);
> > +
> > +     return ret;
>
> Please change this into a single return statement.
ok
>
> > +EXPORT_SYMBOL(rtrs_addr_to_sockaddr);
> > +
> > +void rtrs_ib_dev_pool_init(enum ib_pd_flags pd_flags,
> > +                         struct rtrs_ib_dev_pool *pool)
> > +{
> > +     WARN_ON(pool->ops && (!pool->ops->alloc ^ !pool->ops->free));
> > +     INIT_LIST_HEAD(&pool->list);
> > +     mutex_init(&pool->mutex);
> > +     pool->pd_flags = pd_flags;
> > +}
> > +EXPORT_SYMBOL(rtrs_ib_dev_pool_init);
> > +
> > +void rtrs_ib_dev_pool_deinit(struct rtrs_ib_dev_pool *pool)
> > +{
> > +     WARN_ON(!list_empty(&pool->list));
> > +}
> > +EXPORT_SYMBOL(rtrs_ib_dev_pool_deinit);
>
> Since rtrs_ib_dev_pool_init() calls mutex_init(), should
> rtrs_ib_dev_pool_deinit() call mutex_destroy()?
You're right.

>
> Thanks,
>
> Bart.
>
Thanks Bart.

  reply index

Thread overview: 89+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2019-12-30 10:29 [PATCH v6 00/25] RTRS (former IBTRS) rdma transport library and RNBD (former IBNBD) rdma network block device Jack Wang
2019-12-30 10:29 ` [PATCH v6 01/25] sysfs: export sysfs_remove_file_self() Jack Wang
2019-12-30 10:29 ` [PATCH v6 02/25] rtrs: public interface header to establish RDMA connections Jack Wang
2019-12-30 19:25   ` Bart Van Assche
2020-01-02 13:35     ` Jinpu Wang
2020-01-02 16:36       ` Bart Van Assche
2020-01-02 16:47         ` Jinpu Wang
2019-12-30 10:29 ` [PATCH v6 03/25] rtrs: private headers with rtrs protocol structs and helpers Jack Wang
2019-12-30 19:48   ` Bart Van Assche
2020-01-02 15:27     ` Jinpu Wang
2020-01-02 17:00       ` Bart Van Assche
2020-01-02 18:26         ` Jason Gunthorpe
2020-01-03 12:31           ` Jinpu Wang
2020-01-03 12:27         ` Jinpu Wang
2019-12-31  0:07   ` Bart Van Assche
2020-01-03 13:48     ` Jinpu Wang
2019-12-30 10:29 ` [PATCH v6 04/25] rtrs: core: lib functions shared between client and server modules Jack Wang
2019-12-30 22:25   ` Bart Van Assche
2020-01-07 12:22     ` Jinpu Wang [this message]
2019-12-30 10:29 ` [PATCH v6 05/25] rtrs: client: private header with client structs and functions Jack Wang
2019-12-30 22:51   ` Bart Van Assche
2020-01-07 12:39     ` Jinpu Wang
2019-12-30 23:03   ` Bart Van Assche
2020-01-07 12:39     ` Jinpu Wang
2019-12-30 10:29 ` [PATCH v6 06/25] rtrs: client: main functionality Jack Wang
2019-12-30 23:53   ` Bart Van Assche
2020-01-02 18:23     ` Jason Gunthorpe
2020-01-03 14:30     ` Jinpu Wang
2020-01-03 16:12       ` Bart Van Assche
2019-12-30 10:29 ` [PATCH v6 07/25] rtrs: client: statistics functions Jack Wang
2020-01-02 21:07   ` Bart Van Assche
2020-01-03 14:39     ` Jinpu Wang
2019-12-30 10:29 ` [PATCH v6 08/25] rtrs: client: sysfs interface functions Jack Wang
2020-01-02 21:14   ` Bart Van Assche
2020-01-03 14:59     ` Jinpu Wang
2019-12-30 10:29 ` [PATCH v6 09/25] rtrs: server: private header with server structs and functions Jack Wang
2020-01-02 21:24   ` Bart Van Assche
2020-01-08 16:33     ` Jinpu Wang
2019-12-30 10:29 ` [PATCH v6 10/25] rtrs: server: main functionality Jack Wang
2020-01-02 22:03   ` Bart Van Assche
2020-01-07 13:19     ` Jinpu Wang
2020-01-07 18:25       ` Jason Gunthorpe
2020-01-10 17:38         ` Jinpu Wang
2019-12-30 10:29 ` [PATCH v6 11/25] rtrs: server: statistics functions Jack Wang
2020-01-02 22:02   ` Bart Van Assche
2020-01-08 12:55     ` Jinpu Wang
2019-12-30 10:29 ` [PATCH v6 12/25] rtrs: server: sysfs interface functions Jack Wang
2020-01-02 22:06   ` Bart Van Assche
2020-01-07 14:40     ` Jinpu Wang
2019-12-30 10:29 ` [PATCH v6 13/25] rtrs: include client and server modules into kernel compilation Jack Wang
2020-01-02 22:11   ` Bart Van Assche
2020-01-03 16:19     ` Jinpu Wang
2019-12-30 10:29 ` [PATCH v6 14/25] rtrs: a bit of documentation Jack Wang
2019-12-30 23:19   ` Bart Van Assche
2020-01-07 14:48     ` Jinpu Wang
2020-01-02 22:21   ` Bart Van Assche
2020-01-07 15:49     ` Jinpu Wang
2019-12-30 10:29 ` [PATCH v6 15/25] rnbd: private headers with rnbd protocol structs and helpers Jack Wang
2020-01-02 22:34   ` Bart Van Assche
2020-01-07 16:53     ` Jinpu Wang
2019-12-30 10:29 ` [PATCH v6 16/25] rnbd: client: private header with client structs and functions Jack Wang
2020-01-02 22:37   ` Bart Van Assche
2020-01-07 17:09     ` Jinpu Wang
2019-12-30 10:29 ` [PATCH v6 17/25] rnbd: client: main functionality Jack Wang
2020-01-02 23:55   ` Bart Van Assche
2020-01-08 14:22     ` Jinpu Wang
2020-01-10 14:45     ` Jinpu Wang
2020-01-10 15:09       ` Roman Penyaev
2020-01-10 15:29         ` Jinpu Wang
2019-12-30 10:29 ` [PATCH v6 18/25] rnbd: client: sysfs interface functions Jack Wang
2020-01-03  0:03   ` Bart Van Assche
2020-01-08 13:06     ` Jinpu Wang
2020-01-08 16:39       ` Bart Van Assche
2020-01-08 16:51         ` Jinpu Wang
2019-12-30 10:29 ` [PATCH v6 19/25] rnbd: server: private header with server structs and functions Jack Wang
2019-12-30 10:29 ` [PATCH v6 20/25] rnbd: server: main functionality Jack Wang
2019-12-30 10:29 ` [PATCH v6 21/25] rnbd: server: functionality for IO submission to file or block dev Jack Wang
2019-12-30 10:29 ` [PATCH v6 22/25] rnbd: server: sysfs interface functions Jack Wang
2019-12-30 10:29 ` [PATCH v6 23/25] rnbd: include client and server modules into kernel compilation Jack Wang
2019-12-30 10:29 ` [PATCH v6 24/25] rnbd: a bit of documentation Jack Wang
2019-12-30 10:29 ` [PATCH v6 25/25] MAINTAINERS: Add maintainers for RNBD/RTRS modules Jack Wang
2019-12-30 12:22   ` Gal Pressman
2020-01-02  8:41     ` Jinpu Wang
2019-12-31  0:11 ` [PATCH v6 00/25] RTRS (former IBTRS) rdma transport library and RNBD (former IBNBD) rdma network block device Bart Van Assche
2020-01-02  8:48   ` Jinpu Wang
2019-12-31  2:39 ` Bart Van Assche
2020-01-02  9:20   ` Jinpu Wang
2020-01-02 18:28   ` Jason Gunthorpe
2020-01-03 12:34     ` Jinpu Wang

Reply instructions:

You may reply publically to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to='CAMGffEkG3N7ESyneN6PObq3L0Ps5cGSYboYac=Ti6yt-BqC2xA@mail.gmail.com' \
    --to=jinpu.wang@cloud.ionos.com \
    --cc=axboe@kernel.dk \
    --cc=bvanassche@acm.org \
    --cc=danil.kipnis@cloud.ionos.com \
    --cc=dledford@redhat.com \
    --cc=hch@infradead.org \
    --cc=jinpuwang@gmail.com \
    --cc=leon@kernel.org \
    --cc=linux-block@vger.kernel.org \
    --cc=linux-rdma@vger.kernel.org \
    --cc=rpenyaev@suse.de \
    --cc=sagi@grimberg.me \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

Linux-RDMA Archive on lore.kernel.org

Archives are clonable:
	git clone --mirror https://lore.kernel.org/linux-rdma/0 linux-rdma/git/0.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 linux-rdma linux-rdma/ https://lore.kernel.org/linux-rdma \
		linux-rdma@vger.kernel.org
	public-inbox-index linux-rdma

Example config snippet for mirrors

Newsgroup available over NNTP:
	nntp://nntp.lore.kernel.org/org.kernel.vger.linux-rdma


AGPL code for this site: git clone https://public-inbox.org/public-inbox.git