Linux-Block Archive on lore.kernel.org
 help / color / Atom feed
From: Jinpu Wang <jinpu.wang@cloud.ionos.com>
To: Leon Romanovsky <leon@kernel.org>
Cc: Jack Wang <jinpuwang@gmail.com>,
	linux-block@vger.kernel.org, linux-rdma@vger.kernel.org,
	Jens Axboe <axboe@kernel.dk>,
	Christoph Hellwig <hch@infradead.org>,
	Sagi Grimberg <sagi@grimberg.me>,
	Bart Van Assche <bvanassche@acm.org>,
	Doug Ledford <dledford@redhat.com>,
	Jason Gunthorpe <jgg@ziepe.ca>,
	Danil Kipnis <danil.kipnis@cloud.ionos.com>,
	Roman Penyaev <rpenyaev@suse.de>
Subject: Re: [PATCH v7 17/25] block/rnbd: client: main functionality
Date: Wed, 22 Jan 2020 14:12:19 +0100
Message-ID: <CAMGffEmpRvruSn6iz6EfgfAjE9xrnsihwPaQU8Ft9e7qLD5avw@mail.gmail.com> (raw)
In-Reply-To: <20200122122548.GB7018@unreal>

On Wed, Jan 22, 2020 at 1:25 PM Leon Romanovsky <leon@kernel.org> wrote:
>
> On Wed, Jan 22, 2020 at 12:22:43PM +0100, Jinpu Wang wrote:
> > > > +/**
> > > > + * rnbd_get_cpu_qlist() - finds a list with HW queues to be rerun
> > > > + * @sess:    Session to find a queue for
> > > > + * @cpu:     Cpu to start the search from
> > > > + *
> > > > + * Description:
> > > > + *     Each CPU has a list of HW queues, which needs to be rerun.  If a list
> > > > + *     is not empty - it is marked with a bit.  This function finds first
> > > > + *     set bit in a bitmap and returns corresponding CPU list.
> > > > + */
> > > > +static struct rnbd_cpu_qlist *
> > > > +rnbd_get_cpu_qlist(struct rnbd_clt_session *sess, int cpu)
> > > > +{
> > > > +     int bit;
> > > > +
> > > > +     /* First half */
> > > > +     bit = find_next_bit(sess->cpu_queues_bm, nr_cpu_ids, cpu);
> > >
> > > Is it protected by any lock?
> > We hold requeue_lock when set/clear bit, and disable preemption via
> > get_cpu_ptr when find_next_bit.
> > even it fails to get latest bit, it just cause an rerun the queue.
>
> It is not clear here at all.
>
> > >
> > > > +     if (bit < nr_cpu_ids) {
> > > > +             return per_cpu_ptr(sess->cpu_queues, bit);
> > > > +     } else if (cpu != 0) {
> > > > +             /* Second half */
> > > > +             bit = find_next_bit(sess->cpu_queues_bm, cpu, 0);
> > > > +             if (bit < cpu)
> > > > +                     return per_cpu_ptr(sess->cpu_queues, bit);
> > > > +     }
> > > > +
> > > > +     return NULL;
> > > > +}
> > > > +
> > > > +static inline int nxt_cpu(int cpu)
> > > > +{
> > > > +     return (cpu + 1) % nr_cpu_ids;
> > > > +}
> > > > +
> > > > +/**
> > > > + * rnbd_rerun_if_needed() - rerun next queue marked as stopped
> > > > + * @sess:    Session to rerun a queue on
> > > > + *
> > > > + * Description:
> > > > + *     Each CPU has it's own list of HW queues, which should be rerun.
> > > > + *     Function finds such list with HW queues, takes a list lock, picks up
> > > > + *     the first HW queue out of the list and requeues it.
> > > > + *
> > > > + * Return:
> > > > + *     True if the queue was requeued, false otherwise.
> > > > + *
> > > > + * Context:
> > > > + *     Does not matter.
> > > > + */
> > > > +static inline bool rnbd_rerun_if_needed(struct rnbd_clt_session *sess)
> > >
> > > No inline function in C files.
> > First time saw such request, there are so many inline functions in C
>
> 15) The inline disease
> https://elixir.bootlin.com/linux/latest/source/Documentation/process/coding-style.rst#L882
ok, will review the inline usage, and drop some.
>
> > files across the tree
> > grep inline drivers/infiniband/core/*.c
> > drivers/infiniband/core/addr.c:static inline bool
> > ib_nl_is_good_ip_resp(const struct nlmsghdr *nlh)
> > drivers/infiniband/core/cma.c:static inline u8 cma_get_ip_ver(const
> > struct cma_hdr *hdr)
> > drivers/infiniband/core/cma.c:static inline void cma_set_ip_ver(struct
> > cma_hdr *hdr, u8 ip_ver)
> > drivers/infiniband/core/cma.c:static inline void release_mc(struct kref *kref)
> > drivers/infiniband/core/cma.c:static inline struct sockaddr
> > *cma_src_addr(struct rdma_id_private *id_priv)
> > drivers/infiniband/core/cma.c:static inline struct sockaddr
> > *cma_dst_addr(struct rdma_id_private *id_priv)
> >
> > >
> > > > +{
> > > > +     struct rnbd_queue *q = NULL;
> > > > +     struct rnbd_cpu_qlist *cpu_q;
> > > > +     unsigned long flags;
> > > > +     int *cpup;
> > > > +
> > > > +     /*
> > > > +      * To keep fairness and not to let other queues starve we always
> > > > +      * try to wake up someone else in round-robin manner.  That of course
> > > > +      * increases latency but queues always have a chance to be executed.
> > > > +      */
> > > > +     cpup = get_cpu_ptr(sess->cpu_rr);
> > > > +     for (cpu_q = rnbd_get_cpu_qlist(sess, nxt_cpu(*cpup)); cpu_q;
> > > > +          cpu_q = rnbd_get_cpu_qlist(sess, nxt_cpu(cpu_q->cpu))) {
> > > > +             if (!spin_trylock_irqsave(&cpu_q->requeue_lock, flags))
> > > > +                     continue;
> > > > +             if (likely(test_bit(cpu_q->cpu, sess->cpu_queues_bm))) {
> > >
> > > Success oriented approach please.
> > sorry, I don't quite get your point.
>
> The flows are better to be written:
> if (err)
>   return or conitnue
> <...>
> do_something
>
> in your case
> if (!test_bit(...))
>  continue;
> do_work_here.
In our case,
 if we failed to get requeue_lock, we continue to next cpu_q and do the work
I guess you miss read the code.

Thanks

  reply index

Thread overview: 51+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2020-01-16 12:58 [PATCH v7 00/25] RTRS (former IBTRS) RDMA Transport Library and RNBD (former IBNBD) RDMA Network Block Device Jack Wang
2020-01-16 12:58 ` [PATCH v7 01/25] sysfs: export sysfs_remove_file_self() Jack Wang
2020-01-16 12:58 ` [PATCH v7 02/25] RDMA/rtrs: public interface header to establish RDMA connections Jack Wang
2020-01-16 12:58 ` [PATCH v7 03/25] RDMA/rtrs: private headers with rtrs protocol structs and helpers Jack Wang
2020-01-16 12:58 ` [PATCH v7 04/25] RDMA/rtrs: core: lib functions shared between client and server modules Jack Wang
2020-01-19 14:48   ` Leon Romanovsky
2020-01-20 11:32     ` Jinpu Wang
2020-01-20 13:30       ` Leon Romanovsky
2020-01-20 14:08         ` Jinpu Wang
2020-01-16 12:58 ` [PATCH v7 05/25] RDMA/rtrs: client: private header with client structs and functions Jack Wang
2020-01-16 12:58 ` [PATCH v7 06/25] RDMA/rtrs: client: main functionality Jack Wang
2020-01-16 14:53   ` Leon Romanovsky
2020-01-16 15:43     ` Jinpu Wang
2020-01-16 15:53       ` Jason Gunthorpe
2020-01-16 16:48         ` Jinpu Wang
2020-01-16 15:58       ` Leon Romanovsky
2020-01-16 16:24         ` Jinpu Wang
2020-01-18 10:12           ` Leon Romanovsky
2020-01-20 11:24             ` Jinpu Wang
2020-01-16 12:58 ` [PATCH v7 07/25] RDMA/rtrs: client: statistics functions Jack Wang
2020-01-16 12:58 ` [PATCH v7 08/25] RDMA/rtrs: client: sysfs interface functions Jack Wang
2020-01-16 12:58 ` [PATCH v7 09/25] RDMA/rtrs: server: private header with server structs and functions Jack Wang
2020-01-16 12:59 ` [PATCH v7 10/25] RDMA/rtrs: server: main functionality Jack Wang
2020-01-16 12:59 ` [PATCH v7 11/25] RDMA/rtrs: server: statistics functions Jack Wang
2020-01-16 12:59 ` [PATCH v7 12/25] RDMA/rtrs: server: sysfs interface functions Jack Wang
2020-01-16 12:59 ` [PATCH v7 13/25] RDMA/rtrs: include client and server modules into kernel compilation Jack Wang
2020-01-16 12:59 ` [PATCH v7 14/25] RDMA/rtrs: a bit of documentation Jack Wang
2020-01-16 12:59 ` [PATCH v7 15/25] block/rnbd: private headers with rnbd protocol structs and helpers Jack Wang
2020-01-16 12:59 ` [PATCH v7 16/25] block/rnbd: client: private header with client structs and functions Jack Wang
2020-01-16 12:59 ` [PATCH v7 17/25] block/rnbd: client: main functionality Jack Wang
2020-01-20 13:48   ` Leon Romanovsky
2020-01-20 17:30     ` Jinpu Wang
2020-01-22 12:31       ` Leon Romanovsky
2020-01-22 12:52         ` Jinpu Wang
2020-01-22 11:22     ` Jinpu Wang
2020-01-22 12:25       ` Leon Romanovsky
2020-01-22 13:12         ` Jinpu Wang [this message]
2020-01-22 14:07           ` Leon Romanovsky
2020-01-22 14:18             ` Jinpu Wang
2020-01-16 12:59 ` [PATCH v7 18/25] block/rnbd: client: sysfs interface functions Jack Wang
2020-01-16 12:59 ` [PATCH v7 19/25] block/rnbd: server: private header with server structs and functions Jack Wang
2020-01-16 12:59 ` [PATCH v7 20/25] block/rnbd: server: main functionality Jack Wang
2020-01-16 12:59 ` [PATCH v7 21/25] block/rnbd: server: functionality for IO submission to file or block dev Jack Wang
2020-01-16 12:59 ` [PATCH v7 22/25] block/rnbd: server: sysfs interface functions Jack Wang
2020-01-16 12:59 ` [PATCH v7 23/25] block/rnbd: include client and server modules into kernel compilation Jack Wang
2020-01-16 14:40   ` Leon Romanovsky
2020-01-16 14:54     ` Jinpu Wang
2020-01-16 15:59       ` Leon Romanovsky
2020-01-16 16:53         ` Jinpu Wang
2020-01-16 12:59 ` [PATCH v7 24/25] block/rnbd: a bit of documentation Jack Wang
2020-01-16 12:59 ` [PATCH v7 25/25] MAINTAINERS: Add maintainers for RNBD/RTRS modules Jack Wang

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=CAMGffEmpRvruSn6iz6EfgfAjE9xrnsihwPaQU8Ft9e7qLD5avw@mail.gmail.com \
    --to=jinpu.wang@cloud.ionos.com \
    --cc=axboe@kernel.dk \
    --cc=bvanassche@acm.org \
    --cc=danil.kipnis@cloud.ionos.com \
    --cc=dledford@redhat.com \
    --cc=hch@infradead.org \
    --cc=jgg@ziepe.ca \
    --cc=jinpuwang@gmail.com \
    --cc=leon@kernel.org \
    --cc=linux-block@vger.kernel.org \
    --cc=linux-rdma@vger.kernel.org \
    --cc=rpenyaev@suse.de \
    --cc=sagi@grimberg.me \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

Linux-Block Archive on lore.kernel.org

Archives are clonable:
	git clone --mirror https://lore.kernel.org/linux-block/0 linux-block/git/0.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 linux-block linux-block/ https://lore.kernel.org/linux-block \
		linux-block@vger.kernel.org
	public-inbox-index linux-block

Example config snippet for mirrors

Newsgroup available over NNTP:
	nntp://nntp.lore.kernel.org/org.kernel.vger.linux-block


AGPL code for this site: git clone https://public-inbox.org/public-inbox.git