From: Christoph Hellwig <hch-jcswGhMUV9g@public.gmane.org> To: Jason Gunthorpe <jgunthorpe-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org> Cc: Bart Van Assche <bart.vanassche-XdAiOPVOjttBDgjK7y7TUQ@public.gmane.org>, Christoph Hellwig <hch-jcswGhMUV9g@public.gmane.org>, linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org, sagig-LDSdmyG8hGV8YrgS2mwiifqBs+8SCbDb@public.gmane.org, axboe-b10kYP2dOMg@public.gmane.org, linux-scsi-u79uwXL29TY76Z2rM5mHXA@public.gmane.org, linux-kernel-u79uwXL29TY76Z2rM5mHXA@public.gmane.org Subject: Re: [PATCH 2/9] IB: add a proper completion queue abstraction Date: Sat, 14 Nov 2015 08:13:44 +0100 [thread overview] Message-ID: <20151114071344.GE27738@lst.de> (raw) In-Reply-To: <20151113220636.GA32133-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org> On Fri, Nov 13, 2015 at 03:06:36PM -0700, Jason Gunthorpe wrote: > Looking at that thread and then at the patch a bit more.. > > +void ib_process_cq_direct(struct ib_cq *cq) > [..] > + __ib_process_cq(cq, INT_MAX); > > INT_MAX is not enough, it needs to loop. > This is missing a ib_req_notify also. No. Direct cases _never_ calls ib_req_notify. Its for the case where the SRP case polls the send CQ only from the same context it sends for without any interrupt notification at al. > +static int __ib_process_cq(struct ib_cq *cq, int budget) > + while ((n = ib_poll_cq(cq, IB_POLL_BATCH, cq->wc)) > 0) { > > Does an unnecessary ib_poll_cq call in common cases. I'd suggest > change the result to bool and do: > > // true return means the caller should attempt ib_req_notify_cq > while ((n = ib_poll_cq(cq, IB_POLL_BATCH, cq->wc)) > 0) { > for (...) > if (n != IB_POLL_BATCH) > return true; > completed += n; > if (completed > budget) > return false; > } > return true; > > And then change call site like: > > static void ib_cq_poll_work(struct work_struct *work) > { > if (__ib_process_cq(...)) > if (ib_req_notify_cq(cq, IB_POLL_FLAGS) == 0) > return; > // Else we need to loop again. > queue_work(ib_comp_wq, &cq->work); > } > > Which avoids the rearm. > > void ib_process_cq_direct(struct ib_cq *cq) > { > while (1) { > if (__ib_process_cq(..) && > ib_req_notify_cq(cq, IB_POLL_FLAGS) == 0) > return; > } > } > > Which adds the inf loop and rearm. > > etc for softirq For the workqueue and softirq cases this looks reasonable. For the direct case there is no rearming, though. > Perhaps ib_req_notify_cq should be folded into __ib_process_cq, then > it can trivially honour the budget on additional loops from > IB_CQ_REPORT_MISSED_EVENTS. Which also defeats this proposal. -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo info at http://vger.kernel.org/majordomo-info.html
WARNING: multiple messages have this Message-ID (diff)
From: Christoph Hellwig <hch@lst.de> To: Jason Gunthorpe <jgunthorpe@obsidianresearch.com> Cc: Bart Van Assche <bart.vanassche@sandisk.com>, Christoph Hellwig <hch@lst.de>, linux-rdma@vger.kernel.org, sagig@dev.mellanox.co.il, axboe@fb.com, linux-scsi@vger.kernel.org, linux-kernel@vger.kernel.org Subject: Re: [PATCH 2/9] IB: add a proper completion queue abstraction Date: Sat, 14 Nov 2015 08:13:44 +0100 [thread overview] Message-ID: <20151114071344.GE27738@lst.de> (raw) In-Reply-To: <20151113220636.GA32133@obsidianresearch.com> On Fri, Nov 13, 2015 at 03:06:36PM -0700, Jason Gunthorpe wrote: > Looking at that thread and then at the patch a bit more.. > > +void ib_process_cq_direct(struct ib_cq *cq) > [..] > + __ib_process_cq(cq, INT_MAX); > > INT_MAX is not enough, it needs to loop. > This is missing a ib_req_notify also. No. Direct cases _never_ calls ib_req_notify. Its for the case where the SRP case polls the send CQ only from the same context it sends for without any interrupt notification at al. > +static int __ib_process_cq(struct ib_cq *cq, int budget) > + while ((n = ib_poll_cq(cq, IB_POLL_BATCH, cq->wc)) > 0) { > > Does an unnecessary ib_poll_cq call in common cases. I'd suggest > change the result to bool and do: > > // true return means the caller should attempt ib_req_notify_cq > while ((n = ib_poll_cq(cq, IB_POLL_BATCH, cq->wc)) > 0) { > for (...) > if (n != IB_POLL_BATCH) > return true; > completed += n; > if (completed > budget) > return false; > } > return true; > > And then change call site like: > > static void ib_cq_poll_work(struct work_struct *work) > { > if (__ib_process_cq(...)) > if (ib_req_notify_cq(cq, IB_POLL_FLAGS) == 0) > return; > // Else we need to loop again. > queue_work(ib_comp_wq, &cq->work); > } > > Which avoids the rearm. > > void ib_process_cq_direct(struct ib_cq *cq) > { > while (1) { > if (__ib_process_cq(..) && > ib_req_notify_cq(cq, IB_POLL_FLAGS) == 0) > return; > } > } > > Which adds the inf loop and rearm. > > etc for softirq For the workqueue and softirq cases this looks reasonable. For the direct case there is no rearming, though. > Perhaps ib_req_notify_cq should be folded into __ib_process_cq, then > it can trivially honour the budget on additional loops from > IB_CQ_REPORT_MISSED_EVENTS. Which also defeats this proposal.
next prev parent reply other threads:[~2015-11-14 7:13 UTC|newest] Thread overview: 144+ messages / expand[flat|nested] mbox.gz Atom feed top 2015-11-13 13:46 add a proper completion queue abstraction Christoph Hellwig 2015-11-13 13:46 ` [PATCH 1/9] move blk_iopoll to limit and make it generally available Christoph Hellwig [not found] ` <1447422410-20891-2-git-send-email-hch-jcswGhMUV9g@public.gmane.org> 2015-11-13 15:23 ` Or Gerlitz 2015-11-13 15:23 ` Or Gerlitz [not found] ` <CAJ3xEMgj2ycv61K38ZOowTRbrri_UhQgBcaKT0ZnnMHiBrmL5A-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org> 2015-11-14 7:02 ` Christoph Hellwig 2015-11-14 7:02 ` Christoph Hellwig [not found] ` <20151114070200.GA27738-jcswGhMUV9g@public.gmane.org> 2015-11-15 8:48 ` Sagi Grimberg 2015-11-15 8:48 ` Sagi Grimberg [not found] ` <564846E9.9070301-LDSdmyG8hGV8YrgS2mwiifqBs+8SCbDb@public.gmane.org> 2015-11-15 9:04 ` Or Gerlitz 2015-11-15 9:04 ` Or Gerlitz [not found] ` <CAJ3xEMgvttM1D3bePz0CWhZAZ3gCSQsf_qgmq9Ny4gzK5d0bXw-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org> 2015-11-15 13:16 ` Sagi Grimberg 2015-11-15 13:16 ` Sagi Grimberg 2015-11-15 12:51 ` Christoph Hellwig 2015-11-15 12:51 ` Christoph Hellwig 2015-11-13 19:19 ` Bart Van Assche 2015-11-13 19:19 ` Bart Van Assche 2015-11-14 7:02 ` Christoph Hellwig 2015-11-17 17:16 ` Bart Van Assche 2015-11-17 17:16 ` Bart Van Assche 2015-11-17 17:27 ` Bart Van Assche 2015-11-18 13:58 ` Christoph Hellwig 2015-11-13 13:46 ` [PATCH 2/9] IB: add a proper completion queue abstraction Christoph Hellwig 2015-11-15 9:40 ` Sagi Grimberg [not found] ` <564852F2.5080602-LDSdmyG8hGV8YrgS2mwiifqBs+8SCbDb@public.gmane.org> 2015-11-15 12:55 ` Christoph Hellwig 2015-11-15 12:55 ` Christoph Hellwig [not found] ` <20151115125501.GB2218-jcswGhMUV9g@public.gmane.org> 2015-11-15 13:21 ` Sagi Grimberg 2015-11-15 13:21 ` Sagi Grimberg [not found] ` <1447422410-20891-3-git-send-email-hch-jcswGhMUV9g@public.gmane.org> 2015-11-13 18:25 ` Jason Gunthorpe 2015-11-13 18:25 ` Jason Gunthorpe [not found] ` <20151113182513.GB21808-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org> 2015-11-13 19:57 ` Bart Van Assche 2015-11-13 19:57 ` Bart Van Assche 2015-11-13 22:06 ` Jason Gunthorpe [not found] ` <20151113220636.GA32133-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org> 2015-11-14 7:13 ` Christoph Hellwig [this message] 2015-11-14 7:13 ` Christoph Hellwig [not found] ` <20151114071344.GE27738-jcswGhMUV9g@public.gmane.org> 2015-11-23 20:37 ` Jason Gunthorpe 2015-11-23 20:37 ` Jason Gunthorpe 2015-11-23 21:04 ` Bart Van Assche 2015-11-23 21:04 ` Bart Van Assche 2015-11-23 21:28 ` Jason Gunthorpe 2015-11-23 21:54 ` Bart Van Assche 2015-11-23 22:18 ` Jason Gunthorpe 2015-11-23 22:33 ` Bart Van Assche 2015-11-23 23:06 ` Jason Gunthorpe [not found] ` <B24F4DDE-709A-4D2D-8B26-4E83325DBB1A@asomi.com> 2015-11-24 0:00 ` Jason Gunthorpe [not found] ` <20151124000011.GA9301-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org> 2015-11-24 0:34 ` Tom Talpey 2015-11-24 0:34 ` Tom Talpey [not found] ` <5653B0AD.7090402-CLs1Zie5N5HQT0dZR+AlfA@public.gmane.org> 2015-11-24 0:40 ` Jason Gunthorpe 2015-11-24 0:40 ` Jason Gunthorpe 2015-11-24 2:35 ` Caitlin Bestler 2015-11-24 2:35 ` Caitlin Bestler [not found] ` <5653CCF0.7050501-DpaxOq6QOWMAvxtiuMwx3w@public.gmane.org> 2015-11-24 7:03 ` Jason Gunthorpe 2015-11-24 7:03 ` Jason Gunthorpe [not found] ` <20151124070301.GA23597-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org> 2015-11-24 12:52 ` Tom Talpey 2015-11-24 12:52 ` Tom Talpey 2015-11-14 7:08 ` Christoph Hellwig 2015-11-14 7:08 ` Christoph Hellwig 2015-11-23 20:01 ` Jason Gunthorpe [not found] ` <20151123200136.GA5640-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org> 2015-11-23 20:57 ` Christoph Hellwig 2015-11-23 20:57 ` Christoph Hellwig 2015-11-17 17:52 ` Bart Van Assche 2015-11-17 17:52 ` Bart Van Assche [not found] ` <564B697A.2020601-XdAiOPVOjttBDgjK7y7TUQ@public.gmane.org> 2015-11-18 7:55 ` Sagi Grimberg 2015-11-18 7:55 ` Sagi Grimberg [not found] ` <564C2F01.6020407-LDSdmyG8hGV8YrgS2mwiifqBs+8SCbDb@public.gmane.org> 2015-11-18 18:20 ` Bart Van Assche 2015-11-18 18:20 ` Bart Van Assche 2015-11-20 10:16 ` Christoph Hellwig 2015-11-20 16:50 ` Bart Van Assche 2015-11-22 9:51 ` Sagi Grimberg 2015-11-22 10:13 ` Christoph Hellwig [not found] ` <20151122101308.GA12189-jcswGhMUV9g@public.gmane.org> 2015-11-22 10:36 ` Sagi Grimberg 2015-11-22 10:36 ` Sagi Grimberg [not found] ` <56519A90.5010502-LDSdmyG8hGV8YrgS2mwiifqBs+8SCbDb@public.gmane.org> 2015-11-22 13:23 ` Christoph Hellwig 2015-11-22 13:23 ` Christoph Hellwig [not found] ` <20151122132352.GA14154-jcswGhMUV9g@public.gmane.org> 2015-11-22 14:57 ` Sagi Grimberg 2015-11-22 14:57 ` Sagi Grimberg 2015-11-22 16:55 ` Bart Van Assche 2015-11-18 14:00 ` Christoph Hellwig 2015-11-18 14:00 ` Christoph Hellwig 2015-11-13 13:46 ` [PATCH 3/9] IB: add a helper to safely drain a QP Christoph Hellwig [not found] ` <1447422410-20891-4-git-send-email-hch-jcswGhMUV9g@public.gmane.org> 2015-11-13 16:16 ` Steve Wise 2015-11-13 16:16 ` Steve Wise [not found] ` <56460CC4.3030001-7bPotxP6k4+P2YhJcF5u+vpXobYPEAuW@public.gmane.org> 2015-11-14 7:05 ` Christoph Hellwig 2015-11-14 7:05 ` Christoph Hellwig 2015-11-18 11:32 ` Sagi Grimberg 2015-11-18 11:32 ` Sagi Grimberg [not found] ` <564C61C3.3050307-LDSdmyG8hGV8YrgS2mwiifqBs+8SCbDb@public.gmane.org> 2015-11-18 14:06 ` Christoph Hellwig 2015-11-18 14:06 ` Christoph Hellwig [not found] ` <20151118140645.GI18820-jcswGhMUV9g@public.gmane.org> 2015-11-18 15:21 ` Steve Wise 2015-11-18 15:21 ` Steve Wise 2015-11-15 9:34 ` Sagi Grimberg [not found] ` <564851BB.1020004-LDSdmyG8hGV8YrgS2mwiifqBs+8SCbDb@public.gmane.org> 2015-11-16 16:38 ` Steve Wise 2015-11-16 16:38 ` Steve Wise [not found] ` <564A067B.8030504-7bPotxP6k4+P2YhJcF5u+vpXobYPEAuW@public.gmane.org> 2015-11-16 18:30 ` Steve Wise 2015-11-16 18:30 ` Steve Wise 2015-11-16 18:37 ` Sagi Grimberg [not found] ` <564A2270.1040004-LDSdmyG8hGV8YrgS2mwiifqBs+8SCbDb@public.gmane.org> 2015-11-16 19:03 ` Steve Wise 2015-11-16 19:03 ` Steve Wise 2015-11-17 8:54 ` Sagi Grimberg 2015-11-17 8:54 ` Sagi Grimberg 2015-11-23 10:28 ` Sagi Grimberg 2015-11-23 10:28 ` Sagi Grimberg 2015-11-23 10:35 ` Sagi Grimberg 2015-11-23 14:33 ` 'Christoph Hellwig' [not found] ` <5652EC00.8010705-LDSdmyG8hGV8YrgS2mwiifqBs+8SCbDb@public.gmane.org> 2015-11-23 14:48 ` Steve Wise 2015-11-23 14:48 ` Steve Wise 2015-11-23 14:44 ` Steve Wise 2015-11-23 14:44 ` Steve Wise 2015-11-17 17:06 ` Bart Van Assche 2015-11-17 17:06 ` Bart Van Assche [not found] ` <564B5E7D.9030309-XdAiOPVOjttBDgjK7y7TUQ@public.gmane.org> 2015-11-18 7:59 ` Sagi Grimberg 2015-11-18 7:59 ` Sagi Grimberg [not found] ` <1447422410-20891-1-git-send-email-hch-jcswGhMUV9g@public.gmane.org> 2015-11-13 13:46 ` [PATCH 4/9] srpt: chain RDMA READ/WRITE requests Christoph Hellwig 2015-11-13 13:46 ` Christoph Hellwig 2015-11-18 1:17 ` Bart Van Assche 2015-11-18 1:17 ` Bart Van Assche 2015-11-18 9:15 ` Sagi Grimberg 2015-11-18 16:32 ` Bart Van Assche 2015-11-18 16:32 ` Bart Van Assche [not found] ` <564CA83B.4060403-XdAiOPVOjttBDgjK7y7TUQ@public.gmane.org> 2015-11-20 10:20 ` Christoph Hellwig 2015-11-20 10:20 ` Christoph Hellwig [not found] ` <564BD1AF.60200-XdAiOPVOjttBDgjK7y7TUQ@public.gmane.org> 2015-11-18 14:06 ` Christoph Hellwig 2015-11-18 14:06 ` Christoph Hellwig 2015-11-13 13:46 ` [PATCH 5/9] srpt: use the new CQ API Christoph Hellwig [not found] ` <1447422410-20891-6-git-send-email-hch-jcswGhMUV9g@public.gmane.org> 2015-11-17 18:22 ` Bart Van Assche 2015-11-17 18:22 ` Bart Van Assche 2015-11-17 19:38 ` Bart Van Assche 2015-11-17 19:38 ` Bart Van Assche [not found] ` <564B8248.7050407-XdAiOPVOjttBDgjK7y7TUQ@public.gmane.org> 2015-11-18 14:03 ` Christoph Hellwig 2015-11-18 14:03 ` Christoph Hellwig 2015-11-13 13:46 ` [PATCH 6/9] srp: " Christoph Hellwig [not found] ` <1447422410-20891-7-git-send-email-hch-jcswGhMUV9g@public.gmane.org> 2015-11-17 19:56 ` Bart Van Assche 2015-11-17 19:56 ` Bart Van Assche 2015-11-18 14:03 ` Christoph Hellwig 2015-11-13 13:46 ` [PATCH 7/9] IB/iser: Use a dedicated descriptor for login Christoph Hellwig [not found] ` <1447422410-20891-8-git-send-email-hch-jcswGhMUV9g@public.gmane.org> 2015-11-15 9:14 ` Or Gerlitz 2015-11-15 9:14 ` Or Gerlitz 2015-11-13 13:46 ` [PATCH 8/9] IB/iser: Use helper for container_of Christoph Hellwig 2015-11-13 13:46 ` [PATCH 9/9] IB/iser: Convert to CQ abstraction Christoph Hellwig [not found] ` <1447422410-20891-10-git-send-email-hch-jcswGhMUV9g@public.gmane.org> 2015-11-15 9:21 ` Or Gerlitz 2015-11-15 9:21 ` Or Gerlitz [not found] <20151124100839.48b52fb35c6f209c51bccbb9807b6df0.f113bf890f.wbe@email24.secureserver.net> [not found] ` <20151124100839.48b52fb35c6f209c51bccbb9807b6df0.f113bf890f.wbe-wCzC33v8tqnShzhksYgB+AejPw4fNl8p@public.gmane.org> 2015-11-24 17:52 ` [PATCH 2/9] IB: add a proper completion queue abstraction Jason Gunthorpe 2015-11-24 17:52 ` Jason Gunthorpe [not found] ` <56552132.7090701@asomi.com> [not found] ` <56552132.7090701-DpaxOq6QOWMAvxtiuMwx3w@public.gmane.org> 2015-11-25 6:21 ` Jason Gunthorpe 2015-11-25 6:21 ` Jason Gunthorpe
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to=20151114071344.GE27738@lst.de \ --to=hch-jcswghmuv9g@public.gmane.org \ --cc=axboe-b10kYP2dOMg@public.gmane.org \ --cc=bart.vanassche-XdAiOPVOjttBDgjK7y7TUQ@public.gmane.org \ --cc=jgunthorpe-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org \ --cc=linux-kernel-u79uwXL29TY76Z2rM5mHXA@public.gmane.org \ --cc=linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org \ --cc=linux-scsi-u79uwXL29TY76Z2rM5mHXA@public.gmane.org \ --cc=sagig-LDSdmyG8hGV8YrgS2mwiifqBs+8SCbDb@public.gmane.org \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: linkBe sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.