From: Tom Talpey <tom-CLs1Zie5N5HQT0dZR+AlfA@public.gmane.org> To: Chuck Lever <chuck.lever-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org>, Jason Gunthorpe <jgunthorpe-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org> Cc: Sagi Grimberg <sagi-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org>, Leon Romanovsky <leon-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org>, Robert LeBlanc <robert-4JaGZRWAfWbajFs6igw21g@public.gmane.org>, Marta Rybczynska <mrybczyn-FNhOzJFKnXGHXe+LvDLADg@public.gmane.org>, Max Gurtovoy <maxg-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>, Christoph Hellwig <hch-wEGCiKHe2LqWVfeAwA7xHQ@public.gmane.org>, "Gruher, Joseph R" <joseph.r.gruher-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>, "shahar.salzman" <shahar.salzman-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>, Laurence Oberman <loberman-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>, "Riches Jr, Robert M" <robert.m.riches.jr-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>, linux-rdma <linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org>, linux-nvme-IAPFreCvJWM7uuMidbF8XUB+6BGkLq7r@public.gmane.org, Liran Liss <liranl-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>, Bart Van Assche <bart.vanassche-XdAiOPVOjttBDgjK7y7TUQ@public.gmane.org> Subject: Re: Unexpected issues with 2 NVME initiators using the same target Date: Tue, 11 Jul 2017 09:23:54 -0400 [thread overview] Message-ID: <8dd77b19-3846-96ea-f50f-22182989e941@talpey.com> (raw) In-Reply-To: <718A099D-3597-4262-9A33-0BA7EFE5461F-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org> On 7/10/2017 11:57 PM, Chuck Lever wrote: > >> On Jul 10, 2017, at 6:09 PM, Jason Gunthorpe <jgunthorpe-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org> wrote: >> >> On Mon, Jul 10, 2017 at 06:04:18PM -0400, Chuck Lever wrote: >> >>>> The server sounds fine, how does the client work? >>> >>> The client does not initiate RDMA Read or Write today. >> >> Right, but it provides an rkey that the server uses for READ or WRITE. >> >> The invalidate of that rkey at the client must follow the same rules >> as inline send. > > Ah, I see. > > The RPC reply handler calls frwr_op_unmap_sync to invalidate > any MRs associated with the RPC. > > frwr_op_unmap_sync has to sort the rkeys that are remotely > invalidated, and those that have not been. Does the reply handler consider the possibility that the reply is being signaled before the send WRs? There are some really interesting races on shared or multiple CQs when the completion upcalls start to back up under heavy load that we've seen in Windows SMB Direct. In the end, we had to put explicit reference counts on each and every object, and added rundown references to everything before completing an operation and signaling the upper layer (SMB3, in our case). This found a surprising number of double completions, and missing completions from drivers as well. > The first step is to ensure all the rkeys for an RPC are > invalid. The rkey that was remotely invalidated is skipped > here, and a chain of LocalInv WRs is posted to invalidate > any remaining rkeys. The last WR in the chain is signaled. > > If one or more LocalInv WRs are posted, this function waits > for LocalInv completion. > > The last step is always DMA unmapping. Note that we can't > get a completion for a remotely invalidated rkey, and we > have to wait for LocalInv to complete anyway. So the DMA > unmapping is always handled here instead of in a > completion handler. > > When frwr_op_unmap_sync returns to the RPC reply handler, > the handler calls xprt_complete_rqst, and the RPC is > terminated. This guarantees that the MRs are invalid before > control is returned to the RPC consumer. > > > In the ^C case, frwr_op_unmap_safe is invoked during RPC > termination. The MRs are passed to the background recovery > task, which invokes frwr_op_recover_mr. That worries me. How do you know it's going in sequence, and that it will result in an invalidated MR? > frwr_op_recover_mr destroys the fr_mr and DMA unmaps the > memory. (It's also used when registration or invalidation > flushes, which is why it uses a hammer). > > So here, we're a little fast/loose: the ordering of > invalidation and unmapping is correct, but the MRs can be > invalidated after the RPC completes. Since RPC termination > can't wait, this is the best I can do for now. That would worry me even more. "fast/loose" isn't a good situation when storage is concerned. Shouldn't you just be closing the connection? Tom. -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo info at http://vger.kernel.org/majordomo-info.html
WARNING: multiple messages have this Message-ID (diff)
From: tom@talpey.com (Tom Talpey) Subject: Unexpected issues with 2 NVME initiators using the same target Date: Tue, 11 Jul 2017 09:23:54 -0400 [thread overview] Message-ID: <8dd77b19-3846-96ea-f50f-22182989e941@talpey.com> (raw) In-Reply-To: <718A099D-3597-4262-9A33-0BA7EFE5461F@oracle.com> On 7/10/2017 11:57 PM, Chuck Lever wrote: > >> On Jul 10, 2017,@6:09 PM, Jason Gunthorpe <jgunthorpe@obsidianresearch.com> wrote: >> >> On Mon, Jul 10, 2017@06:04:18PM -0400, Chuck Lever wrote: >> >>>> The server sounds fine, how does the client work? >>> >>> The client does not initiate RDMA Read or Write today. >> >> Right, but it provides an rkey that the server uses for READ or WRITE. >> >> The invalidate of that rkey at the client must follow the same rules >> as inline send. > > Ah, I see. > > The RPC reply handler calls frwr_op_unmap_sync to invalidate > any MRs associated with the RPC. > > frwr_op_unmap_sync has to sort the rkeys that are remotely > invalidated, and those that have not been. Does the reply handler consider the possibility that the reply is being signaled before the send WRs? There are some really interesting races on shared or multiple CQs when the completion upcalls start to back up under heavy load that we've seen in Windows SMB Direct. In the end, we had to put explicit reference counts on each and every object, and added rundown references to everything before completing an operation and signaling the upper layer (SMB3, in our case). This found a surprising number of double completions, and missing completions from drivers as well. > The first step is to ensure all the rkeys for an RPC are > invalid. The rkey that was remotely invalidated is skipped > here, and a chain of LocalInv WRs is posted to invalidate > any remaining rkeys. The last WR in the chain is signaled. > > If one or more LocalInv WRs are posted, this function waits > for LocalInv completion. > > The last step is always DMA unmapping. Note that we can't > get a completion for a remotely invalidated rkey, and we > have to wait for LocalInv to complete anyway. So the DMA > unmapping is always handled here instead of in a > completion handler. > > When frwr_op_unmap_sync returns to the RPC reply handler, > the handler calls xprt_complete_rqst, and the RPC is > terminated. This guarantees that the MRs are invalid before > control is returned to the RPC consumer. > > > In the ^C case, frwr_op_unmap_safe is invoked during RPC > termination. The MRs are passed to the background recovery > task, which invokes frwr_op_recover_mr. That worries me. How do you know it's going in sequence, and that it will result in an invalidated MR? > frwr_op_recover_mr destroys the fr_mr and DMA unmaps the > memory. (It's also used when registration or invalidation > flushes, which is why it uses a hammer). > > So here, we're a little fast/loose: the ordering of > invalidation and unmapping is correct, but the MRs can be > invalidated after the RPC completes. Since RPC termination > can't wait, this is the best I can do for now. That would worry me even more. "fast/loose" isn't a good situation when storage is concerned. Shouldn't you just be closing the connection? Tom.
next prev parent reply other threads:[~2017-07-11 13:23 UTC|newest] Thread overview: 171+ messages / expand[flat|nested] mbox.gz Atom feed top 2017-02-21 19:38 Unexpected issues with 2 NVME initiators using the same target shahar.salzman [not found] ` <08131a05-1f56-ef61-990a-7fff04eea095-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> 2017-02-21 22:50 ` Sagi Grimberg 2017-02-21 22:50 ` Sagi Grimberg [not found] ` <de1a559a-bf24-0d73-5fc7-148d6cd4d4e0-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org> 2017-02-22 16:52 ` Laurence Oberman 2017-02-22 16:52 ` Laurence Oberman [not found] ` <1848296658.37025722.1487782361271.JavaMail.zimbra-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org> 2017-02-22 19:39 ` Sagi Grimberg 2017-02-22 19:39 ` Sagi Grimberg 2017-02-26 8:03 ` shahar.salzman 2017-02-26 8:03 ` shahar.salzman 2017-02-26 17:58 ` Gruher, Joseph R [not found] ` <DE927C68B458BE418D582EC97927A92854655137-8oqHQFITsIFcIJlls4ac1rfspsVTdybXVpNB7YpNyf8@public.gmane.org> 2017-02-27 20:33 ` Sagi Grimberg 2017-02-27 20:33 ` Sagi Grimberg 2017-02-27 20:57 ` Gruher, Joseph R 2017-03-05 18:23 ` Leon Romanovsky 2017-03-06 0:07 ` Max Gurtovoy [not found] ` <26912d0c-578f-26e9-490d-94fc95bdf259-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org> 2017-03-06 11:28 ` Sagi Grimberg 2017-03-06 11:28 ` Sagi Grimberg 2017-03-07 9:27 ` Max Gurtovoy [not found] ` <fbd647dd-3a16-8155-107d-f98e8326cc63-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org> 2017-03-07 13:41 ` Sagi Grimberg 2017-03-07 13:41 ` Sagi Grimberg [not found] ` <c5a6f55b-a633-53ca-4378-7a7eaf7f77bd-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org> 2017-03-09 12:18 ` shahar.salzman 2017-03-09 12:18 ` shahar.salzman 2017-03-12 12:33 ` Vladimir Neyelov [not found] ` <AM4PR0501MB278621363209E177A738D75FCB220-dp/nxUn679hhbxXPg6FtWcDSnupUy6xnnBOFsp37pqbUKgpGm//BTAC/G2K4zDHf@public.gmane.org> 2017-03-13 9:43 ` Sagi Grimberg 2017-03-13 9:43 ` Sagi Grimberg 2017-03-14 8:55 ` Max Gurtovoy 2017-03-14 19:57 ` Gruher, Joseph R 2017-03-14 23:42 ` Gruher, Joseph R 2017-03-16 0:03 ` Gruher, Joseph R 2017-03-17 18:37 ` Gruher, Joseph R 2017-03-17 19:49 ` Max Gurtovoy [not found] ` <DE927C68B458BE418D582EC97927A928550391C2@ORSMSX113.amr.corp.intel.com> 2017-03-24 18:30 ` Gruher, Joseph R 2017-03-27 14:17 ` Max Gurtovoy 2017-03-27 15:39 ` Gruher, Joseph R 2017-03-28 8:38 ` Max Gurtovoy 2017-03-28 10:21 ` shahar.salzman [not found] ` <DE927C68B458BE418D582EC97927A928550419FA-8oqHQFITsIFQxe9IK+vIArfspsVTdybXVpNB7YpNyf8@public.gmane.org> 2017-03-28 11:34 ` Sagi Grimberg 2017-03-28 11:34 ` Sagi Grimberg [not found] ` <809f87ab-b787-9d40-5840-07500d12e81a-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org> 2017-04-10 11:40 ` Marta Rybczynska 2017-04-10 11:40 ` Marta Rybczynska 2017-04-10 14:09 ` Max Gurtovoy [not found] ` <33e2cc35-f147-d4a4-9a42-8f1245e35842-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org> 2017-04-11 12:47 ` Marta Rybczynska 2017-04-11 12:47 ` Marta Rybczynska 2017-04-20 10:18 ` Sagi Grimberg 2017-04-20 10:18 ` Sagi Grimberg 2017-04-26 11:56 ` Max Gurtovoy [not found] ` <af9044c2-fb7c-e8b3-d8fc-4874cfd1bb67-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org> 2017-04-26 14:45 ` Sagi Grimberg 2017-04-26 14:45 ` Sagi Grimberg 2017-05-12 19:20 ` Gruher, Joseph R [not found] ` <DE927C68B458BE418D582EC97927A92855088C6F-8oqHQFITsIFQxe9IK+vIArfspsVTdybXVpNB7YpNyf8@public.gmane.org> 2017-05-15 12:00 ` Sagi Grimberg 2017-05-15 12:00 ` Sagi Grimberg [not found] ` <82dd5b24-5657-ae5e-8a33-646fddd8b75b-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org> 2017-05-15 13:31 ` Leon Romanovsky 2017-05-15 13:31 ` Leon Romanovsky [not found] ` <20170515133122.GG3616-U/DQcQFIOTAAJjI8aNfphQ@public.gmane.org> 2017-05-15 13:43 ` Sagi Grimberg 2017-05-15 13:43 ` Sagi Grimberg [not found] ` <9465cd0c-83db-b058-7615-5626ef60dbb0-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org> 2017-05-15 14:36 ` Leon Romanovsky 2017-05-15 14:36 ` Leon Romanovsky [not found] ` <20170515143632.GH3616-U/DQcQFIOTAAJjI8aNfphQ@public.gmane.org> 2017-05-15 14:59 ` Christoph Hellwig 2017-05-15 14:59 ` Christoph Hellwig [not found] ` <20170515145952.GA7871-wEGCiKHe2LqWVfeAwA7xHQ@public.gmane.org> 2017-05-15 17:05 ` Leon Romanovsky 2017-05-15 17:05 ` Leon Romanovsky [not found] ` <20170515170506.GK3616-U/DQcQFIOTAAJjI8aNfphQ@public.gmane.org> 2017-05-17 12:56 ` Marta Rybczynska 2017-05-17 12:56 ` Marta Rybczynska [not found] ` <779753075.36035391.1495025796237.JavaMail.zimbra-FNhOzJFKnXGHXe+LvDLADg@public.gmane.org> 2017-05-18 13:34 ` Leon Romanovsky 2017-05-18 13:34 ` Leon Romanovsky [not found] ` <20170518133439.GD3616-U/DQcQFIOTAAJjI8aNfphQ@public.gmane.org> 2017-06-19 17:21 ` Robert LeBlanc 2017-06-19 17:21 ` Robert LeBlanc [not found] ` <CAANLjFrCLpX3nb3q7LpFPpLJKciU+1Hvmt_hxyTovQJM2-zQmg-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org> 2017-06-20 6:39 ` Sagi Grimberg 2017-06-20 6:39 ` Sagi Grimberg [not found] ` <6073e553-e8c2-6d14-ba5d-c2bd5aff15eb-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org> 2017-06-20 7:46 ` Leon Romanovsky 2017-06-20 7:46 ` Leon Romanovsky [not found] ` <20170620074639.GP17846-U/DQcQFIOTAAJjI8aNfphQ@public.gmane.org> 2017-06-20 7:58 ` Sagi Grimberg 2017-06-20 7:58 ` Sagi Grimberg [not found] ` <1c706958-992e-b104-6bae-4a6616c0a9f9-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org> 2017-06-20 8:33 ` Leon Romanovsky 2017-06-20 8:33 ` Leon Romanovsky [not found] ` <20170620083309.GQ17846-U/DQcQFIOTAAJjI8aNfphQ@public.gmane.org> 2017-06-20 9:33 ` Sagi Grimberg 2017-06-20 9:33 ` Sagi Grimberg 2017-06-20 10:31 ` Max Gurtovoy [not found] ` <78b2c1db-6ece-0274-c4c9-5ee1f7c88469-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org> 2017-06-20 22:58 ` Robert LeBlanc 2017-06-20 22:58 ` Robert LeBlanc 2017-06-27 7:16 ` Sagi Grimberg 2017-06-27 7:16 ` Sagi Grimberg [not found] ` <bd0b986f-9bed-3dfa-7454-0661559a527b-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org> 2017-06-20 12:02 ` Sagi Grimberg 2017-06-20 12:02 ` Sagi Grimberg 2017-06-20 13:28 ` Max Gurtovoy [not found] ` <614481c7-22dd-d93b-e97e-52f868727ec3-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org> 2017-06-20 17:01 ` Chuck Lever 2017-06-20 17:01 ` Chuck Lever [not found] ` <59FF0C04-2BFB-4F66-81BA-A598A9A087FC-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org> 2017-06-20 17:12 ` Sagi Grimberg 2017-06-20 17:12 ` Sagi Grimberg 2017-06-20 17:35 ` Jason Gunthorpe 2017-06-20 17:35 ` Jason Gunthorpe [not found] ` <20170620173532.GA827-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org> 2017-06-20 18:17 ` Chuck Lever 2017-06-20 18:17 ` Chuck Lever [not found] ` <D3DC49A2-FFC9-4F62-8876-3E6AD5167DE5-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org> 2017-06-20 19:27 ` Jason Gunthorpe 2017-06-20 19:27 ` Jason Gunthorpe [not found] ` <20170620192742.GB827-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org> 2017-06-20 20:56 ` Chuck Lever 2017-06-20 20:56 ` Chuck Lever [not found] ` <C14B071E-F1B2-466A-82CF-4E20BFAD9DC1-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org> 2017-06-20 21:19 ` Jason Gunthorpe 2017-06-20 21:19 ` Jason Gunthorpe [not found] ` <20170620211958.GA5574-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org> 2017-06-27 7:37 ` Sagi Grimberg 2017-06-27 7:37 ` Sagi Grimberg [not found] ` <4f0812f1-0067-4e63-e383-b913ee1f319d-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org> 2017-06-27 14:42 ` Chuck Lever 2017-06-27 14:42 ` Chuck Lever [not found] ` <28F6F58E-B6F4-4114-8DFF-B72353CE814B-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org> 2017-06-27 16:07 ` Sagi Grimberg 2017-06-27 16:07 ` Sagi Grimberg [not found] ` <52ad3547-efcf-f428-6b39-117efda3379f-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org> 2017-06-27 16:28 ` Jason Gunthorpe 2017-06-27 16:28 ` Jason Gunthorpe [not found] ` <20170627162800.GA22592-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org> 2017-06-28 7:03 ` Sagi Grimberg 2017-06-28 7:03 ` Sagi Grimberg 2017-06-27 16:28 ` Chuck Lever 2017-06-27 16:28 ` Chuck Lever [not found] ` <9990B5CB-E0FF-481E-9F34-21EACF0E796E-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org> 2017-06-28 7:08 ` Sagi Grimberg 2017-06-28 7:08 ` Sagi Grimberg [not found] ` <f1f1a68c-90db-e6bf-e35e-55c4b469c339-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org> 2017-06-28 16:11 ` Chuck Lever 2017-06-28 16:11 ` Chuck Lever [not found] ` <7D1C540B-FEA0-4101-8B58-87BCB7DB5492-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org> 2017-06-29 5:35 ` Sagi Grimberg 2017-06-29 5:35 ` Sagi Grimberg [not found] ` <66b1b8be-e506-50b8-c01f-fa0e3cea98a4-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org> 2017-06-29 14:55 ` Chuck Lever 2017-06-29 14:55 ` Chuck Lever [not found] ` <9D8C7BC8-7E18-405A-9017-9DB23A6B5C15-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org> 2017-07-02 9:45 ` Sagi Grimberg 2017-07-02 9:45 ` Sagi Grimberg [not found] ` <11aa1a24-9f0b-dbb8-18eb-ad357c7727b2-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org> 2017-07-02 18:17 ` Chuck Lever 2017-07-02 18:17 ` Chuck Lever [not found] ` <9E30754F-464A-4B62-ADE7-F6B2F6D95763-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org> 2017-07-09 16:47 ` Jason Gunthorpe 2017-07-09 16:47 ` Jason Gunthorpe [not found] ` <20170709164755.GB3058-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org> 2017-07-10 19:03 ` Chuck Lever 2017-07-10 19:03 ` Chuck Lever [not found] ` <A7C8C159-E916-4060-9FD1-8726D816B3C0-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org> 2017-07-10 20:05 ` Jason Gunthorpe 2017-07-10 20:05 ` Jason Gunthorpe [not found] ` <20170710200522.GA19293-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org> 2017-07-10 20:51 ` Chuck Lever 2017-07-10 20:51 ` Chuck Lever [not found] ` <C04891DF-5B3B-4156-9E04-9E18B238864A-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org> 2017-07-10 21:14 ` Jason Gunthorpe 2017-07-10 21:14 ` Jason Gunthorpe 2017-07-10 21:24 ` Jason Gunthorpe 2017-07-10 21:24 ` Jason Gunthorpe [not found] ` <20170710212430.GA21721-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org> 2017-07-10 21:29 ` Chuck Lever 2017-07-10 21:29 ` Chuck Lever [not found] ` <C142385D-3A54-44C9-BA3D-0AABBC5E9E7B-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org> 2017-07-10 21:32 ` Jason Gunthorpe 2017-07-10 21:32 ` Jason Gunthorpe [not found] ` <20170710213251.GA21908-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org> 2017-07-10 22:04 ` Chuck Lever 2017-07-10 22:04 ` Chuck Lever [not found] ` <A342254B-1ECB-4644-8D68-328A52940C52-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org> 2017-07-10 22:09 ` Jason Gunthorpe 2017-07-10 22:09 ` Jason Gunthorpe [not found] ` <20170710220905.GA22589-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org> 2017-07-11 3:57 ` Chuck Lever 2017-07-11 3:57 ` Chuck Lever [not found] ` <718A099D-3597-4262-9A33-0BA7EFE5461F-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org> 2017-07-11 13:23 ` Tom Talpey [this message] 2017-07-11 13:23 ` Tom Talpey [not found] ` <8dd77b19-3846-96ea-f50f-22182989e941-CLs1Zie5N5HQT0dZR+AlfA@public.gmane.org> 2017-07-11 14:55 ` Chuck Lever 2017-07-11 14:55 ` Chuck Lever 2017-06-27 18:08 ` Bart Van Assche 2017-06-27 18:08 ` Bart Van Assche [not found] ` <1498586933.14963.1.camel-Sjgp3cTcYWE@public.gmane.org> 2017-06-27 18:14 ` Jason Gunthorpe 2017-06-27 18:14 ` Jason Gunthorpe 2017-06-28 7:16 ` Sagi Grimberg 2017-06-28 7:16 ` Sagi Grimberg [not found] ` <47bbf598-6c82-610f-dc1d-706a1d869b8d-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org> 2017-06-28 9:43 ` Bart Van Assche 2017-06-28 9:43 ` Bart Van Assche 2017-06-20 17:08 ` Robert LeBlanc 2017-06-20 17:08 ` Robert LeBlanc [not found] ` <CAANLjFrpHzapvqgBajUu7QpgNNPvNjThMZrsXGcKt58E+6siMA-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org> 2017-06-20 17:19 ` Sagi Grimberg 2017-06-20 17:19 ` Sagi Grimberg [not found] ` <3f3830eb-68c5-f862-58c7-7021e6462f6f-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org> 2017-06-20 17:28 ` Robert LeBlanc 2017-06-20 17:28 ` Robert LeBlanc [not found] ` <CAANLjFo_1E5nPkXMmG0VxtXpQ2m6=WNd0OtvG4rd-__1TD0-VQ-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org> 2017-06-27 7:22 ` Sagi Grimberg 2017-06-27 7:22 ` Sagi Grimberg 2017-06-20 14:43 ` Robert LeBlanc 2017-06-20 14:43 ` Robert LeBlanc 2017-06-20 14:41 ` Robert LeBlanc 2017-06-20 14:41 ` Robert LeBlanc [not found] ` <1554c1d1-6bf4-9ca2-12d4-a0125d8c5715-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> 2017-02-27 20:13 ` Sagi Grimberg 2017-02-27 20:13 ` Sagi Grimberg
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to=8dd77b19-3846-96ea-f50f-22182989e941@talpey.com \ --to=tom-cls1zie5n5hqt0dzr+alfa@public.gmane.org \ --cc=bart.vanassche-XdAiOPVOjttBDgjK7y7TUQ@public.gmane.org \ --cc=chuck.lever-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org \ --cc=hch-wEGCiKHe2LqWVfeAwA7xHQ@public.gmane.org \ --cc=jgunthorpe-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org \ --cc=joseph.r.gruher-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org \ --cc=leon-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org \ --cc=linux-nvme-IAPFreCvJWM7uuMidbF8XUB+6BGkLq7r@public.gmane.org \ --cc=linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org \ --cc=liranl-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org \ --cc=loberman-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org \ --cc=maxg-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org \ --cc=mrybczyn-FNhOzJFKnXGHXe+LvDLADg@public.gmane.org \ --cc=robert-4JaGZRWAfWbajFs6igw21g@public.gmane.org \ --cc=robert.m.riches.jr-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org \ --cc=sagi-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org \ --cc=shahar.salzman-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: linkBe sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.