From: Sagi Grimberg <sagi-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org> To: Chuck Lever <chuck.lever-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org> Cc: Leon Romanovsky <leon-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org>, Robert LeBlanc <robert-4JaGZRWAfWbajFs6igw21g@public.gmane.org>, Marta Rybczynska <mrybczyn-FNhOzJFKnXGHXe+LvDLADg@public.gmane.org>, Max Gurtovoy <maxg-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>, Christoph Hellwig <hch-wEGCiKHe2LqWVfeAwA7xHQ@public.gmane.org>, "Gruher, Joseph R" <joseph.r.gruher-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>, "shahar.salzman" <shahar.salzman-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>, Laurence Oberman <loberman-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>, "Riches Jr, Robert M" <robert.m.riches.jr-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>, linux-rdma <linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org>, linux-nvme-IAPFreCvJWM7uuMidbF8XUB+6BGkLq7r@public.gmane.org, Jason Gunthorpe <jgunthorpe-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org>, Liran Liss <liranl-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>, Bart Van Assche <bart.vanassche-XdAiOPVOjttBDgjK7y7TUQ@public.gmane.org> Subject: Re: Unexpected issues with 2 NVME initiators using the same target Date: Tue, 20 Jun 2017 20:12:50 +0300 [thread overview] Message-ID: <5f38efc7-af8a-c27e-37ae-8f8aed3d43a4@grimberg.me> (raw) In-Reply-To: <59FF0C04-2BFB-4F66-81BA-A598A9A087FC-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org> > So on occasion there is a Remote Access Error. That would > trigger connection loss, and the retransmitted Send request > is discarded (if there was externally exposed memory involved > with the original transaction that is now invalid). I'm actually not concerned about the remote invalidation, that is good that its discarded or failed. Its the inline sends that are a bug here. > But the real problem is preventing retransmitted Sends from > causing a ULP request to be executed multiple times. Exactly. >> Signalling all send completions and also finishing I/Os only after we >> got them will add latency, and that sucks... > > Typically, Sends will complete before the response arrives. > The additional cost will be handling extra interrupts, IMO. Not quite, heavy traffic _can_ results in dropped acks, my gut feeling is that it can happen more than we suspect. and yea, extra interrupt, extra cachelines, extra state, but I do not see any other way around it. > With FRWR, won't subsequent WRs be delayed until the HCA is > done with the Send? I don't think a signal is necessary in > every case. Send Queue accounting currently relies on that. Not really, the Send after the FRWR might have a fence (not strong ordered one) and CX3/CX4 strong order FRWR so for them that is a non-issue. The problem is that ULPs can't rely on it. > RPC-over-RDMA relies on the completion of Local Invalidation > to ensure that the initial Send WR is complete. Wait, is that guaranteed? > For Remote > Invalidation and pure inline, there is nothing to fence that > Send. > > The question I have is: how often do these Send retransmissions > occur? Is it enough to have a robust recovery mechanism, or > do we have to wire in assumptions about retransmission to > every Send operation? Even if its rare, we don't have any way to protect against devices retrying the send operation. -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo info at http://vger.kernel.org/majordomo-info.html
WARNING: multiple messages have this Message-ID (diff)
From: sagi@grimberg.me (Sagi Grimberg) Subject: Unexpected issues with 2 NVME initiators using the same target Date: Tue, 20 Jun 2017 20:12:50 +0300 [thread overview] Message-ID: <5f38efc7-af8a-c27e-37ae-8f8aed3d43a4@grimberg.me> (raw) In-Reply-To: <59FF0C04-2BFB-4F66-81BA-A598A9A087FC@oracle.com> > So on occasion there is a Remote Access Error. That would > trigger connection loss, and the retransmitted Send request > is discarded (if there was externally exposed memory involved > with the original transaction that is now invalid). I'm actually not concerned about the remote invalidation, that is good that its discarded or failed. Its the inline sends that are a bug here. > But the real problem is preventing retransmitted Sends from > causing a ULP request to be executed multiple times. Exactly. >> Signalling all send completions and also finishing I/Os only after we >> got them will add latency, and that sucks... > > Typically, Sends will complete before the response arrives. > The additional cost will be handling extra interrupts, IMO. Not quite, heavy traffic _can_ results in dropped acks, my gut feeling is that it can happen more than we suspect. and yea, extra interrupt, extra cachelines, extra state, but I do not see any other way around it. > With FRWR, won't subsequent WRs be delayed until the HCA is > done with the Send? I don't think a signal is necessary in > every case. Send Queue accounting currently relies on that. Not really, the Send after the FRWR might have a fence (not strong ordered one) and CX3/CX4 strong order FRWR so for them that is a non-issue. The problem is that ULPs can't rely on it. > RPC-over-RDMA relies on the completion of Local Invalidation > to ensure that the initial Send WR is complete. Wait, is that guaranteed? > For Remote > Invalidation and pure inline, there is nothing to fence that > Send. > > The question I have is: how often do these Send retransmissions > occur? Is it enough to have a robust recovery mechanism, or > do we have to wire in assumptions about retransmission to > every Send operation? Even if its rare, we don't have any way to protect against devices retrying the send operation.
next prev parent reply other threads:[~2017-06-20 17:12 UTC|newest] Thread overview: 171+ messages / expand[flat|nested] mbox.gz Atom feed top 2017-02-21 19:38 Unexpected issues with 2 NVME initiators using the same target shahar.salzman [not found] ` <08131a05-1f56-ef61-990a-7fff04eea095-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> 2017-02-21 22:50 ` Sagi Grimberg 2017-02-21 22:50 ` Sagi Grimberg [not found] ` <de1a559a-bf24-0d73-5fc7-148d6cd4d4e0-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org> 2017-02-22 16:52 ` Laurence Oberman 2017-02-22 16:52 ` Laurence Oberman [not found] ` <1848296658.37025722.1487782361271.JavaMail.zimbra-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org> 2017-02-22 19:39 ` Sagi Grimberg 2017-02-22 19:39 ` Sagi Grimberg 2017-02-26 8:03 ` shahar.salzman 2017-02-26 8:03 ` shahar.salzman 2017-02-26 17:58 ` Gruher, Joseph R [not found] ` <DE927C68B458BE418D582EC97927A92854655137-8oqHQFITsIFcIJlls4ac1rfspsVTdybXVpNB7YpNyf8@public.gmane.org> 2017-02-27 20:33 ` Sagi Grimberg 2017-02-27 20:33 ` Sagi Grimberg 2017-02-27 20:57 ` Gruher, Joseph R 2017-03-05 18:23 ` Leon Romanovsky 2017-03-06 0:07 ` Max Gurtovoy [not found] ` <26912d0c-578f-26e9-490d-94fc95bdf259-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org> 2017-03-06 11:28 ` Sagi Grimberg 2017-03-06 11:28 ` Sagi Grimberg 2017-03-07 9:27 ` Max Gurtovoy [not found] ` <fbd647dd-3a16-8155-107d-f98e8326cc63-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org> 2017-03-07 13:41 ` Sagi Grimberg 2017-03-07 13:41 ` Sagi Grimberg [not found] ` <c5a6f55b-a633-53ca-4378-7a7eaf7f77bd-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org> 2017-03-09 12:18 ` shahar.salzman 2017-03-09 12:18 ` shahar.salzman 2017-03-12 12:33 ` Vladimir Neyelov [not found] ` <AM4PR0501MB278621363209E177A738D75FCB220-dp/nxUn679hhbxXPg6FtWcDSnupUy6xnnBOFsp37pqbUKgpGm//BTAC/G2K4zDHf@public.gmane.org> 2017-03-13 9:43 ` Sagi Grimberg 2017-03-13 9:43 ` Sagi Grimberg 2017-03-14 8:55 ` Max Gurtovoy 2017-03-14 19:57 ` Gruher, Joseph R 2017-03-14 23:42 ` Gruher, Joseph R 2017-03-16 0:03 ` Gruher, Joseph R 2017-03-17 18:37 ` Gruher, Joseph R 2017-03-17 19:49 ` Max Gurtovoy [not found] ` <DE927C68B458BE418D582EC97927A928550391C2@ORSMSX113.amr.corp.intel.com> 2017-03-24 18:30 ` Gruher, Joseph R 2017-03-27 14:17 ` Max Gurtovoy 2017-03-27 15:39 ` Gruher, Joseph R 2017-03-28 8:38 ` Max Gurtovoy 2017-03-28 10:21 ` shahar.salzman [not found] ` <DE927C68B458BE418D582EC97927A928550419FA-8oqHQFITsIFQxe9IK+vIArfspsVTdybXVpNB7YpNyf8@public.gmane.org> 2017-03-28 11:34 ` Sagi Grimberg 2017-03-28 11:34 ` Sagi Grimberg [not found] ` <809f87ab-b787-9d40-5840-07500d12e81a-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org> 2017-04-10 11:40 ` Marta Rybczynska 2017-04-10 11:40 ` Marta Rybczynska 2017-04-10 14:09 ` Max Gurtovoy [not found] ` <33e2cc35-f147-d4a4-9a42-8f1245e35842-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org> 2017-04-11 12:47 ` Marta Rybczynska 2017-04-11 12:47 ` Marta Rybczynska 2017-04-20 10:18 ` Sagi Grimberg 2017-04-20 10:18 ` Sagi Grimberg 2017-04-26 11:56 ` Max Gurtovoy [not found] ` <af9044c2-fb7c-e8b3-d8fc-4874cfd1bb67-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org> 2017-04-26 14:45 ` Sagi Grimberg 2017-04-26 14:45 ` Sagi Grimberg 2017-05-12 19:20 ` Gruher, Joseph R [not found] ` <DE927C68B458BE418D582EC97927A92855088C6F-8oqHQFITsIFQxe9IK+vIArfspsVTdybXVpNB7YpNyf8@public.gmane.org> 2017-05-15 12:00 ` Sagi Grimberg 2017-05-15 12:00 ` Sagi Grimberg [not found] ` <82dd5b24-5657-ae5e-8a33-646fddd8b75b-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org> 2017-05-15 13:31 ` Leon Romanovsky 2017-05-15 13:31 ` Leon Romanovsky [not found] ` <20170515133122.GG3616-U/DQcQFIOTAAJjI8aNfphQ@public.gmane.org> 2017-05-15 13:43 ` Sagi Grimberg 2017-05-15 13:43 ` Sagi Grimberg [not found] ` <9465cd0c-83db-b058-7615-5626ef60dbb0-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org> 2017-05-15 14:36 ` Leon Romanovsky 2017-05-15 14:36 ` Leon Romanovsky [not found] ` <20170515143632.GH3616-U/DQcQFIOTAAJjI8aNfphQ@public.gmane.org> 2017-05-15 14:59 ` Christoph Hellwig 2017-05-15 14:59 ` Christoph Hellwig [not found] ` <20170515145952.GA7871-wEGCiKHe2LqWVfeAwA7xHQ@public.gmane.org> 2017-05-15 17:05 ` Leon Romanovsky 2017-05-15 17:05 ` Leon Romanovsky [not found] ` <20170515170506.GK3616-U/DQcQFIOTAAJjI8aNfphQ@public.gmane.org> 2017-05-17 12:56 ` Marta Rybczynska 2017-05-17 12:56 ` Marta Rybczynska [not found] ` <779753075.36035391.1495025796237.JavaMail.zimbra-FNhOzJFKnXGHXe+LvDLADg@public.gmane.org> 2017-05-18 13:34 ` Leon Romanovsky 2017-05-18 13:34 ` Leon Romanovsky [not found] ` <20170518133439.GD3616-U/DQcQFIOTAAJjI8aNfphQ@public.gmane.org> 2017-06-19 17:21 ` Robert LeBlanc 2017-06-19 17:21 ` Robert LeBlanc [not found] ` <CAANLjFrCLpX3nb3q7LpFPpLJKciU+1Hvmt_hxyTovQJM2-zQmg-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org> 2017-06-20 6:39 ` Sagi Grimberg 2017-06-20 6:39 ` Sagi Grimberg [not found] ` <6073e553-e8c2-6d14-ba5d-c2bd5aff15eb-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org> 2017-06-20 7:46 ` Leon Romanovsky 2017-06-20 7:46 ` Leon Romanovsky [not found] ` <20170620074639.GP17846-U/DQcQFIOTAAJjI8aNfphQ@public.gmane.org> 2017-06-20 7:58 ` Sagi Grimberg 2017-06-20 7:58 ` Sagi Grimberg [not found] ` <1c706958-992e-b104-6bae-4a6616c0a9f9-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org> 2017-06-20 8:33 ` Leon Romanovsky 2017-06-20 8:33 ` Leon Romanovsky [not found] ` <20170620083309.GQ17846-U/DQcQFIOTAAJjI8aNfphQ@public.gmane.org> 2017-06-20 9:33 ` Sagi Grimberg 2017-06-20 9:33 ` Sagi Grimberg 2017-06-20 10:31 ` Max Gurtovoy [not found] ` <78b2c1db-6ece-0274-c4c9-5ee1f7c88469-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org> 2017-06-20 22:58 ` Robert LeBlanc 2017-06-20 22:58 ` Robert LeBlanc 2017-06-27 7:16 ` Sagi Grimberg 2017-06-27 7:16 ` Sagi Grimberg [not found] ` <bd0b986f-9bed-3dfa-7454-0661559a527b-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org> 2017-06-20 12:02 ` Sagi Grimberg 2017-06-20 12:02 ` Sagi Grimberg 2017-06-20 13:28 ` Max Gurtovoy [not found] ` <614481c7-22dd-d93b-e97e-52f868727ec3-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org> 2017-06-20 17:01 ` Chuck Lever 2017-06-20 17:01 ` Chuck Lever [not found] ` <59FF0C04-2BFB-4F66-81BA-A598A9A087FC-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org> 2017-06-20 17:12 ` Sagi Grimberg [this message] 2017-06-20 17:12 ` Sagi Grimberg 2017-06-20 17:35 ` Jason Gunthorpe 2017-06-20 17:35 ` Jason Gunthorpe [not found] ` <20170620173532.GA827-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org> 2017-06-20 18:17 ` Chuck Lever 2017-06-20 18:17 ` Chuck Lever [not found] ` <D3DC49A2-FFC9-4F62-8876-3E6AD5167DE5-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org> 2017-06-20 19:27 ` Jason Gunthorpe 2017-06-20 19:27 ` Jason Gunthorpe [not found] ` <20170620192742.GB827-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org> 2017-06-20 20:56 ` Chuck Lever 2017-06-20 20:56 ` Chuck Lever [not found] ` <C14B071E-F1B2-466A-82CF-4E20BFAD9DC1-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org> 2017-06-20 21:19 ` Jason Gunthorpe 2017-06-20 21:19 ` Jason Gunthorpe [not found] ` <20170620211958.GA5574-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org> 2017-06-27 7:37 ` Sagi Grimberg 2017-06-27 7:37 ` Sagi Grimberg [not found] ` <4f0812f1-0067-4e63-e383-b913ee1f319d-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org> 2017-06-27 14:42 ` Chuck Lever 2017-06-27 14:42 ` Chuck Lever [not found] ` <28F6F58E-B6F4-4114-8DFF-B72353CE814B-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org> 2017-06-27 16:07 ` Sagi Grimberg 2017-06-27 16:07 ` Sagi Grimberg [not found] ` <52ad3547-efcf-f428-6b39-117efda3379f-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org> 2017-06-27 16:28 ` Jason Gunthorpe 2017-06-27 16:28 ` Jason Gunthorpe [not found] ` <20170627162800.GA22592-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org> 2017-06-28 7:03 ` Sagi Grimberg 2017-06-28 7:03 ` Sagi Grimberg 2017-06-27 16:28 ` Chuck Lever 2017-06-27 16:28 ` Chuck Lever [not found] ` <9990B5CB-E0FF-481E-9F34-21EACF0E796E-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org> 2017-06-28 7:08 ` Sagi Grimberg 2017-06-28 7:08 ` Sagi Grimberg [not found] ` <f1f1a68c-90db-e6bf-e35e-55c4b469c339-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org> 2017-06-28 16:11 ` Chuck Lever 2017-06-28 16:11 ` Chuck Lever [not found] ` <7D1C540B-FEA0-4101-8B58-87BCB7DB5492-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org> 2017-06-29 5:35 ` Sagi Grimberg 2017-06-29 5:35 ` Sagi Grimberg [not found] ` <66b1b8be-e506-50b8-c01f-fa0e3cea98a4-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org> 2017-06-29 14:55 ` Chuck Lever 2017-06-29 14:55 ` Chuck Lever [not found] ` <9D8C7BC8-7E18-405A-9017-9DB23A6B5C15-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org> 2017-07-02 9:45 ` Sagi Grimberg 2017-07-02 9:45 ` Sagi Grimberg [not found] ` <11aa1a24-9f0b-dbb8-18eb-ad357c7727b2-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org> 2017-07-02 18:17 ` Chuck Lever 2017-07-02 18:17 ` Chuck Lever [not found] ` <9E30754F-464A-4B62-ADE7-F6B2F6D95763-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org> 2017-07-09 16:47 ` Jason Gunthorpe 2017-07-09 16:47 ` Jason Gunthorpe [not found] ` <20170709164755.GB3058-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org> 2017-07-10 19:03 ` Chuck Lever 2017-07-10 19:03 ` Chuck Lever [not found] ` <A7C8C159-E916-4060-9FD1-8726D816B3C0-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org> 2017-07-10 20:05 ` Jason Gunthorpe 2017-07-10 20:05 ` Jason Gunthorpe [not found] ` <20170710200522.GA19293-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org> 2017-07-10 20:51 ` Chuck Lever 2017-07-10 20:51 ` Chuck Lever [not found] ` <C04891DF-5B3B-4156-9E04-9E18B238864A-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org> 2017-07-10 21:14 ` Jason Gunthorpe 2017-07-10 21:14 ` Jason Gunthorpe 2017-07-10 21:24 ` Jason Gunthorpe 2017-07-10 21:24 ` Jason Gunthorpe [not found] ` <20170710212430.GA21721-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org> 2017-07-10 21:29 ` Chuck Lever 2017-07-10 21:29 ` Chuck Lever [not found] ` <C142385D-3A54-44C9-BA3D-0AABBC5E9E7B-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org> 2017-07-10 21:32 ` Jason Gunthorpe 2017-07-10 21:32 ` Jason Gunthorpe [not found] ` <20170710213251.GA21908-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org> 2017-07-10 22:04 ` Chuck Lever 2017-07-10 22:04 ` Chuck Lever [not found] ` <A342254B-1ECB-4644-8D68-328A52940C52-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org> 2017-07-10 22:09 ` Jason Gunthorpe 2017-07-10 22:09 ` Jason Gunthorpe [not found] ` <20170710220905.GA22589-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org> 2017-07-11 3:57 ` Chuck Lever 2017-07-11 3:57 ` Chuck Lever [not found] ` <718A099D-3597-4262-9A33-0BA7EFE5461F-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org> 2017-07-11 13:23 ` Tom Talpey 2017-07-11 13:23 ` Tom Talpey [not found] ` <8dd77b19-3846-96ea-f50f-22182989e941-CLs1Zie5N5HQT0dZR+AlfA@public.gmane.org> 2017-07-11 14:55 ` Chuck Lever 2017-07-11 14:55 ` Chuck Lever 2017-06-27 18:08 ` Bart Van Assche 2017-06-27 18:08 ` Bart Van Assche [not found] ` <1498586933.14963.1.camel-Sjgp3cTcYWE@public.gmane.org> 2017-06-27 18:14 ` Jason Gunthorpe 2017-06-27 18:14 ` Jason Gunthorpe 2017-06-28 7:16 ` Sagi Grimberg 2017-06-28 7:16 ` Sagi Grimberg [not found] ` <47bbf598-6c82-610f-dc1d-706a1d869b8d-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org> 2017-06-28 9:43 ` Bart Van Assche 2017-06-28 9:43 ` Bart Van Assche 2017-06-20 17:08 ` Robert LeBlanc 2017-06-20 17:08 ` Robert LeBlanc [not found] ` <CAANLjFrpHzapvqgBajUu7QpgNNPvNjThMZrsXGcKt58E+6siMA-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org> 2017-06-20 17:19 ` Sagi Grimberg 2017-06-20 17:19 ` Sagi Grimberg [not found] ` <3f3830eb-68c5-f862-58c7-7021e6462f6f-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org> 2017-06-20 17:28 ` Robert LeBlanc 2017-06-20 17:28 ` Robert LeBlanc [not found] ` <CAANLjFo_1E5nPkXMmG0VxtXpQ2m6=WNd0OtvG4rd-__1TD0-VQ-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org> 2017-06-27 7:22 ` Sagi Grimberg 2017-06-27 7:22 ` Sagi Grimberg 2017-06-20 14:43 ` Robert LeBlanc 2017-06-20 14:43 ` Robert LeBlanc 2017-06-20 14:41 ` Robert LeBlanc 2017-06-20 14:41 ` Robert LeBlanc [not found] ` <1554c1d1-6bf4-9ca2-12d4-a0125d8c5715-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> 2017-02-27 20:13 ` Sagi Grimberg 2017-02-27 20:13 ` Sagi Grimberg
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to=5f38efc7-af8a-c27e-37ae-8f8aed3d43a4@grimberg.me \ --to=sagi-nqwnxtmzq1alnmji0ikvqw@public.gmane.org \ --cc=bart.vanassche-XdAiOPVOjttBDgjK7y7TUQ@public.gmane.org \ --cc=chuck.lever-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org \ --cc=hch-wEGCiKHe2LqWVfeAwA7xHQ@public.gmane.org \ --cc=jgunthorpe-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org \ --cc=joseph.r.gruher-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org \ --cc=leon-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org \ --cc=linux-nvme-IAPFreCvJWM7uuMidbF8XUB+6BGkLq7r@public.gmane.org \ --cc=linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org \ --cc=liranl-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org \ --cc=loberman-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org \ --cc=maxg-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org \ --cc=mrybczyn-FNhOzJFKnXGHXe+LvDLADg@public.gmane.org \ --cc=robert-4JaGZRWAfWbajFs6igw21g@public.gmane.org \ --cc=robert.m.riches.jr-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org \ --cc=shahar.salzman-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: linkBe sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.