From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([2001:4830:134:3::10]:44004) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1allca-0008Ox-AT for qemu-devel@nongnu.org; Thu, 31 Mar 2016 19:03:29 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1allcX-0001l2-4B for qemu-devel@nongnu.org; Thu, 31 Mar 2016 19:03:24 -0400 Received: from mail.avalus.com ([89.16.176.221]:49560) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1allcW-0001kT-RT for qemu-devel@nongnu.org; Thu, 31 Mar 2016 19:03:21 -0400 From: Alex Bligh Date: Fri, 1 Apr 2016 00:03:19 +0100 Message-Id: <1459465399-56203-1-git-send-email-alex@alex.org.uk> Subject: [Qemu-devel] [PATCH] Improve documentation of FUA and FLUSH List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: Eric Blake , Wouter Verhelst Cc: "nbd-general@lists.sourceforge.net" , "qemu-devel@nongnu.org" , Alex Bligh Improve the documentation of NBD_CMD_FLUSH and NBD_CMD_FLAG_FUA. Specifically the latter may be set on any command, and its semantics on commands other than NBD_CMD_WRITE need explaining. Further, explain how these relate to reordering of commands. Signed-off-by: Alex Bligh --- doc/proto.md | 52 ++++++++++++++++++++++++++++++++++++++++++---------- 1 file changed, 42 insertions(+), 10 deletions(-) diff --git a/doc/proto.md b/doc/proto.md index c1e05c5..bc4483d 100644 --- a/doc/proto.md +++ b/doc/proto.md @@ -197,6 +197,37 @@ handle as was sent by the client in the corresponding request. In this way, the client can correlate which request is receiving a response. +#### Ordering of messages and writes + +The server MAY process commands out of order, and MAY reply out of +order, save that: + +* All write commands (that includes both `NBD_CMD_WRITE` and + `NBD_CMD_TRIM`) that the server completes (i.e. replies to) + prior to processing to a `NBD_CMD_FLUSH` MUST be written to non-volatile + storage prior to replying to that `NBD_CMD_FLUSH`. The server SHOULD ensure + that all write command received prior to processing the `NBD_CMD_FLUSH` + (whether they are replied to or not) are written to non-volatile + storage prior to processing an `NBD_CMD_FLUSH`; note this is a + stronger condition than the previous 'MUST' condition. This + paragram only applies if `NBD_FLAG_SEND_FLUSH` is set within + the transmission flags, as otherwise `NBD_CMD_FLUSH` will never + be sent by the client to the server. + +* A server MUST NOT reply to a command that has `NBD_CMD_FLAG_FUA` set + in its command flags until the data area referred to by that command + is persisted to non-volatile storage. This only applies if + `NBD_FLAG_SEND_FLUSH` is set within the transmission flags, as otherwise + `NBD_CMD_FLAG_FUA` will not be set on any commands sent to the server + by the client. + +`NBD_CMD_FLUSH` is modelled on the Linux kernel empty bio with +`REQ_FLUSH` set. `NBD_CMD_FLAG_FUA` is modelled on the Linux +kernel bio with `REQ_FUA` set. In case of ambiguity in this +specification, the +[kernel documentation](https://www.kernel.org/doc/Documentation/block/writeback_cache_control.txt) +may be useful. + #### Request message The request message, sent by the client, looks as follows: @@ -444,10 +475,17 @@ affects a particular command. Clients MUST NOT set a command flag bit that is not documented for the particular command; and whether a flag is valid may depend on negotiation during the handshake phase. -- bit 0, `NBD_CMD_FLAG_FUA`; valid during `NBD_CMD_WRITE`. SHOULD be - set to 1 if the client requires "Force Unit Access" mode of - operation. MUST NOT be set unless transmission flags included - `NBD_FLAG_SEND_FUA`. +- bit 0, `NBD_CMD_FLAG_FUA`. This bit + MUST be set to 0 unless the `NBD_FLAG_SEND_FUA` flag ("Force Unit Access") + was set in the transmission flags field. If the `NBD_FLAG_SEND_FUA` + is set in the transmission flags field, the client MAY set + `NBD_CMD_FLAG_FUA` in any request. If this bit is set, the server + MUST NOT send a reply until it has ensured that any data referred to + by this request (i.e. written data on a write or trim, read data on + a read) has reached permanent storage. There will be certain commands + (e.g. `NBD_CMD_DISC`) for which this flag will thus not alter behaviour + (as the command does not refer to any data), in which case the server + MUST ignore this bit. #### Request types @@ -479,12 +517,6 @@ The following request types exist: message. The server MAY send the reply message before the data has reached permanent storage. - If the `NBD_FLAG_SEND_FUA` flag ("Force Unit Access") was set in the - transmission flags field, the client MAY set the flag `NBD_CMD_FLAG_FUA` in - the command flags field. If this flag was set, the server MUST NOT send - the reply until it has ensured that the newly-written data has reached - permanent storage. - If an error occurs, the server SHOULD set the appropriate error code in the error field. The server MAY then close the connection. -- 1.9.1