[PATCH v2 0/3] public/io/netif.h: support for toeplitz hashing

[PATCH v2 0/3] public/io/netif.h: support for toeplitz hashing

[Paul Durrant]

This series documents changes needed to support toeplitz hashing in a
backend, configurable by the frontend.

Patch #1 is a clean-up patch that clarifies the guest transmit and
receive side packet formats.

Patch #2 documents a new 'control ring' for passing bulk data between
frontend and backend. This is needed for passing the hash mapping table
and hash key. It also documents messages to allow a frontend to configure
toeplitz hashing.

Patch #3 documents a new extra info segment that can be used for passing
hash values along with packets on both the transmit and receive side.

[PATCH v3 1/3] public/io/netif.h: document transmit and receive wire formats separately

[Paul Durrant]

Currently there is no documented wire format for guest receive-side
packets but the location of the 'wire format' comment block suggests
it is the same as transmit-side. This is almost true but there is a
subtle difference in the use of the 'size' field for the first fragment.

For clarity this patch creates separate comment blocks for receive
and transmit side packet wire formats, tries to be more clear about the
distinction between 'fragments' and 'extras', and documents the subtlety
concerning the size field of the first fragment.

Signed-off-by: Paul Durrant <paul.durrant@citrix.com>
Cc: Ian Campbell <ian.campbell@citrix.com>
Cc: Ian Jackson <ian.jackson@eu.citrix.com>
Cc: Jan Beulich <jbeulich@suse.com>
Cc: Keir Fraser <keir@xen.org>
Cc: Tim Deegan <tim@xen.org>
---
 xen/include/public/io/netif.h | 39 ++++++++++++++++++++++++++-------------
 1 file changed, 26 insertions(+), 13 deletions(-)

diff --git a/xen/include/public/io/netif.h b/xen/include/public/io/netif.h
index e103cf3..1790ea0 100644
--- a/xen/include/public/io/netif.h
+++ b/xen/include/public/io/netif.h
@@ -151,22 +151,22 @@
  */
 
 /*
- * This is the 'wire' format for packets:
- *  Request 1: netif_tx_request_t -- NETTXF_* (any flags)
- * [Request 2: netif_extra_info_t] (only if request 1 has
- *                                  NETTXF_extra_info)
- * [Request 3: netif_extra_info_t] (only if request 2 has
- *                                  XEN_NETIF_EXTRA_MORE)
- *  Request 4: netif_tx_request_t -- NETTXF_more_data
- *  Request 5: netif_tx_request_t -- NETTXF_more_data
- *  ...
- *  Request N: netif_tx_request_t -- 0
- */
-
-/*
  * Guest transmit
  * ==============
  *
+ * This is the 'wire' format for packets:
+ *  Fragment 1: netif_tx_request_t  - flags = NETTXF_*
+ *                                    size = total packet size
+ * [Extra 1: netif_extra_info_t]    - (only if fragment 1 flags include
+ *                                     NETTXF_extra_info)
+ * [Extra N: netif_extra_info_t]    - (only if extra N-1 flags include
+ *                                     XEN_NETIF_EXTRA_MORE)
+ *  ...
+ *  Fragment N: netif_tx_request_t  - (only if fragment N-1 flags include
+ *                                     NETTXF_more_data)
+ *                                    flags = 0
+ *                                    size = fragment size
+ *
  * Ring slot size is 12 octets, however not all request/response
  * structs use the full size.
  *
@@ -202,6 +202,19 @@
  * Guest receive
  * =============
  *
+ * This is the 'wire' format for packets:
+ *  Fragment 1: netif_rx_request_t  - flags = NETRXF_*
+ *                                    size = fragment size
+ * [Extra 1: netif_extra_info_t]    - (only if fragment 1 flags include
+ *                                     NETRXF_extra_info)
+ * [Extra N: netif_extra_info_t]    - (only if extra N-1 flags include
+ *                                     XEN_NETIF_EXTRA_MORE)
+ *  ...
+ *  Fragment N: netif_rx_request_t  - (only if fragment N-1 flags include
+ *                                     NETRXF_more_data)
+ *                                    flags = 0
+ *                                    size = fragment size
+ *
  * Ring slot size is 8 octets.
  *
  * rx request (netif_rx_request_t)
-- 
2.1.4

[PATCH v3 2/3] public/io/netif.h: document control ring and toeplitz hashing

[Paul Durrant]

This patch documents a new shared ring between frontend and backend that
can be used to pass bulk out-of-band data, such as that required to
implement toeplitz hashing in the backend such that it is configurable by
the frontend.

The patch then goes on to document the messages passed over the control
ring that can be used to configure toeplitz hashing.

Signed-off-by: Paul Durrant <paul.durrant@citrix.com>
Cc: Ian Campbell <ian.campbell@citrix.com>
Cc: Ian Jackson <ian.jackson@eu.citrix.com>
Cc: Jan Beulich <jbeulich@suse.com>
Cc: Keir Fraser <keir@xen.org>
Cc: Tim Deegan <tim@xen.org>
---
v3:
 - Fix commit comment

v2:
 - Use a balanced fix-sized message ring for the control ring
   (bulk data now passed by grant reference).
---
 xen/include/public/io/netif.h | 264 ++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 264 insertions(+)

diff --git a/xen/include/public/io/netif.h b/xen/include/public/io/netif.h
index 1790ea0..ace74f3 100644
--- a/xen/include/public/io/netif.h
+++ b/xen/include/public/io/netif.h
@@ -151,6 +151,270 @@
  */
 
 /*
+ * Control ring
+ * ============
+ *
+ * Some features, such as toeplitz hashing (detailed below), require a
+ * significant amount of out-of-band data to be passed from frontend to
+ * backend. Use of xenstore is not suitable for large quantities of data
+ * because of quota limitations and so a dedicated 'control ring' is used.
+ * The ability of the backend to use a control ring is advertised by
+ * setting:
+ *
+ * /local/domain/X/backend/<domid>/<vif>/feature-control-ring = "1"
+ *
+ * The frontend provides a control ring to the backend by setting:
+ *
+ * /local/domain/<domid>/device/vif/<vif>/ctrl-ring-ref = <gref>
+ * /local/domain/<domid>/device/vif/<vif>/event-channel-ctrl = <port>
+ *
+ * where <gref> is the grant reference of the shared page used to
+ * implement the control ring and <port> is an event channel to be used
+ * as a mailbox interrupt, before the frontend moves into the connected
+ * state.
+ *
+ * The control ring uses a fixed request/response message size and is
+ * balanced (i.e. one request to one response), so operationally it is much
+ * the same as a tramsmit or receive ring.
+ */
+
+/*
+ * Toeplitz hash types
+ * ===================
+ *
+ * For the purposes of the definitions below, 'Packet[]' is an array of
+ * octets containing an IP packet without options, 'Array[X..Y]' means a
+ * sub-array of 'Array' containing bytes X thru Y inclusive, and '+' is
+ * used to indicate concatenation of arrays.
+ */
+
+/*
+ * A hash calculated over an IP version 4 header as follows:
+ *
+ * Buffer[0..8] = Packet[12..15] + Packet[16..19]
+ * Result = ToeplitzHash(Buffer, 8)
+ */
+#define _NETIF_CTRL_TOEPLITZ_FLAG_IPV4     0
+#define NETIF_CTRL_TOEPLITZ_FLAG_IPV4      (1 << _NETIF_CTRL_TOEPLITZ_FLAG_IPV4)
+
+/*
+ * A hash calculated over an IP version 4 header and TCP header as
+ * follows:
+ *
+ * Buffer[0..12] = Packet[12..15] + Packet[16..19] +
+ *                 Packet[20..21] + Packet[22..23]
+ * Result = ToeplitzHash(Buffer, 12)
+ */
+#define _NETIF_CTRL_TOEPLITZ_FLAG_IPV4_TCP 1
+#define NETIF_CTRL_TOEPLITZ_FLAG_IPV4_TCP  (1 << _NETIF_CTRL_TOEPLITZ_FLAG_IPV4_TCP)
+
+/*
+ * A hash calculated over an IP version 6 header as follows:
+ *
+ * Buffer[0..32] = Packet[8..23] + Packet[24..39]
+ * Result = ToeplitzHash(Buffer, 32)
+ */
+#define _NETIF_CTRL_TOEPLITZ_FLAG_IPV6     2
+#define NETIF_CTRL_TOEPLITZ_FLAG_IPV6      (1 << _NETIF_CTRL_TOEPLITZ_FLAG_IPV4)
+
+/*
+ * A hash calculated over an IP version 6 header and TCP header as
+ * follows:
+ *
+ * Buffer[0..36] = Packet[8..23] + Packet[24..39] +
+ *                 Packet[40..41] + Packet[42..43]
+ * Result = ToeplitzHash(Buffer, 36)
+ */
+#define _NETIF_CTRL_TOEPLITZ_FLAG_IPV6_TCP 3
+#define NETIF_CTRL_TOEPLITZ_FLAG_IPV6_TCP  (1 << _NETIF_CTRL_TOEPLITZ_FLAG_IPV4_TCP)
+
+/*
+ * Control requests (netif_ctrl_request_t)
+ * =======================================
+ *
+ * All requests have the following format:
+ *
+ *    0     1     2     3     4     5     6     7  octet
+ * +-----+-----+-----+-----+-----+-----+-----+-----+
+ * |    id     |   type    |         data[0]       |
+ * +-----+-----+-----+-----+-----+-----+-----+-----+
+ * |         data[1]       |
+ * +-----+-----+-----+-----+
+ *
+ * id: the request identifier, echoed in response.
+ * type: the type of request (see below)
+ * data[]: any data associated with the request (determined by type)
+ */
+
+struct netif_ctrl_request {
+    uint16_t id;
+    uint16_t type;
+
+#define NETIF_CTRL_TYPE_INVALID              0
+#define NETIF_CTRL_TYPE_GET_TOEPLITZ_FLAGS   1
+#define NETIF_CTRL_TYPE_SET_TOEPLITZ_FLAGS   2
+#define NETIF_CTRL_TYPE_SET_TOEPLITZ_KEY     3
+#define NETIF_CTRL_TYPE_SET_TOEPLITZ_MAPPING 4
+
+    uint32_t data[2];
+};
+typedef struct netif_ctrl_request netif_ctrl_request_t;
+
+/*
+ * type = NETIF_CTRL_TYPE_GET_TOEPLITZ_FLAGS:
+ *
+ * This is sent by the frontend to query the types of toeplitz
+ * hash supported by the backend. No data is required and to the
+ * data[] field is set to 0.
+ *
+ * type = NETIF_CTRL_TYPE_SET_TOEPLITZ_FLAGS:
+ *
+ * This is sent by the frontend to set the types of toeplitz hash that
+ * the backend should calculate. Note that the 'maximal' type of hash
+ * should always be chosen. For example, if the frontend sets both IPV4
+ * and IPV4_TCP hash types then the latter hash type should be calculated
+ * for any TCP packet and the former only calculated for non-TCP packets.
+ * The data[0] field is a bitwise OR of NETIF_CTRL_TOEPLITZ_FLAG_* values
+ * defined above. The data[1] field is set to 0.
+ *
+ * NOTE: Setting data[0] to 0 disables toeplitz hashing and the backend
+ *       is free to choose how it steers packets to queues (which is the
+ *       default state).
+ *
+ * type = NETIF_CTRL_TYPE_SET_TOEPLITZ_KEY:
+ *
+ * This is sent by the frontend to set the key of toeplitz hash that
+ * the backend should calculate. The toeplitz algorithm is illustrated
+ * by the following pseudo-code:
+ *
+ * (Buffer[] and Key[] are treated as shift-registers where the MSB of
+ * Buffer/Key[0] is considered 'left-most' and the LSB of Buffer/Key[N-1]
+ * is the 'right-most').
+ *
+ * Value = 0
+ * For number of bits in Buffer[]
+ *    If (left-most bit of Buffer[] is 1)
+ *        Value ^= left-most 32 bits of Key[]
+ *    Key[] << 1
+ *    Buffer[] << 1
+ *
+ * The data[0] field is set to the size of key in octets. The data[1]
+ * field is set to a grant reference of a page containing the key. The
+ * reference must remain valid until the corresponding
+ * netif_ctrl_response_t has been processed.
+ *
+ * type = NETIF_CTRL_TYPE_SET_TOEPLITZ_MAPPING:
+ *
+ * This is sent by the frontend to set the mapping of toeplitz hash to
+ * queue number to be applied by the backend.
+ *
+ * The data[0] field is set to the order of the mapping. The data[1] field
+ * is set to a grant reference of a page containing the mapping. The
+ * reference must remain valid until the corresponding
+ * netif_ctrl_response_t has been processed.
+ *
+ * The format of the mapping is:
+ *
+ *    0     1     2     3     4     5     6     7  octet
+ * +-----+-----+-----+-----+-----+-----+-----+-----+
+ * |                    queue[0]                   |
+ * +-----+-----+-----+-----+-----+-----+-----+-----+
+ * |                    queue[1]                   |
+ * +-----+-----+-----+-----+-----+-----+-----+-----+
+ * |                    queue[2]                   |
+ * +-----+-----+-----+-----+-----+-----+-----+-----+
+ * |                    queue[3]                   |
+ *                         .
+ *                         .
+ * |                    queue[N-1]                 |
+ * +-----+-----+-----+-----+-----+-----+-----+-----+
+ *
+ * where each queue value is less than "multi-queue-num-queues" (see above)
+ * and N is 1 << data[0].
+ *
+ * NOTE: Before a specific mapping is set using this request, the backend
+ *       should map all toeplitz hash values to queue 0 (which is the only
+ *       queue guaranteed to exist in all cases).
+ */
+
+/*
+ * Control responses (netif_ctrl_response_t)
+ * =========================================
+ *
+ * All responses have the following format:
+ *
+ *    0     1     2     3     4     5     6     7  octet
+ * +-----+-----+-----+-----+-----+-----+-----+-----+
+ * |    id     |   pad     |         status        |
+ * +-----+-----+-----+-----+-----+-----+-----+-----+
+ * |         data          |
+ * +-----+-----+-----+-----+
+ *
+ * id: the corresponding request identifier
+ * pad: set to 0
+ * status: the status of request processing
+ * data: any data associated with the response (determined by type and
+ *       status)
+ */
+
+struct netif_ctrl_response {
+    uint16_t id;
+    uint16_t type;
+    uint32_t status;
+
+#define NETIF_CTRL_STATUS_SUCCESS           0
+#define NETIF_CTRL_STATUS_NOT_SUPPORTED     1
+#define NETIF_CTRL_STATUS_INVALID_PARAMETER 2
+#define NETIF_CTRL_STATUS_BUFFER_OVERFLOW   3
+
+    uint32_t data;
+};
+typedef struct netif_ctrl_response netif_ctrl_response_t;
+
+/*
+ * type = <unknown>
+ *
+ * The default response for any unrecognised request has the status field
+ * set to NETIF_CTRL_STATUS_NOT_SUPPORTED and the data field set to 0.
+ *
+ * type = NETIF_CTRL_MSG_GET_TOEPLITZ_FLAGS:
+ *
+ * Since the request carries no data there is no reason for processing to
+ * fail, hence the status field is set to NETIF_CTRL_STATUS_SUCCESS and the
+ * data field is a bitwise OR of NETIF_CTRL_TOEPLITZ_FLAG_* values (defined
+ * above) indicating which hash types are supported by the backend.
+ * If no hashing is supported then the data field should be set to 0.
+ *
+ * type = NETIF_CTRL_MSG_SET_TOEPLITZ_FLAGS:
+ *
+ * If the data[0] field in the request is invalid (i.e. contains unsupported
+ * hash types) then the status field is set to
+ * NETIF_CTRL_STATUS_INVALID_PARAMETER. Otherwise the requset should succeed
+ * and hence the status field is set to NETIF_CTRL_STATUS_SUCCESS.
+ * The data field should be set to 0.
+ *
+ * type = NETIF_CTRL_MSG_SET_TOEPLITZ_KEY:
+ *
+ * If the data[0] field in the request is an invalid key length (too big)
+ * then the status field is set to NETIF_CTRL_STATUS_BUFFER_OVERFLOW, If the
+ * data[1] field is an invalid grant reference then the status field is set
+ * to NETIF_CTRL_STATUS_INVALID_PARAMETER. Otherwise the request should
+ * succeed and hence the status field is set to NETIF_CTRL_STATUS_SUCCESS.
+ * The data field should be set to 0.
+ *
+ * type = NETIF_CTRL_MSG_SET_TOEPLITZ_MAPPING:
+ *
+ * If the data[0] field in the request is an invalid mapping order (too big)
+ * then the status field is set to NETIF_CTRL_STATUS_BUFFER_OVERFLOW, If the
+ * data[1] field is an invalid grant reference then the status field is set
+ * to NETIF_CTRL_STATUS_INVALID_PARAMETER. Otherwise the requset should
+ * succeed and hence the status field is set to NETIF_CTRL_STATUS_SUCCESS.
+ * The data field should be set to 0.
+ */
+
+DEFINE_RING_TYPES(netif_ctrl, struct netif_ctrl_request, struct netif_ctrl_response);
+
+/*
  * Guest transmit
  * ==============
  *
-- 
2.1.4

Re: [PATCH v3 2/3] public/io/netif.h: document control ring and toeplitz hashing

[Paul Durrant]

> -----Original Message-----
> From: Paul Durrant [mailto:paul.durrant@citrix.com]
> Sent: 06 January 2016 13:12
> To: xen-devel@lists.xenproject.org
> Cc: Paul Durrant; Ian Campbell; Ian Jackson; Jan Beulich; Keir (Xen.org); Tim
> (Xen.org)
> Subject: [PATCH v3 2/3] public/io/netif.h: document control ring and toeplitz
> hashing
> 
[snip]
> + * Control responses (netif_ctrl_response_t)
> + * =========================================
> + *
> + * All responses have the following format:
> + *
> + *    0     1     2     3     4     5     6     7  octet
> + * +-----+-----+-----+-----+-----+-----+-----+-----+
> + * |    id     |   pad     |         status        |
> + * +-----+-----+-----+-----+-----+-----+-----+-----+
> + * |         data          |
> + * +-----+-----+-----+-----+
> + *
> + * id: the corresponding request identifier
> + * pad: set to 0
> + * status: the status of request processing
> + * data: any data associated with the response (determined by type and
> + *       status)
> + */
> +
> +struct netif_ctrl_response {
> +    uint16_t id;
> +    uint16_t type;

Reading through this again, I noticed I forgot to update the struct definition here to match the comments above. I'll send v4.

  Paul

[PATCH v3 3/3] public/io/netif.h: document new extra info for passing hash values

[Paul Durrant]

To properly support NDIS RSS, the Windows frontend PV driver needs the
toeplitz hash value calculated by the backend (otherwise it would have to
duplicate the calculation).

This patch adds documentation for "feature-hash" and a definition of a
new XEN_NETIF_EXTRA_TYPE_HASH extra info segment which both frontends
and backends can use to pass hash values to the other end.

Signed-off-by: Paul Durrant <paul.durrant@citrix.com>
Cc: Ian Campbell <ian.campbell@citrix.com>
Cc: Ian Jackson <ian.jackson@eu.citrix.com>
Cc: Jan Beulich <jbeulich@suse.com>
Cc: Keir Fraser <keir@xen.org>
Cc: Tim Deegan <tim@xen.org>
---
 xen/include/public/io/netif.h | 36 +++++++++++++++++++++++++++++++++++-
 1 file changed, 35 insertions(+), 1 deletion(-)

diff --git a/xen/include/public/io/netif.h b/xen/include/public/io/netif.h
index ace74f3..e12c73f 100644
--- a/xen/include/public/io/netif.h
+++ b/xen/include/public/io/netif.h
@@ -151,6 +151,12 @@
  */
 
 /*
+ * "feature-hash" advertises the capability to accept extra info slots of
+ * type XEN_NETIF_EXTRA_TYPE_HASH. They will not be sent by either end
+ * unless the other end advertises this feature.
+ */
+
+/*
  * Control ring
  * ============
  *
@@ -574,6 +580,18 @@ DEFINE_RING_TYPES(netif_ctrl, struct netif_ctrl_request, struct netif_ctrl_respo
  * type: Must be XEN_NETIF_EXTRA_TYPE_MCAST_{ADD,DEL}
  * flags: XEN_NETIF_EXTRA_FLAG_*
  * addr: address to add/remove
+ *
+ * XEN_NETIF_EXTRA_TYPE_HASH:
+ *
+ *    0     1     2     3     4     5     6     7  octet
+ * +-----+-----+-----+-----+-----+-----+-----+-----+
+ * |type |flags|htype| pad |LSB ---- value ---- MSB|
+ * +-----+-----+-----+-----+-----+-----+-----+-----+
+ *
+ * type: Must be XEN_NETIF_EXTRA_TYPE_HASH
+ * flags: XEN_NETIF_EXTRA_FLAG_*
+ * htype: XEN_NETIF_HASH_TYPE_*
+ * value: Hash value
  */
 
 /* Protocol checksum field is blank in the packet (hardware offload)? */
@@ -607,7 +625,8 @@ typedef struct netif_tx_request netif_tx_request_t;
 #define XEN_NETIF_EXTRA_TYPE_GSO       (1)  /* u.gso */
 #define XEN_NETIF_EXTRA_TYPE_MCAST_ADD (2)  /* u.mcast */
 #define XEN_NETIF_EXTRA_TYPE_MCAST_DEL (3)  /* u.mcast */
-#define XEN_NETIF_EXTRA_TYPE_MAX       (4)
+#define XEN_NETIF_EXTRA_TYPE_HASH      (4)  /* u.hash */
+#define XEN_NETIF_EXTRA_TYPE_MAX       (5)
 
 /* netif_extra_info_t flags. */
 #define _XEN_NETIF_EXTRA_FLAG_MORE (0)
@@ -619,6 +638,16 @@ typedef struct netif_tx_request netif_tx_request_t;
 #define XEN_NETIF_GSO_TYPE_TCPV6        (2)
 
 /*
+ * Hash types. (See NETIF_CTRL_TOEPLITZ_FLAG_* definitions above
+ * for more information).
+ */
+#define XEN_NETIF_HASH_TYPE_NONE        0
+#define XEN_NETIF_HASH_TYPE_IPV4        1
+#define XEN_NETIF_HASH_TYPE_IPV4_TCP    2
+#define XEN_NETIF_HASH_TYPE_IPV6        3
+#define XEN_NETIF_HASH_TYPE_IPV6_TCP    4
+
+/*
  * This structure needs to fit within both netif_tx_request_t and
  * netif_rx_response_t for compatibility.
  */
@@ -635,6 +664,11 @@ struct netif_extra_info {
         struct {
             uint8_t addr[6];
         } mcast;
+        struct {
+            uint8_t type;
+            uint8_t pad;
+            uint8_t value[4];
+        } hash;
         uint16_t pad[3];
     } u;
 };
-- 
2.1.4