* [PATCH 0/3] Add support for specific Lustre Qos
@ 2010-01-13 14:49 sebastien dugue
2010-01-13 14:51 ` [PATCH 1/3] rdma_cm: Add support for a new RDMA_PS_LUSTRE Lustre port space sebastien dugue
` (2 more replies)
0 siblings, 3 replies; 20+ messages in thread
From: sebastien dugue @ 2010-01-13 14:49 UTC (permalink / raw)
To: linux-rdma; +Cc: Roland Dreier, Sasha Khapyorsky
These patches allow to have a specific QoS for Lustre traffic.
- Patch 1 adds a new Lustre port space to the rdma_cm kernel module
- Patch 2 defines a new Lustre service ID (matching the new RDMA CMA port
space) and a new 'lustre' keyword in the OpenSM QoS policy parser.
- Patch 3 adds a new port_space parameter for the Lustre ko2iblnd kernel
module that allows to specify the port space to be used by LNET. This
patch has already been submitted to the Lustre community and is only
given here as a reference. The relevant discussion is at
https://bugzilla.lustre.org/show_bug.cgi?id=21732
In order the exercise QoS using this Lustre port space, the following setup
may be used:
1. OpenSM configuration - etc/opensm/opensm.conf
#
# QoS OPTIONS
#
# Enable QoS setup
qos TRUE
# QoS policy file to be used
qos_policy_file /etc/opensm/qospolicy.conf
# QoS default options
qos_max_vls 8
qos_high_limit 1
qos_vlarb_high 0:0,1:0,2:0,3:0,4:0,5:0
qos_vlarb_low 0:1,1:16,2:32,3:64,4:1,5:0
qos_sl2vl 0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15
The VLARB weights are only indicative.
2. OpenSM QoS policy configuration - /etc/opensm/qos-policy.conf:
qosulps
default : 0 # default SL
lustre, targetportguid 0x1234 : 1 # lustre traffic to a specific
# port guid (ex MDS traffic)
lustre : 2 # lustre default traffic
endqosulps
3. Lustre module configuration - /etc/modprobe.d/lustre.conf
Add the following line to all the Lustre nodes:
options ko2iblnd port_space=339
Sebastien.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 20+ messages in thread
* [PATCH 1/3] rdma_cm: Add support for a new RDMA_PS_LUSTRE Lustre port space
2010-01-13 14:49 [PATCH 0/3] Add support for specific Lustre Qos sebastien dugue
@ 2010-01-13 14:51 ` sebastien dugue
2010-01-13 16:56 ` Sean Hefty
2010-01-13 14:54 ` [PATCH 2/3] opensm/qos_policy: Add a new service ID and keyword for Lustre QoS sebastien dugue
2010-01-13 14:56 ` [PATCH 3/3] Add new port_space parameter to ko2iblnd module sebastien dugue
2 siblings, 1 reply; 20+ messages in thread
From: sebastien dugue @ 2010-01-13 14:51 UTC (permalink / raw)
To: linux-rdma; +Cc: Roland Dreier, Sasha Khapyorsky
This patch adds a new port space for use by Lustre traffic.
This, along with patches to OpenSM and Lustre, allow to define a
specific QoS for lustre.
Signed-off-by: Sebastien Dugue <sebastien.dugue-6ktuUTfB/bM@public.gmane.org>
---
drivers/infiniband/core/cma.c | 5 +++++
include/rdma/rdma_cm.h | 11 ++++++-----
2 files changed, 11 insertions(+), 5 deletions(-)
diff --git a/drivers/infiniband/core/cma.c b/drivers/infiniband/core/cma.c
index cc9b594..6b9e75e 100644
--- a/drivers/infiniband/core/cma.c
+++ b/drivers/infiniband/core/cma.c
@@ -78,6 +78,7 @@ static DEFINE_IDR(sdp_ps);
static DEFINE_IDR(tcp_ps);
static DEFINE_IDR(udp_ps);
static DEFINE_IDR(ipoib_ps);
+static DEFINE_IDR(lustre_ps);
static int next_port;
struct cma_device {
@@ -2066,6 +2067,9 @@ static int cma_get_port(struct rdma_id_private *id_priv)
case RDMA_PS_IPOIB:
ps = &ipoib_ps;
break;
+ case RDMA_PS_LUSTRE:
+ ps = &lustre_ps;
+ break;
default:
return -EPROTONOSUPPORT;
}
@@ -3034,6 +3038,7 @@ static void __exit cma_cleanup(void)
idr_destroy(&tcp_ps);
idr_destroy(&udp_ps);
idr_destroy(&ipoib_ps);
+ idr_destroy(&lustre_ps);
}
module_init(cma_init);
diff --git a/include/rdma/rdma_cm.h b/include/rdma/rdma_cm.h
index c6b2962..d662799 100644
--- a/include/rdma/rdma_cm.h
+++ b/include/rdma/rdma_cm.h
@@ -63,11 +63,12 @@ enum rdma_cm_event_type {
};
enum rdma_port_space {
- RDMA_PS_SDP = 0x0001,
- RDMA_PS_IPOIB = 0x0002,
- RDMA_PS_TCP = 0x0106,
- RDMA_PS_UDP = 0x0111,
- RDMA_PS_SCTP = 0x0183
+ RDMA_PS_SDP = 0x0001,
+ RDMA_PS_IPOIB = 0x0002,
+ RDMA_PS_TCP = 0x0106,
+ RDMA_PS_UDP = 0x0111,
+ RDMA_PS_LUSTRE = 0x0153,
+ RDMA_PS_SCTP = 0x0183
};
struct rdma_addr {
--
1.6.0.4
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply related [flat|nested] 20+ messages in thread
* [PATCH 2/3] opensm/qos_policy: Add a new service ID and keyword for Lustre QoS
2010-01-13 14:49 [PATCH 0/3] Add support for specific Lustre Qos sebastien dugue
2010-01-13 14:51 ` [PATCH 1/3] rdma_cm: Add support for a new RDMA_PS_LUSTRE Lustre port space sebastien dugue
@ 2010-01-13 14:54 ` sebastien dugue
2010-01-13 14:56 ` [PATCH 3/3] Add new port_space parameter to ko2iblnd module sebastien dugue
2 siblings, 0 replies; 20+ messages in thread
From: sebastien dugue @ 2010-01-13 14:54 UTC (permalink / raw)
To: linux-rdma; +Cc: Roland Dreier, Sasha Khapyorsky
This patch adds a new service ID for the Lustre ULP as well as the
lexer and parser bits for the policy definition simplified syntax.
The QoS policy parser now supports the following 'qos-ulps' keywords:
lustre : 0 #default SL for Lustre
lustre, target-port-guid 0x1234 : 1 #SL for Lustre with target port guid
lustre, port-num 10000-20000 : 2 #SL for Lustre port range
This, along with patches to the kernel rdma cma and Lustre, allow to define a
specific QoS for lustre.
Signed-off-by: Sebastien Dugue <sebastien.dugue-6ktuUTfB/bM@public.gmane.org>
---
opensm/include/opensm/osm_qos_policy.h | 11 ++--
opensm/opensm/osm_qos_parser_l.l | 8 +++
opensm/opensm/osm_qos_parser_y.y | 108 ++++++++++++++++++++++++++++++++
3 files changed, 122 insertions(+), 5 deletions(-)
diff --git a/opensm/include/opensm/osm_qos_policy.h b/opensm/include/opensm/osm_qos_policy.h
index 03ee891..ea3998a 100644
--- a/opensm/include/opensm/osm_qos_policy.h
+++ b/opensm/include/opensm/osm_qos_policy.h
@@ -54,11 +54,12 @@
#define OSM_QOS_POLICY_MAX_PORTS_ON_SWITCH 128
#define OSM_QOS_POLICY_DEFAULT_LEVEL_NAME "default"
-#define OSM_QOS_POLICY_ULP_SDP_SERVICE_ID 0x0000000000010000ULL
-#define OSM_QOS_POLICY_ULP_RDS_SERVICE_ID 0x0000000001060000ULL
-#define OSM_QOS_POLICY_ULP_RDS_PORT 0x48CA
-#define OSM_QOS_POLICY_ULP_ISER_SERVICE_ID 0x0000000001060000ULL
-#define OSM_QOS_POLICY_ULP_ISER_PORT 0x0CBC
+#define OSM_QOS_POLICY_ULP_SDP_SERVICE_ID 0x0000000000010000ULL
+#define OSM_QOS_POLICY_ULP_RDS_SERVICE_ID 0x0000000001060000ULL
+#define OSM_QOS_POLICY_ULP_RDS_PORT 0x48CA
+#define OSM_QOS_POLICY_ULP_ISER_SERVICE_ID 0x0000000001060000ULL
+#define OSM_QOS_POLICY_ULP_ISER_PORT 0x0CBC
+#define OSM_QOS_POLICY_ULP_LUSTRE_SERVICE_ID 0x0000000001530000ULL
#define OSM_QOS_POLICY_NODE_TYPE_CA (((uint8_t)1)<<IB_NODE_TYPE_CA)
#define OSM_QOS_POLICY_NODE_TYPE_SWITCH (((uint8_t)1)<<IB_NODE_TYPE_SWITCH)
diff --git a/opensm/opensm/osm_qos_parser_l.l b/opensm/opensm/osm_qos_parser_l.l
index 4633e0e..8f5d147 100644
--- a/opensm/opensm/osm_qos_parser_l.l
+++ b/opensm/opensm/osm_qos_parser_l.l
@@ -117,6 +117,9 @@ static void reset_new_line_flags();
#define START_ULP_SRP_GUID {in_list_of_num_ranges = TRUE;} /* comma-separated list of hex or dec num ranges */
#define START_ULP_IPOIB_DEFAULT {in_single_number = TRUE;} /* single number */
#define START_ULP_IPOIB_PKEY {in_list_of_num_ranges = TRUE;} /* comma-separated list of hex or dec num ranges */
+#define START_ULP_LUSTRE_DEFAULT {in_single_number = TRUE;} /* single number */
+#define START_ULP_LUSTRE_PORT {in_list_of_num_ranges = TRUE;} /* comma-separated list of hex or dec num ranges */
+#define START_ULP_LUSTRE_PORT_GUID {in_list_of_num_ranges = TRUE;} /* comma-separated list of hex or dec num ranges */
%}
@@ -187,6 +190,7 @@ ULP_SDP [Ss][Dd][Pp]
ULP_SRP [Ss][Rr][Pp]
ULP_RDS [Rr][Dd][Ss]
ULP_IPOIB [Ii][Pp][Oo][Ii][Bb]
+ULP_LUSTRE [Ll][Uu][Ss][Tt][Rr][Ee]
ULP_ISER [Ii][Ss][Ee][Rr]
ULP_ANY [Aa][Nn][Yy]
ULP_DEFAULT [Dd][Ee][Ff][Aa][Uu][Ll][Tt]
@@ -289,6 +293,10 @@ QUOTED_TEXT \"[^\"]*\"
{ULP_IPOIB}{WHITE_DOTDOT_WHITE} { SAVE_POS; HANDLE_IF_IN_DESCRIPTION; START_ULP_IPOIB_DEFAULT; return TK_ULP_IPOIB_DEFAULT; }
{ULP_IPOIB}{WHITE_COMMA_WHITE}{PKEY} { SAVE_POS; HANDLE_IF_IN_DESCRIPTION; START_ULP_IPOIB_PKEY; return TK_ULP_IPOIB_PKEY; }
+{ULP_LUSTRE}{WHITE_DOTDOT_WHITE} { SAVE_POS; HANDLE_IF_IN_DESCRIPTION; START_ULP_LUSTRE_DEFAULT; return TK_ULP_LUSTRE_DEFAULT; }
+{ULP_LUSTRE}{WHITE_COMMA_WHITE}{PORT_NUM} { SAVE_POS; HANDLE_IF_IN_DESCRIPTION; START_ULP_LUSTRE_PORT; return TK_ULP_LUSTRE_PORT; }
+{ULP_LUSTRE}{WHITE_COMMA_WHITE}{TARGET_PORT_GUID} { SAVE_POS; HANDLE_IF_IN_DESCRIPTION; START_ULP_LUSTRE_PORT_GUID; return TK_ULP_LUSTRE_PORT_GUID; }
+
0[xX][0-9a-fA-F]+ {
SAVE_POS;
yylval = strdup(yytext);
diff --git a/opensm/opensm/osm_qos_parser_y.y b/opensm/opensm/osm_qos_parser_y.y
index 7118b79..ccaac91 100644
--- a/opensm/opensm/osm_qos_parser_y.y
+++ b/opensm/opensm/osm_qos_parser_y.y
@@ -268,6 +268,9 @@ static cl_list_t __ulp_match_rules;
%token TK_ULP_SRP_GUID
%token TK_ULP_IPOIB_DEFAULT
%token TK_ULP_IPOIB_PKEY
+%token TK_ULP_LUSTRE_DEFAULT
+%token TK_ULP_LUSTRE_PORT
+%token TK_ULP_LUSTRE_PORT_GUID
%start head
@@ -302,6 +305,9 @@ qos_policy_entry: qos_ulps_section
* iser : 4 #default SL for iSER
* ipoib, pkey 0x0001 : 5 #SL for IPoIB on partition with pkey 0x0001
* ipoib : 6 #default IPoIB partition - pkey=0x7FFF
+ * lustre : 0 #default SL for Lustre
+ * lustre, port-num 10000-20000 : 2 #SL for Lustre port range
+ * lustre, target-port-guid 0x1234 : 3 #SL for Lustre with target port guid
* any, service-id 0x6234 : 2
* any, pkey 0x0ABC : 3
* any, target-port-guid 0x0ABC-0xFFFFF : 6
@@ -628,6 +634,9 @@ qos_match_rule_entry: qos_match_rule_use
* iser with port-num
* ipoib
* ipoib with pkey
+ * lustre
+ * lustre with port-num
+ * lustre with port-guid
* any with service-id
* any with pkey
* any with target-port-guid
@@ -951,6 +960,96 @@ qos_ulp: TK_ULP_DEFAULT single_number {
p_current_qos_match_rule->pkey_range_len = range_len;
} qos_ulp_sl
+
+ | qos_ulp_type_lustre_default {
+ /* "lustre : sl" - default SL for Lustre */
+ uint64_t ** range_arr =
+ (uint64_t **)malloc(sizeof(uint64_t *));
+ range_arr[0] = (uint64_t *)malloc(2*sizeof(uint64_t));
+ range_arr[0][0] = OSM_QOS_POLICY_ULP_LUSTRE_SERVICE_ID;
+ range_arr[0][1] = OSM_QOS_POLICY_ULP_LUSTRE_SERVICE_ID + 0xFFFF;
+
+ p_current_qos_match_rule->service_id_range_arr = range_arr;
+ p_current_qos_match_rule->service_id_range_len = 1;
+
+ } qos_ulp_sl
+
+ | qos_ulp_type_lustre_port list_of_ranges TK_DOTDOT {
+ /* Lustre with port numbers */
+ uint64_t ** range_arr;
+ unsigned range_len;
+ unsigned i;
+
+ if (!cl_list_count(&tmp_parser_struct.num_pair_list))
+ {
+ yyerror("Lustre ULP rule doesn't have port numbers");
+ return 1;
+ }
+
+ /* get all the port ranges */
+ __rangelist2rangearr( &tmp_parser_struct.num_pair_list,
+ &range_arr,
+ &range_len );
+ /* now translate these port numbers into service ids */
+ for (i = 0; i < range_len; i++)
+ {
+ if (range_arr[i][0] > 0xFFFF || range_arr[i][1] > 0xFFFF)
+ {
+ yyerror("Lustre port number out of range");
+ return 1;
+ }
+ range_arr[i][0] += OSM_QOS_POLICY_ULP_LUSTRE_SERVICE_ID;
+ range_arr[i][1] += OSM_QOS_POLICY_ULP_LUSTRE_SERVICE_ID;
+ }
+
+ p_current_qos_match_rule->service_id_range_arr = range_arr;
+ p_current_qos_match_rule->service_id_range_len = range_len;
+
+ } qos_ulp_sl
+
+ | qos_ulp_type_lustre_port_guid list_of_ranges TK_DOTDOT {
+ /* lustre, port-guid ... : sl */
+ uint64_t ** range_arr;
+ unsigned range_len;
+
+ if (!cl_list_count(&tmp_parser_struct.num_pair_list))
+ {
+ yyerror("Lustre ULP rule doesn't have port guids");
+ return 1;
+ }
+
+ /* create a new port group with these ports */
+ __parser_port_group_start();
+
+ p_current_port_group->name = strdup("_Lustre_Targets_");
+ p_current_port_group->use = strdup("Generated from ULP rules");
+
+ __rangelist2rangearr( &tmp_parser_struct.num_pair_list,
+ &range_arr,
+ &range_len );
+
+ __parser_add_guid_range_to_port_map(
+ &p_current_port_group->port_map,
+ range_arr,
+ range_len);
+
+ /* add this port group to the destination
+ groups of the current match rule */
+ cl_list_insert_tail(&p_current_qos_match_rule->destination_group_list,
+ p_current_port_group);
+
+ __parser_port_group_end();
+
+ /* setup ranges as in qos_ulp_type_lustre_default */
+ range_arr = (uint64_t **)malloc(sizeof(uint64_t *));
+ range_arr[0] = (uint64_t *)malloc(2*sizeof(uint64_t));
+ range_arr[0][0] = OSM_QOS_POLICY_ULP_LUSTRE_SERVICE_ID;
+ range_arr[0][1] = OSM_QOS_POLICY_ULP_LUSTRE_SERVICE_ID + 0xFFFF;
+
+ p_current_qos_match_rule->service_id_range_arr = range_arr;
+ p_current_qos_match_rule->service_id_range_len = 1;
+
+ } qos_ulp_sl
;
qos_ulp_type_any_service: TK_ULP_ANY_SERVICE_ID
@@ -989,6 +1088,15 @@ qos_ulp_type_ipoib_default: TK_ULP_IPOIB_DEFAULT
qos_ulp_type_ipoib_pkey: TK_ULP_IPOIB_PKEY
{ __parser_ulp_match_rule_start(); };
+qos_ulp_type_lustre_default: TK_ULP_LUSTRE_DEFAULT
+ { __parser_ulp_match_rule_start(); };
+
+qos_ulp_type_lustre_port: TK_ULP_LUSTRE_PORT
+ { __parser_ulp_match_rule_start(); };
+
+qos_ulp_type_lustre_port_guid: TK_ULP_LUSTRE_PORT_GUID
+ { __parser_ulp_match_rule_start(); };
+
qos_ulp_sl: single_number {
/* get the SL for ULP rules */
--
1.6.0.4
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply related [flat|nested] 20+ messages in thread
* [PATCH 3/3] Add new port_space parameter to ko2iblnd module
2010-01-13 14:49 [PATCH 0/3] Add support for specific Lustre Qos sebastien dugue
2010-01-13 14:51 ` [PATCH 1/3] rdma_cm: Add support for a new RDMA_PS_LUSTRE Lustre port space sebastien dugue
2010-01-13 14:54 ` [PATCH 2/3] opensm/qos_policy: Add a new service ID and keyword for Lustre QoS sebastien dugue
@ 2010-01-13 14:56 ` sebastien dugue
2 siblings, 0 replies; 20+ messages in thread
From: sebastien dugue @ 2010-01-13 14:56 UTC (permalink / raw)
To: linux-rdma; +Cc: Roland Dreier, Sasha Khapyorsky
This patch is only given as a reference.
Index: b/lnet/klnds/o2iblnd/o2iblnd.c
===================================================================
--- a/lnet/klnds/o2iblnd/o2iblnd.c
+++ b/lnet/klnds/o2iblnd/o2iblnd.c
@@ -2503,7 +2503,7 @@ kiblnd_startup (lnet_ni_t *ni)
ibdev->ibd_ifip = ip;
strcpy(&ibdev->ibd_ifname[0], ifname);
- id = rdma_create_id(kiblnd_cm_callback, ibdev, RDMA_PS_TCP);
+ id = rdma_create_id(kiblnd_cm_callback, ibdev, *kiblnd_tunables.kib_port_space);
if (IS_ERR(id)) {
CERROR("Can't create listen ID: %ld\n", PTR_ERR(id));
goto failed;
Index: b/lnet/klnds/o2iblnd/o2iblnd.h
===================================================================
--- a/lnet/klnds/o2iblnd/o2iblnd.h
+++ b/lnet/klnds/o2iblnd/o2iblnd.h
@@ -125,6 +125,7 @@ typedef struct
#if defined(CONFIG_SYSCTL) && !CFS_SYSFS_MODULE_PARM
cfs_sysctl_table_header_t *kib_sysctl; /* sysctl interface */
#endif
+ int *kib_port_space; /* port space */
} kib_tunables_t;
extern kib_tunables_t kiblnd_tunables;
Index: b/lnet/klnds/o2iblnd/o2iblnd_cb.c
===================================================================
--- a/lnet/klnds/o2iblnd/o2iblnd_cb.c
+++ b/lnet/klnds/o2iblnd/o2iblnd_cb.c
@@ -1188,7 +1188,7 @@ kiblnd_connect_peer (kib_peer_t *peer)
LASSERT (net != NULL);
LASSERT (peer->ibp_connecting > 0);
- cmid = rdma_create_id(kiblnd_cm_callback, peer, RDMA_PS_TCP);
+ cmid = rdma_create_id(kiblnd_cm_callback, peer, *kiblnd_tunables.kib_port_space);
if (IS_ERR(cmid)) {
CERROR("Can't create CMID for %s: %ld\n",
libcfs_nid2str(peer->ibp_nid), PTR_ERR(cmid));
Index: b/lnet/klnds/o2iblnd/o2iblnd_modparams.c
===================================================================
--- a/lnet/klnds/o2iblnd/o2iblnd_modparams.c
+++ b/lnet/klnds/o2iblnd/o2iblnd_modparams.c
@@ -120,6 +120,10 @@ static int pmr_pool_size = 512;
CFS_MODULE_PARM(pmr_pool_size, "i", int, 0444,
"size of the MR cache pmr pool");
+static int port_space = RDMA_PS_TCP;
+CFS_MODULE_PARM(port_space, "i", int, 0444,
+ "Port space");
+
kib_tunables_t kiblnd_tunables = {
.kib_service = &service,
.kib_cksum = &cksum,
@@ -141,6 +145,7 @@ kib_tunables_t kiblnd_tunables = {
.kib_fmr_flush_trigger = &fmr_flush_trigger,
.kib_fmr_cache = &fmr_cache,
.kib_pmr_pool_size = &pmr_pool_size,
+ .kib_port_space = &port_space,
};
#if defined(CONFIG_SYSCTL) && !CFS_SYSFS_MODULE_PARM
@@ -169,7 +174,8 @@ enum {
O2IBLND_FMR_POOL_SIZE,
O2IBLND_FMR_FLUSH_TRIGGER,
O2IBLND_FMR_CACHE,
- O2IBLND_PMR_POOL_SIZE
+ O2IBLND_PMR_POOL_SIZE,
+ O2IBLND_PORT_SPACE
};
#else
@@ -193,6 +199,7 @@ enum {
#define O2IBLND_FMR_FLUSH_TRIGGER CTL_UNNUMBERED
#define O2IBLND_FMR_CACHE CTL_UNNUMBERED
#define O2IBLND_PMR_POOL_SIZE CTL_UNNUMBERED
+#define O2IBLND_PORT_SPACE CTL_UNNUMBERED
#endif
@@ -358,6 +365,14 @@ static cfs_sysctl_table_t kiblnd_ctl_tab
.mode = 0444,
.proc_handler = &proc_dointvec
},
+ {
+ .ctl_name = O2IBLND_PORT_SPACE,
+ .procname = "port_space",
+ .data = &port_space,
+ .maxlen = sizeof(int),
+ .mode = 0444,
+ .proc_handler = &proc_dointvec
+ },
{0}
};
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 20+ messages in thread
* RE: [PATCH 1/3] rdma_cm: Add support for a new RDMA_PS_LUSTRE Lustre port space
2010-01-13 14:51 ` [PATCH 1/3] rdma_cm: Add support for a new RDMA_PS_LUSTRE Lustre port space sebastien dugue
@ 2010-01-13 16:56 ` Sean Hefty
[not found] ` <7ED07283D76C422C9210FBE7C832731B-Zpru7NauK7drdx17CPfAsdBPR1lH4CV8@public.gmane.org>
0 siblings, 1 reply; 20+ messages in thread
From: Sean Hefty @ 2010-01-13 16:56 UTC (permalink / raw)
To: 'sebastien dugue', linux-rdma; +Cc: Roland Dreier, Sasha Khapyorsky
> This patch adds a new port space for use by Lustre traffic.
>
> This, along with patches to OpenSM and Lustre, allow to define a
>specific QoS for lustre.
In general, I don't like the idea of application port spaces for QoS
support. Can't this be done using port numbers in the existing port
space?
>+ RDMA_PS_LUSTRE = 0x0153,
It looks like this number maps to the VINES protocol.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [PATCH 1/3] rdma_cm: Add support for a new RDMA_PS_LUSTRE Lustre port space
[not found] ` <7ED07283D76C422C9210FBE7C832731B-Zpru7NauK7drdx17CPfAsdBPR1lH4CV8@public.gmane.org>
@ 2010-01-13 17:04 ` Roland Dreier
[not found] ` <ada6376ndgs.fsf-BjVyx320WGW9gfZ95n9DRSW4+XlvGpQz@public.gmane.org>
2010-01-14 12:47 ` sebastien dugue
1 sibling, 1 reply; 20+ messages in thread
From: Roland Dreier @ 2010-01-13 17:04 UTC (permalink / raw)
To: Sean Hefty
Cc: 'sebastien dugue', linux-rdma, Roland Dreier, Sasha Khapyorsky
> In general, I don't like the idea of application port spaces for QoS
> support. Can't this be done using port numbers in the existing port
> space?
I agree. If setting QoS requires a kernel patch for each application, I
think we've messed up QoS somehow. There needs to be a way to control
QoS via configuration, rather than rebuilding.
- R.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [PATCH 1/3] rdma_cm: Add support for a new RDMA_PS_LUSTRE Lustre port space
[not found] ` <7ED07283D76C422C9210FBE7C832731B-Zpru7NauK7drdx17CPfAsdBPR1lH4CV8@public.gmane.org>
2010-01-13 17:04 ` Roland Dreier
@ 2010-01-14 12:47 ` sebastien dugue
2010-01-14 14:09 ` Or Gerlitz
1 sibling, 1 reply; 20+ messages in thread
From: sebastien dugue @ 2010-01-14 12:47 UTC (permalink / raw)
To: Sean Hefty; +Cc: linux-rdma, Roland Dreier, Sasha Khapyorsky
Hi Sean,
On Wed, 13 Jan 2010 08:56:30 -0800
"Sean Hefty" <sean.hefty-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org> wrote:
> > This patch adds a new port space for use by Lustre traffic.
> >
> > This, along with patches to OpenSM and Lustre, allow to define a
> >specific QoS for lustre.
>
> In general, I don't like the idea of application port spaces for QoS
> support. Can't this be done using port numbers in the existing port
> space?
That can be done with port numbers, except that we cannot separate
traffic to Lustre MDS and traffic to Lustre OSS as can be done with
a target-port-guid.
>
> >+ RDMA_PS_LUSTRE = 0x0153,
>
> It looks like this number maps to the VINES protocol.
Argh, haven't seen that. From what I understand, the low 8 bit are to
be taken from IANA IP protocol ID when bit 8 is 1, but what about when
bit 8 is 0 (as is the case with SDP and IPOIB)?
Thanks,
Sebastien.
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [PATCH 1/3] rdma_cm: Add support for a new RDMA_PS_LUSTRE Lustre port space
[not found] ` <ada6376ndgs.fsf-BjVyx320WGW9gfZ95n9DRSW4+XlvGpQz@public.gmane.org>
@ 2010-01-14 12:58 ` sebastien dugue
2010-01-14 17:25 ` Roland Dreier
0 siblings, 1 reply; 20+ messages in thread
From: sebastien dugue @ 2010-01-14 12:58 UTC (permalink / raw)
To: Roland Dreier; +Cc: Sean Hefty, linux-rdma, Roland Dreier, Sasha Khapyorsky
Hi Roland.
On Wed, 13 Jan 2010 09:04:19 -0800
Roland Dreier <rdreier-FYB4Gu1CFyUAvxtiuMwx3w@public.gmane.org> wrote:
>
> > In general, I don't like the idea of application port spaces for QoS
> > support. Can't this be done using port numbers in the existing port
> > space?
>
> I agree. If setting QoS requires a kernel patch for each application, I
> think we've messed up QoS somehow. There needs to be a way to control
> QoS via configuration, rather than rebuilding.
Right, it's possible via configuration only, but you then cannot separate
the different Lustre traffics (MDS and OSS for example).
Sebastien.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [PATCH 1/3] rdma_cm: Add support for a new RDMA_PS_LUSTRE Lustre port space
2010-01-14 12:47 ` sebastien dugue
@ 2010-01-14 14:09 ` Or Gerlitz
[not found] ` <4B4F25A7.6000900-hKgKHo2Ms0FWk0Htik3J/w@public.gmane.org>
0 siblings, 1 reply; 20+ messages in thread
From: Or Gerlitz @ 2010-01-14 14:09 UTC (permalink / raw)
To: sebastien dugue; +Cc: Sean Hefty, linux-rdma, Roland Dreier, Sasha Khapyorsky
sebastien dugue wrote:
> That can be done with port numbers, except that we cannot separate
> traffic to Lustre MDS and traffic to Lustre OSS
Looking on these patches and going with you for a minute, I don't see how this patch set serves you to assign a different QoS level (e.g SL) to MDS vs OSS related traffic. Can you elaborate on that a bit?
Sean Hefty wrote:
> Can't this be done using port numbers in the existing port space?
Indeed, Sebastien what prevents you from using the TCP port space, with one port used for MDS traffic and another port for OSS traffic? how does Lustre get ports to listen on, are they well known or you call bind with port zero and use the port allocated by the rdma-cm?
Or.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [PATCH 1/3] rdma_cm: Add support for a new RDMA_PS_LUSTRE Lustre port space
[not found] ` <4B4F25A7.6000900-hKgKHo2Ms0FWk0Htik3J/w@public.gmane.org>
@ 2010-01-14 14:37 ` sebastien dugue
0 siblings, 0 replies; 20+ messages in thread
From: sebastien dugue @ 2010-01-14 14:37 UTC (permalink / raw)
To: Or Gerlitz; +Cc: Sean Hefty, linux-rdma, Roland Dreier, Sasha Khapyorsky
Hi Or,
On Thu, 14 Jan 2010 16:09:43 +0200
Or Gerlitz <ogerlitz-hKgKHo2Ms0FWk0Htik3J/w@public.gmane.org> wrote:
> sebastien dugue wrote:
> > That can be done with port numbers, except that we cannot separate
> > traffic to Lustre MDS and traffic to Lustre OSS
>
> Looking on these patches and going with you for a minute, I don't see how this patch set serves you to assign a different QoS level (e.g SL) to MDS vs OSS related traffic. Can you elaborate on that a bit?
If you can specify a target-port-guid to the Lustre service ID in OpenSM's
qos-policy.conf then you're all set.
For example:
qos-ulps
default : 0 # default SL
lustre, target-portguid 0x1234,0x1235 : 1 # lustre traffic to MDSs
lustre : 2 # default lustre traffic (to OSSs)
end-qos-ulps
>
> Sean Hefty wrote:
> > Can't this be done using port numbers in the existing port space?
>
> Indeed, Sebastien what prevents you from using the TCP port space, with one port used for MDS traffic and another port for OSS traffic? how does Lustre get ports to listen on, are they well known or you call bind with port zero and use the port allocated by the rdma-cm?
I'm not a Lustre expert (far from it), but after talking with our Lustre guys,
it seems that all Lustre servers, OSS or MDS, use the same TCP port. So there's
no possible way to separate traffic based on TCP port only.
Thanks,
Sébastien.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [PATCH 1/3] rdma_cm: Add support for a new RDMA_PS_LUSTRE Lustre port space
2010-01-14 12:58 ` sebastien dugue
@ 2010-01-14 17:25 ` Roland Dreier
[not found] ` <ada3a28mwdw.fsf-BjVyx320WGW9gfZ95n9DRSW4+XlvGpQz@public.gmane.org>
0 siblings, 1 reply; 20+ messages in thread
From: Roland Dreier @ 2010-01-14 17:25 UTC (permalink / raw)
To: sebastien dugue; +Cc: Sean Hefty, linux-rdma, Roland Dreier, Sasha Khapyorsky
> > I agree. If setting QoS requires a kernel patch for each application, I
> > think we've messed up QoS somehow. There needs to be a way to control
> > QoS via configuration, rather than rebuilding.
>
> Right, it's possible via configuration only, but you then cannot separate
> the different Lustre traffics (MDS and OSS for example).
I guess I don't know enough about Lustre to know why this would be so.
How does creating a new port space for Lustre help with this? Why can't
one do the same thing in one of the existing port spaces?
- R.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [PATCH 1/3] rdma_cm: Add support for a new RDMA_PS_LUSTRE Lustre port space
[not found] ` <ada3a28mwdw.fsf-BjVyx320WGW9gfZ95n9DRSW4+XlvGpQz@public.gmane.org>
@ 2010-01-15 7:41 ` sebastien dugue
2010-01-20 1:12 ` Roland Dreier
0 siblings, 1 reply; 20+ messages in thread
From: sebastien dugue @ 2010-01-15 7:41 UTC (permalink / raw)
To: Roland Dreier; +Cc: Sean Hefty, linux-rdma, Roland Dreier, Sasha Khapyorsky
On Thu, 14 Jan 2010 09:25:31 -0800
Roland Dreier <rdreier-FYB4Gu1CFyUAvxtiuMwx3w@public.gmane.org> wrote:
>
> > > I agree. If setting QoS requires a kernel patch for each application, I
> > > think we've messed up QoS somehow. There needs to be a way to control
> > > QoS via configuration, rather than rebuilding.
> >
> > Right, it's possible via configuration only, but you then cannot separate
> > the different Lustre traffics (MDS and OSS for example).
>
> I guess I don't know enough about Lustre to know why this would be so.
> How does creating a new port space for Lustre help with this? Why can't
> one do the same thing in one of the existing port spaces?
Well, without a specific port space, the default for Lustre is to use the
TCP port space so you cannot distinguish Lustre traffic from other traffic using
that same port space.
Sebastien.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [PATCH 1/3] rdma_cm: Add support for a new RDMA_PS_LUSTRE Lustre port space
2010-01-15 7:41 ` sebastien dugue
@ 2010-01-20 1:12 ` Roland Dreier
[not found] ` <adaiqax37g4.fsf-BjVyx320WGW9gfZ95n9DRSW4+XlvGpQz@public.gmane.org>
0 siblings, 1 reply; 20+ messages in thread
From: Roland Dreier @ 2010-01-20 1:12 UTC (permalink / raw)
To: sebastien dugue; +Cc: Sean Hefty, linux-rdma, Roland Dreier, Sasha Khapyorsky
> Well, without a specific port space, the default for Lustre is to use the
> TCP port space so you cannot distinguish Lustre traffic from other traffic using
> that same port space.
I'm still a bit confused. The problem as I understand it is that Lustre
always uses the same TCP port, so there's no way to apply different QoS
to different types of Lustre traffic. But if we create a new port space
and don't change the ports that Lustre uses, then there's still no way
to apply different QoS for different Lustre traffic. So I guess you
need to change the ports used within the new port space -- but then why
can't you just stay in the TCP space but change the ports used?
- R.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [PATCH 1/3] rdma_cm: Add support for a new RDMA_PS_LUSTRE Lustre port space
[not found] ` <adaiqax37g4.fsf-BjVyx320WGW9gfZ95n9DRSW4+XlvGpQz@public.gmane.org>
@ 2010-01-20 7:55 ` sebastien dugue
2010-01-20 8:03 ` Or Gerlitz
0 siblings, 1 reply; 20+ messages in thread
From: sebastien dugue @ 2010-01-20 7:55 UTC (permalink / raw)
To: Roland Dreier; +Cc: Sean Hefty, linux-rdma, Roland Dreier, Sasha Khapyorsky
Hi Roland,
On Tue, 19 Jan 2010 17:12:43 -0800
Roland Dreier <rdreier-FYB4Gu1CFyUAvxtiuMwx3w@public.gmane.org> wrote:
>
> > Well, without a specific port space, the default for Lustre is to use the
> > TCP port space so you cannot distinguish Lustre traffic from other traffic using
> > that same port space.
>
> I'm still a bit confused. The problem as I understand it is that Lustre
> always uses the same TCP port, so there's no way to apply different QoS
> to different types of Lustre traffic.
Right, all Lustre servers (MDS and OSS) use the same TCP port, so there's no
way to distinguish between the 2 types of traffic.
> But if we create a new port space
> and don't change the ports that Lustre uses, then there's still no way
> to apply different QoS for different Lustre traffic.
With the OpenSM patch and the Lustre port space, you can map Lustre traffic
to specific target port GUIDs onto a specific SL hence separating the flows.
> So I guess you
> need to change the ports used within the new port space -- but then why
> can't you just stay in the TCP space but change the ports used?
No, with the new port space, there's no need to change ports. You only need
to specify the target GUIDs.
For example:
qos-ulps
default : 0 # default SL
lustre, target-portguid 0x1234,0x1235 : 1 # lustre traffic to MDSs
lustre : 2 # default lustre traffic (to OSSs)
end-qos-ulps
Hope this helps clarify things a bit.
Thanks,
Sébastien.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [PATCH 1/3] rdma_cm: Add support for a new RDMA_PS_LUSTRE Lustre port space
2010-01-20 7:55 ` sebastien dugue
@ 2010-01-20 8:03 ` Or Gerlitz
[not found] ` <4B56B8E3.3010909-smomgflXvOZWk0Htik3J/w@public.gmane.org>
0 siblings, 1 reply; 20+ messages in thread
From: Or Gerlitz @ 2010-01-20 8:03 UTC (permalink / raw)
To: sebastien dugue
Cc: Roland Dreier, Sean Hefty, linux-rdma, Roland Dreier, Sasha Khapyorsky
sebastien dugue wrote:
>> So I guess you need to change the ports used within the new port space -- but then
>> why can't you just stay in the TCP space but change the ports used?
>>
>
> No, with the new port space, there's no need to change ports. You only need to specify the target GUIDs. For example:
> lustre, target-portguid 0x1234,0x1235 : 1 # lustre traffic to MDSs
> lustre : 2 # default lustre traffic (to OSSs)
> Hope this helps clarify things a bit.
>
sorry, but it doesn't, as far as I understand there are three
possibilities for what the string "lustre" is being translated to
by the opensm QoS logic:
(A) lustre port in the TCP port space
(B) lustre port space
(C) nothing (that is not a service, in the same manner that ipoib just
doesn't mean anything to opensm)
Assuming C is not the case, then either A or B will yield the same
result and as such the new port space buys you nothing.
Or.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [PATCH 1/3] rdma_cm: Add support for a new RDMA_PS_LUSTRE Lustre port space
[not found] ` <4B56B8E3.3010909-smomgflXvOZWk0Htik3J/w@public.gmane.org>
@ 2010-01-20 9:17 ` sebastien dugue
2010-01-20 15:26 ` Or Gerlitz
0 siblings, 1 reply; 20+ messages in thread
From: sebastien dugue @ 2010-01-20 9:17 UTC (permalink / raw)
To: Or Gerlitz
Cc: Roland Dreier, Sean Hefty, linux-rdma, Roland Dreier, Sasha Khapyorsky
On Wed, 20 Jan 2010 10:03:47 +0200
Or Gerlitz <ogerlitz-smomgflXvOZWk0Htik3J/w@public.gmane.org> wrote:
> sebastien dugue wrote:
> >> So I guess you need to change the ports used within the new port space -- but then
> >> why can't you just stay in the TCP space but change the ports used?
> >>
> >
> > No, with the new port space, there's no need to change ports. You only need to specify the target GUIDs. For example:
> > lustre, target-portguid 0x1234,0x1235 : 1 # lustre traffic to MDSs
> > lustre : 2 # default lustre traffic (to OSSs)
> > Hope this helps clarify things a bit.
> >
> sorry, but it doesn't, as far as I understand there are three
> possibilities for what the string "lustre" is being translated to
> by the opensm QoS logic:
>
> (A) lustre port in the TCP port space
> (B) lustre port space
> (C) nothing (that is not a service, in the same manner that ipoib just
> doesn't mean anything to opensm)
I'm not sure I'm following you here.
With the patches, the 'lustre' keyword is being translated to the lustre
port space and only to that port space, never to the TCP port space. So if
I'm understanding you correctly, it's B.
>
> Assuming C is not the case, then either A or B will yield the same
> result and as such the new port space buys you nothing.
No, because in OpenSM's QoS logic, there's no way to map the TCP port
space with specific target GUIDs onto an SL. You have keywords for SDP, SRP,
RDS, ISER, ... but not for the TCP port space (or am I missing something?).
What B buys you is the ability to map Lustre traffic to a specific
target GUID onto an SL.
An alternative would be to be able to map the TCP port space to a specific
port and specific target GUIDs onto an SL in OpenSM's QoS configuration, but
that's currently not possible.
Sébastien.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [PATCH 1/3] rdma_cm: Add support for a new RDMA_PS_LUSTRE Lustre port space
2010-01-20 9:17 ` sebastien dugue
@ 2010-01-20 15:26 ` Or Gerlitz
[not found] ` <4B572099.3030604-hKgKHo2Ms0FWk0Htik3J/w@public.gmane.org>
0 siblings, 1 reply; 20+ messages in thread
From: Or Gerlitz @ 2010-01-20 15:26 UTC (permalink / raw)
To: sebastien dugue
Cc: Roland Dreier, Sean Hefty, linux-rdma, Roland Dreier, Sasha Khapyorsky
sebastien dugue wrote:
> No, because in OpenSM's QoS logic, there's no way to map the TCP port
> space with specific target GUIDs onto an SL. You have keywords for SDP, SRP,
> RDS, ISER, ... but not for the TCP port space (or am I missing something?).
going with this, what prevents you from patching opensm qos engine to support
the lustre service under the tcp port-space and/or support a combination of service
and target port-guid? all in all, first, I don't see what a kernel patch buys you
and second, if it buys you something you should be able to gain the same effect with
patching open-sm.
thinking on this a bit more, since the rules are processed by order wouldn't the
following scheme let you achieve the same effect?
target-portguid 0x1234,0x1235 : 1 # traffic to MDSs
lustre : 2 # default lustre traffic (to OSSs)
Or.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [PATCH 1/3] rdma_cm: Add support for a new RDMA_PS_LUSTRE Lustre port space
[not found] ` <4B572099.3030604-hKgKHo2Ms0FWk0Htik3J/w@public.gmane.org>
@ 2010-01-21 8:28 ` sebastien dugue
2010-01-21 9:12 ` Or Gerlitz
0 siblings, 1 reply; 20+ messages in thread
From: sebastien dugue @ 2010-01-21 8:28 UTC (permalink / raw)
To: Or Gerlitz
Cc: Roland Dreier, Sean Hefty, linux-rdma, Roland Dreier, Sasha Khapyorsky
Hi Or,
On Wed, 20 Jan 2010 17:26:17 +0200
Or Gerlitz <ogerlitz-hKgKHo2Ms0FWk0Htik3J/w@public.gmane.org> wrote:
> sebastien dugue wrote:
> > No, because in OpenSM's QoS logic, there's no way to map the TCP port
> > space with specific target GUIDs onto an SL. You have keywords for SDP, SRP,
> > RDS, ISER, ... but not for the TCP port space (or am I missing something?).
>
> going with this, what prevents you from patching opensm qos engine to support
> the lustre service under the tcp port-space and/or support a combination of service
> and target port-guid? all in all, first, I don't see what a kernel patch buys you
> and second, if it buys you something you should be able to gain the same effect with
> patching open-sm.
Right, I can indeed achieve that with only a patch to OpenSM. Going with a
specific Lustre port space is simple and less intrusive although you have to
patch the kernel and OpenSM.
OK, then going with the TCP port space, what we need in OpenSM is a
combination of service id (TCP) _and_ TCP port _and_ target GUID.
Something like:
tcp port <lustre TCP port> target-port-guid <Lustre servers GUIDs>
That way, as you point out, there's absolutely no need for a specific
Lustre specific port space and kernel patch. I just need to extend the
QoS parser logic to add the semantic above.
I will look into this.
Thanks, and sorry for the hassle.
Sebastien.
>
> thinking on this a bit more, since the rules are processed by order wouldn't the
> following scheme let you achieve the same effect?
>
> target-portguid 0x1234,0x1235 : 1 # traffic to MDSs
> lustre : 2 # default lustre traffic (to OSSs)
>
> Or.
>
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [PATCH 1/3] rdma_cm: Add support for a new RDMA_PS_LUSTRE Lustre port space
2010-01-21 8:28 ` sebastien dugue
@ 2010-01-21 9:12 ` Or Gerlitz
[not found] ` <4B581A68.500-hKgKHo2Ms0FWk0Htik3J/w@public.gmane.org>
0 siblings, 1 reply; 20+ messages in thread
From: Or Gerlitz @ 2010-01-21 9:12 UTC (permalink / raw)
To: sebastien dugue; +Cc: linux-rdma, Sasha Khapyorsky
sebastien dugue wrote:
> OK, then going with the TCP port space, what we need in OpenSM is a
> combination of service id (TCP) _and_ TCP port _and_ target GUID.
I believe that you can have a 'lustre' keyword in opensm qos parser which stands for the combination of tcp port space + lustre tcp port (maybe it exists now), so in the policy file this would translate to X,{Z1,Z2,..,Zm} (as was in your example) and not to X,Y,{Z1,Z2,..,Zm}.
Or.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [PATCH 1/3] rdma_cm: Add support for a new RDMA_PS_LUSTRE Lustre port space
[not found] ` <4B581A68.500-hKgKHo2Ms0FWk0Htik3J/w@public.gmane.org>
@ 2010-01-21 9:26 ` sebastien dugue
0 siblings, 0 replies; 20+ messages in thread
From: sebastien dugue @ 2010-01-21 9:26 UTC (permalink / raw)
To: Or Gerlitz; +Cc: linux-rdma, Sasha Khapyorsky
On Thu, 21 Jan 2010 11:12:08 +0200
Or Gerlitz <ogerlitz-hKgKHo2Ms0FWk0Htik3J/w@public.gmane.org> wrote:
> sebastien dugue wrote:
> > OK, then going with the TCP port space, what we need in OpenSM is a
> > combination of service id (TCP) _and_ TCP port _and_ target GUID.
>
> I believe that you can have a 'lustre' keyword in opensm qos parser which stands for the combination of tcp port space + lustre tcp port (maybe it exists now), so in the policy file this would translate to X,{Z1,Z2,..,Zm} (as was in your example) and not to X,Y,{Z1,Z2,..,Zm}.
Right, that'd be a nice shortcut. I also think being able to specify both port
and target GUID to the TCP port space might be interesting for other ULPs, not
just Lustre.
Sebastien.
>
> Or.
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 20+ messages in thread
end of thread, other threads:[~2010-01-21 9:26 UTC | newest]
Thread overview: 20+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2010-01-13 14:49 [PATCH 0/3] Add support for specific Lustre Qos sebastien dugue
2010-01-13 14:51 ` [PATCH 1/3] rdma_cm: Add support for a new RDMA_PS_LUSTRE Lustre port space sebastien dugue
2010-01-13 16:56 ` Sean Hefty
[not found] ` <7ED07283D76C422C9210FBE7C832731B-Zpru7NauK7drdx17CPfAsdBPR1lH4CV8@public.gmane.org>
2010-01-13 17:04 ` Roland Dreier
[not found] ` <ada6376ndgs.fsf-BjVyx320WGW9gfZ95n9DRSW4+XlvGpQz@public.gmane.org>
2010-01-14 12:58 ` sebastien dugue
2010-01-14 17:25 ` Roland Dreier
[not found] ` <ada3a28mwdw.fsf-BjVyx320WGW9gfZ95n9DRSW4+XlvGpQz@public.gmane.org>
2010-01-15 7:41 ` sebastien dugue
2010-01-20 1:12 ` Roland Dreier
[not found] ` <adaiqax37g4.fsf-BjVyx320WGW9gfZ95n9DRSW4+XlvGpQz@public.gmane.org>
2010-01-20 7:55 ` sebastien dugue
2010-01-20 8:03 ` Or Gerlitz
[not found] ` <4B56B8E3.3010909-smomgflXvOZWk0Htik3J/w@public.gmane.org>
2010-01-20 9:17 ` sebastien dugue
2010-01-20 15:26 ` Or Gerlitz
[not found] ` <4B572099.3030604-hKgKHo2Ms0FWk0Htik3J/w@public.gmane.org>
2010-01-21 8:28 ` sebastien dugue
2010-01-21 9:12 ` Or Gerlitz
[not found] ` <4B581A68.500-hKgKHo2Ms0FWk0Htik3J/w@public.gmane.org>
2010-01-21 9:26 ` sebastien dugue
2010-01-14 12:47 ` sebastien dugue
2010-01-14 14:09 ` Or Gerlitz
[not found] ` <4B4F25A7.6000900-hKgKHo2Ms0FWk0Htik3J/w@public.gmane.org>
2010-01-14 14:37 ` sebastien dugue
2010-01-13 14:54 ` [PATCH 2/3] opensm/qos_policy: Add a new service ID and keyword for Lustre QoS sebastien dugue
2010-01-13 14:56 ` [PATCH 3/3] Add new port_space parameter to ko2iblnd module sebastien dugue
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.