On Fri, Aug 12, 2016 at 3:03 PM, Hal Rosenstock wrote: > On 8/12/2016 7:55 AM, Hal Rosenstock wrote: >> On 8/12/2016 4:15 AM, Jinpu Wang wrote: >>> On Thu, Aug 11, 2016 at 11:15 PM, Hal Rosenstock wrote: >>>> On 8/11/2016 8:29 AM, Jinpu Wang wrote: >>>>> On Wed, Aug 10, 2016 at 8:52 PM, Hal Rosenstock wrote: >>>>>> On 8/9/2016 12:26 PM, Jinpu Wang wrote: >>>>>>> Hi Sean, >>>>>>> >>>>>>> I'm testing QoS support for IB. I notice ucmatose has equally >>>>>>> performance when set different service type, but set SL in ib_send_bw >>>>>>> works well (different SL show different performance base on opensm >>>>>>> settings) >>>>>>> >>>>>>> I capature packats using ibdump, it shows in in LRH the service level >>>>>>> fields are all 0 when running traffic with ucmatose. >>>>>>> >>>>>>> When running ib_send_bw, it carries the right service level I set. >>>>>>> >>>>>>> Seems in rdma_set_service_type, it sets to tos to id_priv->tos, and >>>>>>> lter set to path_rec->qos_class or traffic_class but not to sl >>>>>>> directly, what's the consideration here? >>>>>>> code snip: >>>>>>> switch (cma_family(id_priv)) { >>>>>>> case AF_INET: >>>>>>> path_rec->qos_class = cpu_to_be16((u16) id_priv->tos); >>>>>>> comp_mask |= IB_SA_PATH_REC_QOS_CLASS; >>>>>>> break; >>>>>>> case AF_INET6: >>>>>>> sin6 = (struct sockaddr_in6 *) cma_src_addr(id_priv); >>>>>>> path_rec->traffic_class = (u8) >>>>>>> (be32_to_cpu(sin6->sin6_flowinfo) >> 20); >>>>>>> comp_mask |= IB_SA_PATH_REC_TRAFFIC_CLASS; >>>>>>> break; >>>>>>> case AF_IB: >>>>>>> sib = (struct sockaddr_ib *) cma_src_addr(id_priv); >>>>>>> path_rec->traffic_class = (u8) >>>>>>> (be32_to_cpu(sib->sib_flowinfo) >> 20); >>>>>>> >>>>>>> >>>>>>> Does it make sense we also set sl here, or service type for ucmatose >>>>>>> is totally different with SL for ib_send_bw? >>>>>> >>>>>> I think this is an OpenSM configuration issue. QoS policy needs to be >>>>>> setup to return the proper SL to use for QoS class or TClass in the >>>>>> PathRecord response. >>>>>> >>>>>> -- Hal >>>>>> >>>>> Thanks Hal, >>>>> >>>>> Configure extra QoS policy seems quite complex. >>>> >>>> Configuration complexity varies depending on the requirements of the QoS >>>> needs. >>>> >>>> Which type of RDMA CM connections are being used (IPv4, IPv6, or native >>>> IB) ? >>>> >>>>> Do you think patch attached make sense? >>>> >>>> Attached patch doesn't appear to relate to upstream. >>> >>> Indeed, it's based on MLNXOFED 3.2 >>> >>>> >>>> It also looks incomplete to me. What invokes rdma_set_service_level ? Is >>>> it some option in ucma.c:ucma_set_option ? >>> >>> The main purpose is for our in house transport kernel module, > > One more thing: > > How does transport module know which SL to request ? > > In general, SL is based on SM configuration. > > Service ID and QoS Class or Traffic Class are the "higher level" IB > architected ways to obtain the SL. > >>> it >>> supports all 3 connections >>> (IPv4, IPv6, and native IB, IB is the default). >> >>>> Current patch doesn't appear to me to be backward compatible. If >>>> rdma_set_service_level is not called in flow, then SL should not be set >>>> in SA PR query which is what happens today. >>> >>> Good point, I will add check only set SL if not 0, >> >> 0 is a valid SL so an extra bit somewhere is needed to indicate whether >> a specific SL is being requested. >> >>> but if >>> rdma_set_service_level is not called, >>> SL should be 0 as before, shouldn't change SA PR query behavior, or I >>> missed something? >> >> Component mask for SL in SA PR query is not on currently so that means >> it's wildcarded rather than 0. >> >>>> Also, if SL is set in query, you probably don't need some of the other >>>> fields that are being set. >>>> >>> Do you mean SL shouldn't be set with other fields, what's the side effect there? >> >> Never mind. It's probably best to leave those other fields as is. >> Thanks, I update my patch to address your comments. For configuration, we introduce a module parameter, sysadmin will set that. similar for ib_send_bw kinds of perf tools. -- Mit freundlichen Grüßen, Best Regards, Jack Wang Linux Kernel Developer Storage ProfitBricks GmbH The IaaS-Company. ProfitBricks GmbH Greifswalder Str. 207 D - 10405 Berlin Tel: +49 30 5770083-42 Fax: +49 30 5770085-98 Email: jinpu.wang-EIkl63zCoXaH+58JC4qpiA@public.gmane.org URL: http://www.profitbricks.de Sitz der Gesellschaft: Berlin. Registergericht: Amtsgericht Charlottenburg, HRB 125506 B. Geschäftsführer: Andreas Gauger, Achim Weiss.