From mboxrd@z Thu Jan 1 00:00:00 1970 From: "Ananyev, Konstantin" Subject: Re: [RFC] Add GRO support in DPDK Date: Wed, 25 Jan 2017 09:51:50 +0000 Message-ID: <2601191342CEEE43887BDE71AB9772583F10C882@irsmsx105.ger.corp.intel.com> References: <1485176592-111525-1-git-send-email-jiayu.hu@intel.com> <20170123091550.212dca35@xeon-e3> <6B5C6BED-CAD4-4C51-8FB7-8509663B813B@intel.com> <2601191342CEEE43887BDE71AB9772583F10AD94@irsmsx105.ger.corp.intel.com> <1F520FF1-C38B-483B-95E1-FBD4C631E7D2@intel.com> <2601191342CEEE43887BDE71AB9772583F10AEBD@irsmsx105.ger.corp.intel.com> <6D277342-5212-462F-A507-93B63E86DA90@intel.com> <2601191342CEEE43887BDE71AB9772583F10C6DC@irsmsx105.ger.corp.intel.com> <20170124130425.35fe2a16@xeon-e3> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Cc: "Hu, Jiayu" , "dev@dpdk.org" , "Kinsella, Ray" , "Gilmore, Walter E" , "Venkatesan, Venky" , "yuanhan.liu@linux.intel.com" To: "Wiles, Keith" , Stephen Hemminger Return-path: Received: from mga11.intel.com (mga11.intel.com [192.55.52.93]) by dpdk.org (Postfix) with ESMTP id B835FB6D for ; Wed, 25 Jan 2017 10:51:55 +0100 (CET) In-Reply-To: Content-Language: en-US List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" > -----Original Message----- > From: Wiles, Keith > Sent: Wednesday, January 25, 2017 3:40 AM > To: Stephen Hemminger > Cc: Ananyev, Konstantin ; Hu, Jiayu ; dev@dpdk.org; Kinsella, Ray > ; Gilmore, Walter E ;= Venkatesan, Venky ; > yuanhan.liu@linux.intel.com > Subject: Re: [dpdk-dev] [RFC] Add GRO support in DPDK >=20 >=20 > > On Jan 24, 2017, at 2:04 PM, Stephen Hemminger wrote: > > > > On Tue, 24 Jan 2017 20:09:07 +0000 > > "Wiles, Keith" wrote: > > > >>> On Jan 24, 2017, at 12:45 PM, Ananyev, Konstantin wrote: > >>> > >>> > >>> > >>>> -----Original Message----- > >>>> From: Wiles, Keith > >>>> Sent: Tuesday, January 24, 2017 2:49 PM > >>>> To: Ananyev, Konstantin > >>>> Cc: Stephen Hemminger ; Hu, Jiayu ; dev@dpdk.org; Kinsella, Ray > >>>> ; Gilmore, Walter E ; Venkatesan, Venky ; > >>>> yuanhan.liu@linux.intel.com > >>>> Subject: Re: [dpdk-dev] [RFC] Add GRO support in DPDK > >>>> > >>>> > >>>>> On Jan 24, 2017, at 3:33 AM, Ananyev, Konstantin wrote: > >>>>> > >>>>> > >>>>> > >>>>>> -----Original Message----- > >>>>>> From: Wiles, Keith > >>>>>> Sent: Tuesday, January 24, 2017 5:26 AM > >>>>>> To: Ananyev, Konstantin > >>>>>> Cc: Stephen Hemminger ; Hu, Jiayu ; dev@dpdk.org; Kinsella, Ray > >>>>>> ; Gilmore, Walter E ; Venkatesan, Venky ; > >>>>>> yuanhan.liu@linux.intel.com > >>>>>> Subject: Re: [dpdk-dev] [RFC] Add GRO support in DPDK > >>>>>> > >>>>>> > >>>>>>> On Jan 23, 2017, at 6:43 PM, Ananyev, Konstantin wrote: > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>>> -----Original Message----- > >>>>>>>> From: Wiles, Keith > >>>>>>>> Sent: Monday, January 23, 2017 9:53 PM > >>>>>>>> To: Stephen Hemminger > >>>>>>>> Cc: Hu, Jiayu ; dev@dpdk.org; Kinsella, Ray = ; Ananyev, Konstantin > >>>>>>>> ; Gilmore, Walter E ; Venkatesan, Venky > >>>>>> ; > >>>>>>>> yuanhan.liu@linux.intel.com > >>>>>>>> Subject: Re: [dpdk-dev] [RFC] Add GRO support in DPDK > >>>>>>>> > >>>>>>>> > >>>>>>>>> On Jan 23, 2017, at 10:15 AM, Stephen Hemminger wrote: > >>>>>>>>> > >>>>>>>>> On Mon, 23 Jan 2017 21:03:12 +0800 > >>>>>>>>> Jiayu Hu wrote: > >>>>>>>>> > >>>>>>>>>> With the support of hardware segmentation techniques in DPDK, = the > >>>>>>>>>> networking stack overheads of send-side of applications, which= directly > >>>>>>>>>> leverage DPDK, have been greatly reduced. But for receive-side= , numbers of > >>>>>>>>>> segmented packets seriously burden the networking stack of app= lications. > >>>>>>>>>> Generic Receive Offload (GRO) is a widely used method to solve= the > >>>>>>>>>> receive-side issue, which gains performance by reducing the am= ount of > >>>>>>>>>> packets processed by the networking stack. But currently, DPDK= doesn't > >>>>>>>>>> support GRO. Therefore, we propose to add GRO support in DPDK,= and this > >>>>>>>>>> RFC is used to explain the basic DPDK GRO design. > >>>>>>>>>> > >>>>>>>>>> DPDK GRO is a SW-based packets assembly library, which provide= s GRO > >>>>>>>>>> abilities for numbers of protocols. In DPDK GRO, packets are m= erged > >>>>>>>>>> before returning to applications and after receiving from driv= ers. > >>>>>>>>>> > >>>>>>>>>> In DPDK, GRO is a capability of NIC drivers. That support GRO = or not and > >>>>>>>>>> what GRO types are supported are up to NIC drivers. Different = drivers may > >>>>>>>>>> support different GRO types. By default, drivers enable all su= pported GRO > >>>>>>>>>> types. For applications, they can inquire the supported GRO ty= pes by > >>>>>>>>>> each driver, and can control what GRO types are applied. For e= xample, > >>>>>>>>>> ixgbe supports TCP and UDP GRO, but the application just needs= TCP GRO. > >>>>>>>>>> The application can disable ixgbe UDP GRO. > >>>>>>>>>> > >>>>>>>>>> To support GRO, a driver should provide a way to tell applicat= ions what > >>>>>>>>>> GRO types are supported, and provides a GRO function, which is= in charge > >>>>>>>>>> of assembling packets. Since different drivers may support dif= ferent GRO > >>>>>>>>>> types, their GRO functions may be different. For applications,= they don't > >>>>>>>>>> need extra operations to enable GRO. But if there are some GRO= types that > >>>>>>>>>> are not needed, applications can use an API, like > >>>>>>>>>> rte_eth_gro_disable_protocols, to disable them. Besides, they = can > >>>>>>>>>> re-enable the disabled ones. > >>>>>>>>>> > >>>>>>>>>> The GRO function processes numbers of packets at a time. In ea= ch > >>>>>>>>>> invocation, what GRO types are applied depends on applications= , and the > >>>>>>>>>> amount of packets to merge depends on the networking status an= d > >>>>>>>>>> applications. Specifically, applications determine the maximum= number of > >>>>>>>>>> packets to be processed by the GRO function, but how many pack= ets are > >>>>>>>>>> actually processed depends on if there are available packets t= o receive. > >>>>>>>>>> For example, the receive-side application asks the GRO functio= n to > >>>>>>>>>> process 64 packets, but the sender only sends 40 packets. At t= his time, > >>>>>>>>>> the GRO function returns after processing 40 packets. To reass= emble the > >>>>>>>>>> given packets, the GRO function performs an "assembly procedur= e" on each > >>>>>>>>>> packet. We use an example to demonstrate this procedure. Suppo= sing the > >>>>>>>>>> GRO function is going to process packetX, it will do the follo= wing two > >>>>>>>>>> things: > >>>>>>>>>> a. Find a L4 assembly function according to the packet type o= f > >>>>>>>>>> packetX. A L4 assembly function is in charge of merging packe= ts of a > >>>>>>>>>> specific type. For example, TCPv4 assembly function merges pa= ckets > >>>>>>>>>> whose L3 IPv4 and L4 is TCP. Each L4 assembly function has a = packet > >>>>>>>>>> array, which keeps the packets that are unable to assemble. > >>>>>>>>>> Initially, the packet array is empty; > >>>>>>>>>> b. The L4 assembly function traverses own packet array to fin= d a > >>>>>>>>>> mergeable packet (comparing Ethernet, IP and L4 header fields= ). If > >>>>>>>>>> finds, merges it and packetX via chaining them together; if d= oesn't, > >>>>>>>>>> allocates a new array element to store packetX and updates el= ement > >>>>>>>>>> number of the array. > >>>>>>>>>> After performing the assembly procedure to all packets, the GR= O function > >>>>>>>>>> combines the results of all packet arrays, and returns these p= ackets to > >>>>>>>>>> applications. > >>>>>>>>>> > >>>>>>>>>> There are lots of ways to implement the above design in DPDK. = One of the > >>>>>>>>>> ways is: > >>>>>>>>>> a. Drivers tell applications what GRO types are supported via > >>>>>>>>>> dev->dev_ops->dev_infos_get; > >>>>>>>>>> b. When initialize, drivers register own GRO function as a RX > >>>>>>>>>> callback, which is invoked inside rte_eth_rx_burst. The name = of the > >>>>>>>>>> GRO function should be like xxx_gro_receive (e.g. ixgbe_gro_r= eceive). > >>>>>>>>>> Currently, the RX callback can only process the packets retur= ned by > >>>>>>>>>> dev->rx_pkt_burst each time, and the maximum packet number > >>>>>>>>>> dev->rx_pkt_burst returns is determined by each driver, which= can't > >>>>>>>>>> be interfered by applications. Therefore, to implement the ab= ove GRO > >>>>>>>>>> design, we have to modify current RX implementation to make d= river > >>>>>>>>>> return packets as many as possible until the packet number me= ets the > >>>>>>>>>> demand of applications or there are not available packets to = receive. > >>>>>>>>>> This modification is also proposed in patch: > >>>>>>>>>> http://dpdk.org/ml/archives/dev/2017-January/055887.html; > >>>>>>>>>> c. The GRO types to apply and the maximum number of packets t= o merge > >>>>>>>>>> are passed by resetting RX callback parameters. It can be ach= ieved by > >>>>>>>>>> invoking rte_eth_rx_callback; > >>>>>>>>>> d. Simply, we can just store packet addresses into the packet= array. > >>>>>>>>>> To check one element, we need to fetch the packet via its add= ress. > >>>>>>>>>> However, this simple design is not efficient enough. Since wh= enever > >>>>>>>>>> checking one packet, one pointer dereference is generated. An= d a > >>>>>>>>>> pointer dereference always causes a cache line miss. A better= way is > >>>>>>>>>> to store some rules in each array element. The rules must be = the > >>>>>>>>>> prerequisites of merging two packets, like the sequence numbe= r of TCP > >>>>>>>>>> packets. We first compare the rules, then retrieve the packet= if the > >>>>>>>>>> rules match. If storing the rules causes the packet array str= ucture > >>>>>>>>>> is cache-unfriendly, we can store a fixed-length signature of= the > >>>>>>>>>> rules instead. For example, the signature can be calculated b= y > >>>>>>>>>> performing XOR operation on IP addresses. Both design can avo= id > >>>>>>>>>> unnecessary pointer dereferences. > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> Since DPDK does burst mode already, GRO is a lot less relevant. > >>>>>>>>> GRO in Linux was invented because there is no burst mode in the= receive API. > >>>>>>>>> > >>>>>>>>> If you look at VPP in FD.io you will see they already do aggreg= ration and > >>>>>>>>> steering at the higher level in the stack. > >>>>>>>>> > >>>>>>>>> The point of GRO is that it is generic, no driver changes are n= ecessary. > >>>>>>>>> Your proposal would add a lot of overhead, and cause drivers to= have to > >>>>>>>>> be aware of higher level flows. > >>>>>>>> > >>>>>>>> NACK > >>>>>>>> > >>>>>>>> The design is not super clear to me here and we need to understa= nd the impact to DPDK, performance and the application. I > would > >>>> like > >>>>>> to > >>>>>>>> have a clean transparent design to the application and as little= impact on performance as possible. > >>>>>>>> > >>>>>>>> Let discuss this as I am not sure my previous concerns were addr= essed in this RFC. > >>>>>>>> > >>>>>>> > >>>>>>> I would agree that design looks overcomplicated and strange: > >>>>>>> If GRO can (and supposed to be) done fully in SW, why do we need = to modify PMDs at all, > >>>>>>> why it can't be just a standalone DPDK library that user can use = on his/her convenience? > >>>>>>> I'd suggest to start with some simple and most widespread case (T= CP?) and try to implement > >>>>>>> a library for it first: something similar to what we have for ip = reassembly. > >>>>>> > >>>>>> The reason this should not be a library the application calls is t= o allow for a transparent design for HW and SW support of this > feature. > >>>> Using > >>>>>> the SW version the application should not need to understand (othe= r then performance) that GRO is being done for this port. > >>>>>> > >>>>> > >>>>> Why is that? > >>>>> Let say we have ip reassembly library that is called explicitly by = the application. > >>>>> I think for L4 grouping we can do the same. > >>>>> After all it is a pure SW feature, so to me it makes sense to allow= application to decide > >>>>> when/where to call it. > >>>>> Again it would allow people to develop/use it without any modificat= ions in current PMDs. > >>>> > >>>> I guess I did not make it clear, we need to support HW and this SW v= ersion transparently just as we handle other features in HW/SW > under a > >>>> generic API for DPDK. > >>> > >>> Ok, I probably wasn't very clear too. > >>> What I meant: > >>> Let's try to implement GRO (in SW) as a standalone DPDK library, > >>> with clean & simple interface and see how fast and useful it would be= . > >>> We can refer to it as step 1. > >>> When (if) we'll have step 1 in place, then we can start thinking > >>> about adding combined HW/SW solution for it (step 2). > >>> I think at that stage it would be much clearer: > >>> is there any point in it at all, > >>> and if yes, how it should be done: > >>> -changes at rte_ethedev or on PMD layers or both > >>> - would changes at rte_ethdev API be needed and if yes what particula= r, etc. > >>> > >>> From my perspective, without step 1 in place, there is no much point= in approaching step 2. > >> > >> Currently I believe they have a SW library version of the code, but I = think we need to look at the design in that form. At this time the > current design or code is not what I would expect needs to be done for th= e transparent version. To many interactions with the application > and a separate Rx/Tx functions were being used (If I remember correctly) > >> > >>> > >>> BTW, any particular HW you have in mind? > >>> Currently, as I can see LRO (HW) is supported only by ixgbe and proba= bly by viritual PMDs (virtio/vmxent3). > >>> Though even for ixgbe there are plenty of limitations: SRIOV mode sho= uld be off, HW CRC stropping should be off, etc. > >>> So my guess, right now step 1 is much more useful and feasible. > >>> > >>>> > >>>>> > >>>>>> As I was told the Linux kernel hides this features and make it tra= nsparent. > >>>>> > >>>>> Yes, but DPDK does a lot things in a different way. > >>>>> So it doesn't look like a compelling reason for me :) > >>>> > >>>> Just looking at different options here and it is a compelling reason= to me as it enforces the design can be transparent to the > application. > >>>> Having the application in a NFV deciding on hw or sw or both is not = a good place to put that logic IMO. > >>> > >>> Actually could you provide an example of linux NIC driver, that uses = HW offloads (and which) to implement GRO? > >>> I presume some might use HW generated hashes, but apart from that, wh= en HW performs actual packet grouping? > >>> From what I've seen Intel ones rely SW implementation for that. > >>> But I am not a linux/GRO expert, so feel free to correct me here. > >>> Konstantin > >> > >> Regards, > >> Keith > >> > > > > Linux uses a push (rather than DPDK pull) model for packet receiving. > > The Linux driver pushes packets into GRO by calling napi_gro_receive. > > > > Since DPDK is pull model the API would be simpler. > > it could be as simple as: > > nb =3D rte_eth_rx_burst(port, rx_pkts, N); > > nb =3D rte_rx_gro(port, rx_pkts, gro_pkts, nb); > > > > I agree with others, look at ip reassembly library as example. > > Also, GRO does not make sense for applications which already do the sam= e vector flow > > processing like VPP which is one reason it should be optional. >=20 > I agree it should be option, but I worry about making it an example. I wo= uld like to see the GRO to be more transparent to the application > and supported as a generic feature for DPDK. Maybe the application needs = to request the support or it is a config option. The problem with > config options is they are hard to test and testing becomes complexed. >=20 > Can we not figure out a way to add the feature inline instead of the appl= ication needing to call these APIs? It would be nice to have IP > fragmentation also a optional feature to the rx/tx ethdev call. It would = take it out of the example zone and move it into DPDK as a real > feature. Today we expect the application to chain all of these little bit= s outside of DPDK into something useful, can we help fix that > problem? If the user would like this feature to be transparent, he/she can always setup a RX callback that would call GRO API inside. Let say TLDK udpfwd example does the same for ip reassemble. Konstantin=20