From mboxrd@z Thu Jan 1 00:00:00 1970 From: "Ananyev, Konstantin" Subject: Re: [PATCH v7 1/3] lib: add Generic Receive Offload API framework Date: Tue, 27 Jun 2017 23:42:01 +0000 Message-ID: <2601191342CEEE43887BDE71AB9772583FB0F4DA@IRSMSX109.ger.corp.intel.com> References: <1498229000-94867-1-git-send-email-jiayu.hu@intel.com> <1498459430-116048-1-git-send-email-jiayu.hu@intel.com> <1498459430-116048-2-git-send-email-jiayu.hu@intel.com> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Cc: "Tan, Jianfeng" , "stephen@networkplumber.org" , "yliu@fridaylinux.org" , "Wu, Jingjing" , "Yao, Lei A" , "Wiles, Keith" , "Bie, Tiwei" To: "Hu, Jiayu" , "dev@dpdk.org" Return-path: Received: from mga03.intel.com (mga03.intel.com [134.134.136.65]) by dpdk.org (Postfix) with ESMTP id 0B0D8271 for ; Wed, 28 Jun 2017 01:42:05 +0200 (CEST) In-Reply-To: <1498459430-116048-2-git-send-email-jiayu.hu@intel.com> Content-Language: en-US List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Hi Jiayu, > -----Original Message----- > From: Hu, Jiayu > Sent: Monday, June 26, 2017 7:44 AM > To: dev@dpdk.org > Cc: Tan, Jianfeng ; Ananyev, Konstantin ; stephen@networkplumber.org; > yliu@fridaylinux.org; Wu, Jingjing ; Yao, Lei A ; Wiles, Keith ; Bie, > Tiwei ; Hu, Jiayu > Subject: [PATCH v7 1/3] lib: add Generic Receive Offload API framework >=20 > Generic Receive Offload (GRO) is a widely used SW-based offloading > technique to reduce per-packet processing overhead. It gains > performance by reassembling small packets into large ones. This > patchset is to support GRO in DPDK. To support GRO, this patch > implements a GRO API framework. >=20 > To enable more flexibility to applications, DPDK GRO is implemented as > a user library. Applications explicitly use the GRO library to merge > small packets into large ones. DPDK GRO provides two reassembly modes. > One is called lightweigth mode, the other is called heavyweight mode. > If applications want to merge packets in a simple way and the number > of packets is relatively small, they can use the lightweigth mode. > If applications need more fine-grained controls, they can choose the > heavyweigth mode. >=20 > rte_gro_reassemble_burst is the main reassembly API which is used in > lightweigth mode and processes N packets at a time. For applications, > performing GRO in lightweigth mode is simple. They just need to invoke > rte_gro_reassemble_burst. Applications can get GROed packets as soon as > rte_gro_reassemble_burst returns. >=20 > rte_gro_reassemble is the main reassembly API which is used in > heavyweight mode and processes one packet at a time. For applications, > performing GRO in heavyweigth mode is relatively complicated. Before > performing GRO, applications need to create a GRO table by > rte_gro_tbl_create. Then they can use rte_gro_reassemble to process > packets one by one. The processed packets are in the GRO table. If > applications want to get them, applications need to manually flush > them by flush APIs. >=20 > Signed-off-by: Jiayu Hu > --- > config/common_base | 5 + > lib/Makefile | 2 + > lib/librte_gro/Makefile | 50 +++++++++ > lib/librte_gro/rte_gro.c | 125 ++++++++++++++++++++++ > lib/librte_gro/rte_gro.h | 205 +++++++++++++++++++++++++++++++= ++++++ > lib/librte_gro/rte_gro_version.map | 12 +++ > mk/rte.app.mk | 1 + > 7 files changed, 400 insertions(+) > create mode 100644 lib/librte_gro/Makefile > create mode 100644 lib/librte_gro/rte_gro.c > create mode 100644 lib/librte_gro/rte_gro.h > create mode 100644 lib/librte_gro/rte_gro_version.map >=20 > diff --git a/config/common_base b/config/common_base > index f6aafd1..167f5ef 100644 > --- a/config/common_base > +++ b/config/common_base > @@ -712,6 +712,11 @@ CONFIG_RTE_LIBRTE_VHOST_DEBUG=3Dn > CONFIG_RTE_LIBRTE_PMD_VHOST=3Dn >=20 > # > +# Compile GRO library > +# > +CONFIG_RTE_LIBRTE_GRO=3Dy > + > +# > #Compile Xen domain0 support > # > CONFIG_RTE_LIBRTE_XEN_DOM0=3Dn > diff --git a/lib/Makefile b/lib/Makefile > index 07e1fd0..ac1c2f6 100644 > --- a/lib/Makefile > +++ b/lib/Makefile > @@ -106,6 +106,8 @@ DIRS-$(CONFIG_RTE_LIBRTE_REORDER) +=3D librte_reorder > DEPDIRS-librte_reorder :=3D librte_eal librte_mempool librte_mbuf > DIRS-$(CONFIG_RTE_LIBRTE_PDUMP) +=3D librte_pdump > DEPDIRS-librte_pdump :=3D librte_eal librte_mempool librte_mbuf librte_e= ther > +DIRS-$(CONFIG_RTE_LIBRTE_GRO) +=3D librte_gro > +DEPDIRS-librte_gro :=3D librte_eal librte_mbuf librte_ether librte_net >=20 > ifeq ($(CONFIG_RTE_EXEC_ENV_LINUXAPP),y) > DIRS-$(CONFIG_RTE_LIBRTE_KNI) +=3D librte_kni > diff --git a/lib/librte_gro/Makefile b/lib/librte_gro/Makefile > new file mode 100644 > index 0000000..7e0f128 > --- /dev/null > +++ b/lib/librte_gro/Makefile > @@ -0,0 +1,50 @@ > +# BSD LICENSE > +# > +# Copyright(c) 2017 Intel Corporation. All rights reserved. > +# All rights reserved. > +# > +# Redistribution and use in source and binary forms, with or without > +# modification, are permitted provided that the following conditions > +# are met: > +# > +# * Redistributions of source code must retain the above copyright > +# notice, this list of conditions and the following disclaimer. > +# * Redistributions in binary form must reproduce the above copyrigh= t > +# notice, this list of conditions and the following disclaimer in > +# the documentation and/or other materials provided with the > +# distribution. > +# * Neither the name of Intel Corporation nor the names of its > +# contributors may be used to endorse or promote products derived > +# from this software without specific prior written permission. > +# > +# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS > +# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT > +# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FO= R > +# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT > +# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL= , > +# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT > +# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE= , > +# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON AN= Y > +# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT > +# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE US= E > +# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. > + > +include $(RTE_SDK)/mk/rte.vars.mk > + > +# library name > +LIB =3D librte_gro.a > + > +CFLAGS +=3D -O3 > +CFLAGS +=3D $(WERROR_FLAGS) -I$(SRCDIR) > + > +EXPORT_MAP :=3D rte_gro_version.map > + > +LIBABIVER :=3D 1 > + > +# source files > +SRCS-$(CONFIG_RTE_LIBRTE_GRO) +=3D rte_gro.c > + > +# install this header file > +SYMLINK-$(CONFIG_RTE_LIBRTE_GRO)-include +=3D rte_gro.h > + > +include $(RTE_SDK)/mk/rte.lib.mk > diff --git a/lib/librte_gro/rte_gro.c b/lib/librte_gro/rte_gro.c > new file mode 100644 > index 0000000..33275e8 > --- /dev/null > +++ b/lib/librte_gro/rte_gro.c > @@ -0,0 +1,125 @@ > +/*- > + * BSD LICENSE > + * > + * Copyright(c) 2017 Intel Corporation. All rights reserved. > + * > + * Redistribution and use in source and binary forms, with or without > + * modification, are permitted provided that the following conditions > + * are met: > + * > + * * Redistributions of source code must retain the above copyright > + * notice, this list of conditions and the following disclaimer. > + * * Redistributions in binary form must reproduce the above copyrig= ht > + * notice, this list of conditions and the following disclaimer in > + * the documentation and/or other materials provided with the > + * distribution. > + * * Neither the name of Intel Corporation nor the names of its > + * contributors may be used to endorse or promote products derived > + * from this software without specific prior written permission. > + * > + * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS > + * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT > + * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS F= OR > + * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGH= T > + * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTA= L, > + * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT > + * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF US= E, > + * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON A= NY > + * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT > + * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE U= SE > + * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE= . > + */ > + > +#include > +#include > + > +#include "rte_gro.h" > + > +static gro_tbl_create_fn tbl_create_functions[GRO_TYPE_MAX_NUM]; > +static gro_tbl_destroy_fn tbl_destroy_functions[GRO_TYPE_MAX_NUM]; > + > +struct rte_gro_tbl *rte_gro_tbl_create(uint16_t socket_id, > + uint16_t max_flow_num, > + uint16_t max_item_per_flow, > + uint32_t max_packet_size, > + uint64_t max_timeout_cycles, > + uint64_t desired_gro_types) > +{ > + gro_tbl_create_fn create_tbl_fn; > + struct rte_gro_tbl *gro_tbl; > + uint64_t gro_type_flag =3D 0; > + uint8_t i; > + > + gro_tbl =3D rte_zmalloc_socket(__func__, > + sizeof(struct rte_gro_tbl), > + RTE_CACHE_LINE_SIZE, > + socket_id); > + gro_tbl->max_packet_size =3D max_packet_size; > + gro_tbl->max_timeout_cycles =3D max_timeout_cycles; > + gro_tbl->desired_gro_types =3D desired_gro_types; > + > + for (i =3D 0; i < GRO_TYPE_MAX_NUM; i++) { > + gro_type_flag =3D 1 << i; > + if (desired_gro_types & gro_type_flag) { > + create_tbl_fn =3D tbl_create_functions[i]; > + if (create_tbl_fn) > + create_tbl_fn(socket_id, > + max_flow_num, > + max_item_per_flow); As I understand, create_tbl_fn(0 can fail. You should handle such situation correctly. > + else > + gro_tbl->tbls[i] =3D NULL; > + } > + } > + return gro_tbl; > +} > + > +void rte_gro_tbl_destroy(struct rte_gro_tbl *gro_tbl) > +{ > + gro_tbl_destroy_fn destroy_tbl_fn; > + uint64_t gro_type_flag; > + uint8_t i; > + > + if (gro_tbl =3D=3D NULL) > + return; > + for (i =3D 0; i < GRO_TYPE_MAX_NUM; i++) { > + gro_type_flag =3D 1 << i; > + if (gro_tbl->desired_gro_types & gro_type_flag) { > + destroy_tbl_fn =3D tbl_destroy_functions[i]; > + if (destroy_tbl_fn) > + destroy_tbl_fn(gro_tbl->tbls[i]); > + gro_tbl->tbls[i] =3D NULL; > + } > + } > + rte_free(gro_tbl); > +} > + > +uint16_t > +rte_gro_reassemble_burst(struct rte_mbuf **pkts __rte_unused, > + const uint16_t nb_pkts, > + const struct rte_gro_param param __rte_unused) > +{ > + return nb_pkts; > +} > + > +int rte_gro_reassemble(struct rte_mbuf *pkt __rte_unused, > + struct rte_gro_tbl *gro_tbl __rte_unused) > +{ > + return -1; > +} > + > +uint16_t rte_gro_flush(struct rte_gro_tbl *gro_tbl __rte_unused, > + uint64_t desired_gro_types __rte_unused, > + struct rte_mbuf **out __rte_unused, > + const uint16_t max_nb_out __rte_unused) > +{ > + return 0; > +} > + > +uint16_t > +rte_gro_timeout_flush(struct rte_gro_tbl *gro_tbl __rte_unused, > + uint64_t desired_gro_types __rte_unused, > + struct rte_mbuf **out __rte_unused, > + const uint16_t max_nb_out __rte_unused) > +{ > + return 0; > +} > diff --git a/lib/librte_gro/rte_gro.h b/lib/librte_gro/rte_gro.h > new file mode 100644 > index 0000000..f9d36e8 > --- /dev/null > +++ b/lib/librte_gro/rte_gro.h > @@ -0,0 +1,205 @@ > +/*- > + * BSD LICENSE > + * > + * Copyright(c) 2017 Intel Corporation. All rights reserved. > + * > + * Redistribution and use in source and binary forms, with or without > + * modification, are permitted provided that the following conditions > + * are met: > + * > + * * Redistributions of source code must retain the above copyright > + * notice, this list of conditions and the following disclaimer. > + * * Redistributions in binary form must reproduce the above copyrig= ht > + * notice, this list of conditions and the following disclaimer in > + * the documentation and/or other materials provided with the > + * distribution. > + * * Neither the name of Intel Corporation nor the names of its > + * contributors may be used to endorse or promote products derived > + * from this software without specific prior written permission. > + * > + * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS > + * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT > + * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS F= OR > + * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGH= T > + * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTA= L, > + * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT > + * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF US= E, > + * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON A= NY > + * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT > + * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE U= SE > + * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE= . > + */ > + > +#ifndef _RTE_GRO_H_ > +#define _RTE_GRO_H_ > + > +#ifdef __cplusplus > +extern "C" { > +#endif > + > +/** > + * the max packets number that rte_gro_reassemble_burst can > + * process in each invocation. > + */ > +#define GRO_MAX_BURST_ITEM_NUM 1024UL > + > +/* max number of supported GRO types */ > +#define GRO_TYPE_MAX_NUM 64 > +#define GRO_TYPE_SUPPORT_NUM 0 /**< current supported GRO num */ Herre and everywhere public macros should start with RTE_ prefix to follow DPDK coding style. > + > +/** > + * GRO table, which is used to merge packets. It keeps many reassembly > + * tables of desired GRO types. Applications need to create GRO tables > + * before using rte_gro_reassemble to perform GRO. > + */ > +struct rte_gro_tbl { > + uint64_t desired_gro_types; /**< GRO types to perform */ > + /* max TTL measured in nanosecond */ > + uint64_t max_timeout_cycles; > + /* max length of merged packet measured in byte */ > + uint32_t max_packet_size; > + /* reassebly tables of desired GRO types */ > + void *tbls[GRO_TYPE_MAX_NUM]; > +}; Not sure why do you need to define that structure here. As I understand it is internal to the library. Just declaration should be enough. > + > +struct rte_gro_param { > + uint64_t desired_gro_types; /**< desired GRO types */ > + uint32_t max_packet_size; /**< max length of merged packets */ > + uint16_t max_flow_num; /**< max flow number */ > + uint16_t max_item_per_flow; /**< max packet number per flow */ > +}; > + > +typedef void *(*gro_tbl_create_fn)(uint16_t socket_id, > + uint16_t max_flow_num, > + uint16_t max_item_per_flow); > +typedef void (*gro_tbl_destroy_fn)(void *tbl); Same here - user probably shouldn't see these typedefs, so better to hide them inside. > + > +/** > + * This function create a GRO table, which is used to merge packets. > + * > + * @param socket_id > + * socket index where the Ethernet port connects to. > + * @param max_flow_num > + * max number of flows in the GRO table. > + * @param max_item_per_flow > + * max packet number per flow. We use the value of (max_flow_num * > + * max_item_per_fow) to calculate table size. > + * @param max_packet_size > + * max length of merged packets. Measured in byte. > + * @param max_timeout_cycles > + * max TTL for a packet in the GRO table. It's measured in nanosecond. > + * @param desired_gro_types > + * GRO types to perform. > + * @return > + * if create successfully, return a pointer which points to the GRO > + * table. Otherwise, return NULL. > + */ > +struct rte_gro_tbl *rte_gro_tbl_create(uint16_t socket_id, > + uint16_t max_flow_num, > + uint16_t max_item_per_flow, > + uint32_t max_packet_size, > + uint64_t max_timeout_cycles, > + uint64_t desired_gro_types); Hm, couldn't we have here struct rte_gro_tbl_param * instead of dozen argum= ents? > +/** > + * This function destroys a GRO table. > + */ > +void rte_gro_tbl_destroy(struct rte_gro_tbl *gro_tbl); > + > +/** > + * This is one of the main reassembly APIs, which merges numbers of > + * packets at a time. It assumes that all inputted packets are with > + * correct checksums. That is, applications should guarantee all > + * inputted packets are correct. Besides, it doesn't re-calculate > + * checksums for merged packets. If inputted packets are IP fragmented, > + * this function assumes them are complete (i.e. with L4 header). After > + * finishing processing, it returns all GROed packets to applications > + * immediately. > + * > + * @param pkts > + * a pointer array which points to the packets to reassemble. Besides, > + * it keeps addresses of GROed packets. > + * @param nb_pkts > + * the number of packets to reassemble. > + * @param param > + * applications use it to tell rte_gro_reassemble_burst what rules > + * are demanded. > + * @return > + * the number of packets after GROed. > + */ > +uint16_t rte_gro_reassemble_burst(struct rte_mbuf **pkts, > + const uint16_t nb_pkts, Here and everywhere - no much point in defining intger input parameter (or any other that is passed by value) as const. > + const struct rte_gro_param param); You probably meant 'const struct rte_gro_param *param' here? > + > +/** > + * Reassembly function, which tries to merge the inputted packet with > + * one packet in a given GRO table. This function assumes the inputted > + * packet is with correct checksums. And it won't update checksums if > + * two packets are merged. Besides, if the inputted packet is IP > + * fragmented, this function assumes it's a complete packet (i.e. with > + * L4 header). > + * > + * If the inputted packet doesn't have data or it's with unsupported GRO > + * type, function returns immediately. Otherwise, the inputted packet is > + * either merged or inserted into the table. If applications want get > + * packets in the table, they need to call flush APIs. > + * > + * @param pkt > + * packet to reassemble. > + * @param gro_tbl > + * a pointer points to a GRO table. > + * @return > + * if merge the packet successfully, return a positive value. If fail > + * to merge, return zero. If the packet doesn't have data, or its GRO > + * type is unsupported, return a negative value. > + */ > +int rte_gro_reassemble(struct rte_mbuf *pkt, > + struct rte_gro_tbl *gro_tbl); Ok, and why tbl one can't do bursts? > + > +/** > + * This function flushed packets from reassembly tables of desired GRO > + * types. It won't re-calculate checksums for merged packets in the > + * tables. That is, the returned packets may be with wrong checksums. > + * > + * @param gro_tbl > + * a pointer points to a GRO table object. > + * @param desired_gro_types > + * GRO types whose packets will be flushed. > + * @param out > + * a pointer array that is used to keep flushed packets. > + * @param nb_out > + * the size of out. > + * @return > + * the number of flushed packets. If no packets are flushed, return 0. > + */ > +uint16_t rte_gro_flush(struct rte_gro_tbl *gro_tbl, > + uint64_t desired_gro_types, > + struct rte_mbuf **out, > + const uint16_t max_nb_out); > + > +/** > + * This function flushes the timeout packets from reassembly tables of > + * desired GRO types. It won't re-calculate checksums for merged packets > + * in the tables. That is, the returned packets may be with wrong > + * checksums. > + * > + * @param gro_tbl > + * a pointer points to a GRO table object. > + * @param desired_gro_types > + * rte_gro_timeout_flush only processes packets which belong to the > + * GRO types specified by desired_gro_types. > + * @param out > + * a pointer array that is used to keep flushed packets. > + * @param nb_out > + * the size of out. > + * @return > + * the number of flushed packets. If no packets are flushed, return 0. > + */ > +uint16_t rte_gro_timeout_flush(struct rte_gro_tbl *gro_tbl, > + uint64_t desired_gro_types, > + struct rte_mbuf **out, > + const uint16_t max_nb_out); No point to have 2 flush() functions. I suggest to merge them together. > +#ifdef __cplusplus > +} > +#endif > + > +#endif > diff --git a/lib/librte_gro/rte_gro_version.map b/lib/librte_gro/rte_gro_= version.map > new file mode 100644 > index 0000000..827596b > --- /dev/null > +++ b/lib/librte_gro/rte_gro_version.map > @@ -0,0 +1,12 @@ > +DPDK_17.08 { > + global: > + > + rte_gro_tbl_create; > + rte_gro_tbl_destroy; > + rte_gro_reassemble_burst; > + rte_gro_reassemble; > + rte_gro_flush; > + rte_gro_timeout_flush; > + > + local: *; > +}; > diff --git a/mk/rte.app.mk b/mk/rte.app.mk > index bcaf1b3..fc3776d 100644 > --- a/mk/rte.app.mk > +++ b/mk/rte.app.mk > @@ -98,6 +98,7 @@ _LDLIBS-$(CONFIG_RTE_LIBRTE_RING) +=3D -lrte_= ring > _LDLIBS-$(CONFIG_RTE_LIBRTE_EAL) +=3D -lrte_eal > _LDLIBS-$(CONFIG_RTE_LIBRTE_CMDLINE) +=3D -lrte_cmdline > _LDLIBS-$(CONFIG_RTE_LIBRTE_REORDER) +=3D -lrte_reorder > +_LDLIBS-$(CONFIG_RTE_LIBRTE_GRO) +=3D -lrte_gro >=20 > ifeq ($(CONFIG_RTE_EXEC_ENV_LINUXAPP),y) > _LDLIBS-$(CONFIG_RTE_LIBRTE_KNI) +=3D -lrte_kni > -- > 2.7.4