Re: [PATCH 1/1] eal: Don't fail secondary if primary is missing tailqs

[Bruce Richardson <bruce.richardson@intel.com>]

On Thu, Nov 15, 2018 at 04:15:36PM +0000, Burdick, Cliff wrote:
> 
> 
> -----Original Message-----
> From: Luca Boccassi [mailto:bluca@debian.org] 
> Sent: Thursday, November 15, 2018 1:33 AM
> To: Burdick, Cliff; Bruce Richardson
> Cc: Thomas Monjalon; Burakov, Anatoly; dev@dpdk.org
> Subject: Re: [dpdk-dev] [PATCH 1/1] eal: Don't fail secondary if primary is missing tailqs
> 
> > On Wed, 2018-11-14 at 18:24 +0000, Burdick, Cliff wrote:
> > > 
> > > -----Original Message-----
> > > From: Luca Boccassi [mailto:bluca@debian.org]
> > > Sent: Wednesday, November 14, 2018 10:15 AM
> > > To: Burdick, Cliff; Bruce Richardson
> > > Cc: Thomas Monjalon; Burakov, Anatoly; dev@dpdk.org
> > > Subject: Re: [dpdk-dev] [PATCH 1/1] eal: Don't fail secondary if 
> > > primary is missing tailqs
> > > 
> > > > On Wed, 2018-11-14 at 17:40 +0000, Burdick, Cliff wrote:
> > > > > 
> > > > > -----Original Message-----
> > > > > From: Bruce Richardson [mailto:bruce.richardson@intel.com]
> > > > > Sent: Wednesday, November 14, 2018 3:48 AM
> > > > > To: Burdick, Cliff
> > > > > Cc: Thomas Monjalon; Burakov, Anatoly; dev@dpdk.org
> > > > > Subject: Re: [dpdk-dev] [PATCH 1/1] eal: Don't fail secondary if 
> > > > > primary is missing tailqs
> > > > > 
> > > > > On Tue, Nov 13, 2018 at 11:42:51PM +0000, Burdick, Cliff wrote:
> > > > > > > 
> > > > > > > ________________________________________
> > > > > > > From: Thomas Monjalon [thomas@monjalon.net]
> > > > > > > Sent: Tuesday, November 13, 2018 2:18 PM
> > > > > > > To: Burdick, Cliff
> > > > > > > Cc: Burakov, Anatoly; dev@dpdk.org; bruce.richardson@intel.co 
> > > > > > > m
> > > > > > > Subject: Re: [dpdk-dev] [PATCH 1/1] eal: Don't fail secondary 
> > > > > > > if primary is missing tailqs
> > > > > > > 
> > > > > > > 13/11/2018 23:08, Burdick, Cliff:
> > > > > > > > From: Thomas Monjalon [mailto:thomas@monjalon.net]
> > > > > > > > > 13/11/2018 17:38, Burdick, Cliff:
> > > > > > > > > > From: Thomas Monjalon [mailto:thomas@monjalon.net]
> > > > > > > > > > 13/11/2018 16:45, Burdick, Cliff:
> > > > > > > > > > > From: Burakov, Anatoly [mailto:anatoly.burakov@intel.
> > > > > > > > > > > com]
> > > > > > > > > > > > On 13-Nov-18 9:21 AM, Thomas Monjalon wrote:
> > > > > > > > > > > > > 13/11/2018 00:33, Burdick, Cliff:
> > > > > > > > > > > > > > This patch was submitted by Jean Tourrilhes over 
> > > > > > > > > > > > > > two years ago, but didn't receive any responses. 
> > > > > > > > > > > > > > I hit the same issue recently when trying to use 
> > > > > > > > > > > > > > cgo
> > > > > > > > > > > > > > (Golang) as a primary process linked to 
> > > > > > > > > > > > > > libdpdk.a against a C++ application linked 
> > > > > > > > > > > > > > against the same library.> > >
> > > > > > > > > > > > > 
> > > > > > > > > > > > > The question is to know why you don't have the 
> > > > > > > > > > > > > same constructors in primary and seconday?
> > > > > > > > > > > > 
> > > > > > > > > > > > I've hit similar things in the past. I believe it 
> > > > > > > > > > > > was caused by our build system stripping out unused 
> > > > > > > > > > > > libraries (such as
> > > > > > > > > > > > rte_hash) from the binary and thus not calling the 
> > > > > > > > > > > > constructor in the primary, but doing so in the 
> > > > > > > > > > > > secondary (or something to that effect). In any 
> > > > > > > > > > > > case, this is caused by linking different number of 
> > > > > > > > > > > > libraries to primary and secondary, and should 
> > > > > > > > > > > > probably be fixed in the build system, not in the 
> > > > > > > > > > > > tailqs code (unless we specifically support having 
> > > > > > > > > > > > different linked libraries to primary and 
> > > > > > > > > > > > secondary?).> > > >
> > > > > > > > > > > 
> > > > > > > > > > > Right, I think the original author of the patch stated 
> > > > > > > > > > > the reasons in the link I provided. The build system 
> > > > > > > > > > > seems like the most appropriate place to fix it, but 
> > > > > > > > > > > the patch got me going quickly. I think the question 
> > > > > > > > > > > is whether you want DPDK to support these other ways 
> > > > > > > > > > > of linking.
> > > > > > > > > > > I'm
> > > > > > > > > > > certainly not the first to use cgo, since there's a 
> > > > > > > > > > > virtual switch project doing the same:
> > > > > > > > > > > 
> > > > > > > > > > > https://urldefense.proofpoint.com/v2/url?u=https-3A__
> > > > > > > > > > > gith
> > > > > > > > > > > ub.co
> > > > > > > > > > > m_lago
> > > > > > > > > > > pu
> > > > > > > > > > > s_vsw&d=DwICAg&c=jcv3orpCsv7C4ly8-
> > > > > > > > > > > ubDoUfrxF5xIGWmptxGWP5vi5w&r =m1RLQ OG
> > > > > > > > > > > Okz9MauvVLZmiGtyWc5mA7DejbPFNE1IDj-
> > > > > > > > > > > 4&m=hQqVCNwW7eoEzB_hLFK97i8 idS8FI qX
> > > > > > > > > > > oPeclwsIZq7Y&s=BMoBlwkqljwWIBY3SE3pPMCfVnOUlDuZLrno4-
> > > > > > > > > > > SojKM&e=
> > > > > > > > > > > 
> > > > > > > > > > > They don't use primary/secondary processes, though, so 
> > > > > > > > > > > the issue is  never hit. I'm in a situation where 
> > > > > > > > > > > using cgo seemed like the easiest  path to accomplish 
> > > > > > > > > > > what I'm doing since I needed specialized  libraries 
> > > > > > > > > > > for it that were not available in C/C++. At some point 
> > > > > > > > > > > I  bet someone would use Cython to start linking 
> > > > > > > > > > > against DPDK as well,  and we'd likely run into the 
> > > > > > > > > > > same issue.> > > > For sure, we want to support using 
> > > > > > > > > > > DPDK with cgo or cython.
> > > > > > > > > > > But it is not clear what is the relation with not 
> > > > > > > > > > > having the same compilation for primary and secondary.
> > > > > > > > > > > Please
> > > > > > > > > > > could you elaborate?> > >
> > > > > > > > > > 
> > > > > > > > > > Hi Thomas, I think Jean explained it well here:
> > > > > > > > > > https://urldefense.proofpoint.com/v2/url?u=https-3A__de
> > > > > > > > > > v.dp
> > > > > > > > > > dk.narkive.
> > > > > > > > > > com_ZM3a7QD1_dpdk-2Ddev-2Dbug-2Dstatic-2Dconstructors-
> > > > > > > > > > 2Dconsider
> > > > > > > > > > ed-2De
> > > > > > > > > > vil&d=DwICAg&c=jcv3orpCsv7C4ly8- 
> > > > > > > > > > ubDoUfrxF5xIGWmptxGWP5vi5w&r=m1R LQOGOk
> > > > > > > > > > z9MauvVLZmiGtyWc5mA7DejbPFNE1IDj- 
> > > > > > > > > > 4&m=C69wDgrjDX8_oXp1M_3bnmWOOZd GqwBBG 
> > > > > > > > > > 9vzkyGDWGQ&s=YYn2N7WrzJpH1ptNrLZad0nPAQdrUyqBckk2uFuWYP
> > > > > > > > > > Q&e=
> > > > > > > > > > 
> > > > > > > > > > "The build system of the application does not have all 
> > > > > > > > > > the subtelties of the DPDK build system, and ends up 
> > > > > > > > > > including
> > > > > > > > > > *all*
> > > > > > > > > > the constructors, wether they are used or not in the 
> > > > > > > > > > code.
> > > > > > > > > > Moreover, they are included in a different order.
> > > > > > > > > > Actually,
> > > > > > > > > > by default the builds include no constructors at all 
> > > > > > > > > > (which is a big fail), so the library needs to be 
> > > > > > > > > > included with --whole-archive (see Snort DPDK 
> > > > > > > > > > instructions)."
> > > > > > > > > > 
> > > > > > > > > > I will get to the bottom of my exact case to understand 
> > > > > > > > > > what's happening, but my primary application is a cgo 
> > > > > > > > > > application that I'm linking to by using almost exactly 
> > > > > > > > > > the same flags that are used in the DPDK build system to 
> > > > > > > > > > build examples. The DPDK libraries I'm linking against 
> > > > > > > > > > is a single location for both primary and secondary; in 
> > > > > > > > > > other words, I don't build DPDK twice.
> > > > > > > > > > 
> > > > > > > > > > OK I understand, thanks.
> > > > > > > > > > 
> > > > > > > > > > You had alluded to a pkg-config for DPDK in the 2015 
> > > > > > > > > > thread, which cgo can use, but I don't know if that ever 
> > > > > > > > > > was implemented. Cgo can use pkg-config if it's 
> > > > > > > > > > available, otherwise the only tools are specifying 
> > > > > > > > > > LDFLAGS and CFLAGS into their build system.
> > > > > > > > > 
> > > > > > > > > Yes, the right answer is still pkg-config :) The good news 
> > > > > > > > > is that it is now implemented thanks to the meson build
> > > > > > > > > system:
> > > > > > > > >     
> > > > > > > > > https://urldefense.proofpoint.com/v2/url?u=http-3A__git.d
> > > > > > > > > pdk.
> > > > > > > > > org_dp
> > > > > > > > > dk_tree_doc_build-2Dsdk-2Dmeson.txt-
> > > > > > > > > 23n182&d=DwICAg&c=jcv3orpCsv7C4
> > > > > > > > > ly8-
> > > > > > > > > ubDoUfrxF5xIGWmptxGWP5vi5w&r=m1RLQOGOkz9MauvVLZmiGtyWc5mA
> > > > > > > > > 7Dej
> > > > > > > > > bP
> > > > > > > > > FNE1IDj-
> > > > > > > > > > 4&m=C69wDgrjDX8_oXp1M_3bnmWOOZdGqwBBG9vzkyGDWGQ&s=oC86K
> > > > > > > > > > D_R
> > > > > > > > > 
> > > > > > > > > J1T6rfzi3x5zFT1Ri13ELpKmsyFqpgDbgFg&e=
> > > > > > > > 
> > > > > > > > Hi Thomas, are there instructions on how to use pkg-config 
> > > > > > > > to build the mlx4/5 PMD? I noticed a patch was submitted in 
> > > > > > > > September to add support for it, but that link you provided 
> > > > > > > > on using meson doesn't say how to build specific drivers. It 
> > > > > > > > appears to be disabled by default.
> > > > > > > > If the dependency is found, meson will enable the PMD when 
> > > > > > > > building DPDK.
> > > > > > > 
> > > > > > > Do you know where exactly that is? I've been using mlx5 for a 
> > > > > > > while on this system, and I can see that 18.11-rc2 meson
> > > > > > > build+ninja built the pmd, but it's not on the --libs listing
> > > > > > > for
> > > > > > > pkg-config. Does it tell me what I was missing?
> > > > > > > 
> > > > > > 
> > > > > > For dynamic linking of applications, the drivers are not 
> > > > > > normally linked in. Instead, they should be loaded from the 
> > > > > > drivers directory as .so files
> > > > > > - normally by default in EAL as the driver .so's should be 
> > > > > > copied to the EAL_PMD_PATH where EAL finds them automatically. 
> > > > > > [This applies to both meson and make builds, though only meson 
> > > > > > generates the .pc file for pkg-config]
> > > > > > 
> > > > > > If you are doing a static build, then you need to explicitly 
> > > > > > link in the drivers. You can get a list from pkg-config using 
> > > > > > the "
> > > > > > --static" flag in this case. A good example of how to use pkg- 
> > > > > > config in this way can be found in the Makefiles for most 
> > > > > > examples, e.g. examples/helloworld/Makefile, reproduced below.
> > > > > > 
> > > > > > Regards,
> > > > > > /Bruce
> > > > > > 
> > > > > > all: shared
> > > > > > .PHONY: shared static
> > > > > > shared: build/$(APP)-shared
> > > > > >         ln -sf $(APP)-shared build/$(APP)
> > > > > > static: build/$(APP)-static
> > > > > >         ln -sf $(APP)-static build/$(APP)
> > > > > > 
> > > > > > PC_FILE := $(shell pkg-config --path libdpdk) CFLAGS += -O3 
> > > > > > $(shell pkg-config --cflags libdpdk) LDFLAGS_SHARED = $(shell 
> > > > > > pkg- config -- libs libdpdk) LDFLAGS_STATIC = -Wl,-Bstatic 
> > > > > > $(shell pkg-config
> > > > > > --
> > > > > > static --libs libdpdk)
> > > > > > 
> > > > > > build/$(APP)-shared: $(SRCS-y) Makefile $(PC_FILE) | build
> > > > > >         $(CC) $(CFLAGS) $(SRCS-y) -o $@ $(LDFLAGS)
> > > > > > $(LDFLAGS_SHARED)
> > > > > > 
> > > > > > build/$(APP)-static: $(SRCS-y) Makefile $(PC_FILE) | build
> > > > > >         $(CC) $(CFLAGS) $(SRCS-y) -o $@ $(LDFLAGS)
> > > > > > $(LDFLAGS_STATIC)
> > > > > > 
> > > > > > build:
> > > > > >         @mkdir -p $@
> > > > > 
> > > > > Thanks Bruce. I tried getting this to compile using cgo, and it's 
> > > > > causing more pain than it's worth. I used the --static --libs 
> > > > > options, and there were still numerous linker errors, some 
> > > > > specific to mlx, and some not. Specifically libmlx5 and libmnl are 
> > > > > both needed, but they're not part of the linker line from 
> > > > > pkg-config. At this point I'll just break the Go application up 
> > > > > into a separate C application that handles all the DPDK parts, and 
> > > > > send messages between them.
> > > > 
> > > > Hi,
> > > > 
> > > > As far as I can see, both mnl and mlx5 (and all the other reverse
> > > > dependencies) are listed correctly in the libdpdk.pc Libs.private 
> > > > entry, and pkg-config --libs --static libdpdk.pc does print them as 
> > > > intended. This is with 18.11-rc3.
> > > > Are you sure everything is correct (pkg-config path is right, -- 
> > > > static is used, etc)?
> > > > 
> > > > --
> > > > Kind regards,
> > > > Luca Boccassi
> > > 
> > > Hi Luca, here is what pkg-config gives:
> > > 
> > > PKG_CONFIG_PATH=meson-private/ pkg-config --libs --static libdpdk 
> > > -L/usr/local/lib/x86_64-linux-gnu -lrte_eventdev -lrte_power 
> > > -lrte_ip_frag -lrte_hash -lrte_pdump -lrte_eal -lrte_pipeline 
> > > -lrte_table -lrte_bbdev -lrte_bpf -lrte_vhost -lrte_metrics 
> > > -lrte_jobstats -lrte_net -lrte_reorder -lrte_ring -lrte_mempool 
> > > -lrte_security -lrte_compressdev -lrte_rawdev -lrte_mbuf -lrte_kvargs 
> > > -lrte_port -lrte_pci -lrte_ethdev -lrte_gro -lrte_gso -lrte_cryptodev 
> > > -lrte_sched -lrte_latencystats -lrte_efd -lrte_distributor -lrte_meter 
> > > -lrte_acl -lrte_member -lrte_cmdline -lrte_lpm -lrte_kni -lrte_cfgfile 
> > > -lrte_bitratestats -lrte_timer -lrte_flow_classify 
> > > -lrte_mempool_bucket -lrte_pmd_null_crypto -lrte_pmd_failsafe 
> > > -lrte_bus_ifpga -lrte_pmd_atlantic -lrte_pmd_mlx4 -lrte_pmd_vmxnet3 
> > > -lrte_pmd_avp -lrte_common_dpaax -lrte_pmd_ena -lcrypto 
> > > -lrte_bus_fslmc -ldl -lrte_pmd_avf -lrte_pmd_crypto_scheduler 
> > > -lrte_pmd_netvsc -lrte_pmd_vhost -lrte_bus_dpaa -lrte_mempool_dpaa2 
> > > -lrte_common_octeontx -lrte_pmd_liquidio -lrte_pmd_ifc -Wl,-Bdynamic 
> > > -lnuma -lrte_pmd_enic -lrte_common_cpt -pthread 
> > > -lrte_pmd_octeontx_crypto -lrte_pmd_qat -lrte_bus_vmbus -lrte_pmd_null 
> > > -lm -lrte_pmd_dpaa -lrte_bus_vdev -lrte_pmd_dpaa2 
> > > -lrte_pmd_skeleton_event -lrte_pmd_af_packet -lrte_pmd_octeontx 
> > > -lrte_pmd_thunderx -lrte_pmd_ring -lrte_pmd_octeontx_event 
> > > -lrte_pmd_sw_event -lrte_pmd_ark -lrte_mempool_octeontx 
> > > -lrte_pmd_ixgbe -lrte_pmd_kni -lrte_mempool_ring 
> > > -lrte_pmd_virtio_crypto -lrte_mempool_dpaa -lrte_pmd_dpaa2_cmdif 
> > > -lrte_bus_pci -lrte_pmd_opdl_event -lrte_pmd_mlx5 -lrte_pmd_virtio 
> > > -lrte_pmd_tap -lrte_pmd_caam_jr -lrte_pmd_dpaa2_sec 
> > > -lrte_pmd_dpaa2_qdma -lrte_pmd_enetc -lrte_pmd_sfc -lrte_pmd_bnxt 
> > > -lrte_pmd_dpaa2_event -lrte_pmd_cxgbe -Wl,--whole-archive -Wl,--no- 
> > > as-needed -lrte_pmd_e1000 -lrte_pmd_softnic -lrte_pmd_bond 
> > > -lrte_pmd_fm10k -lrte_pmd_i40e -lrte_pmd_nfp -lrte_pmd_qede -Wl,--no- 
> > > whole-archive -lrte_pmd_axgbe -lrte_pmd_ccp -lrte_pmd_ifpga_rawdev 
> > > -lrte_pmd_bbdev_null -lrte_pmd_openssl -lrte_pmd_dsw_event 
> > > -lrte_pmd_dpaa_event -lrte_mempool_stack -lrte_pmd_vdev_netvsc 
> > > -lrte_pmd_dpaa_sec -lrte_pmd_skeleton_rawdev 
> > > -lrte_pmd_octeontx_compress
> > > 
> > > You'll notice there's no mlx5 or mnl in that list. I was using 18.11-
> > > rc2 since I cloned early yesterday. Perhaps meson determined not to 
> > > put it in there for some reason?
> > 
> > Were mlx5/mnl found at meson configure time?
> > 
> > --
> > Kind regards,
> > Luca Boccassi
> 
> Hi Luca, yes, it was:
> 
> Library mnl found: YES
> Library mlx4 found: YES
> Library ibverbs found: YES
> Compiler for C supports argument -Wextra: YES
> Compiler for C supports argument -std=c11: YES
> Compiler for C supports argument -Wno-strict-prototypes: YES
> Compiler for C supports argument -D_BSD_SOURCE: YES
> Compiler for C supports argument -D_DEFAULT_SOURCE: YES
> Compiler for C supports argument -D_XOPEN_SOURCE=600: YES
> Checking whether type "struct mlx4_wqe_lso_seg" has member "mss_hdr_size": YES
> Configuring mlx4_autoconf.h using configuration
> Library mnl found: YES
> Library mlx5 found: YES
> Library ibverbs found: YES
> ...
> PKG_CONFIG_PATH=meson-private/ pkg-config --libs --static libdpdk
> -L/usr/local/lib/x86_64-linux-gnu -lrte_eventdev -lrte_power -lrte_ip_frag -lrte_hash -lrte_pdump -lrte_eal -lrte_pipeline -lrte_table -lrte_bbdev -lrte_bpf -lrte_vhost -lrte_metrics -lrte_jobstats -lrte_net -lrte_reorder -lrte_ring -lrte_mempool -lrte_security -lrte_compressdev -lrte_rawdev -lrte_mbuf -lrte_kvargs -lrte_port -lrte_pci -lrte_ethdev -lrte_gro -lrte_gso -lrte_cryptodev -lrte_sched -lrte_latencystats -lrte_efd -lrte_distributor -lrte_meter -lrte_acl -lrte_member -lrte_cmdline -lrte_lpm -lrte_kni -lrte_cfgfile -lrte_bitratestats -lrte_timer -lrte_flow_classify -lrte_pmd_ccp -lrte_mempool_bucket -lrte_pmd_failsafe -lrte_pmd_netvsc -lrte_pmd_null_crypto -lrte_bus_ifpga -lrte_pmd_atlantic -lrte_pmd_octeontx_event -lrte_pmd_avp -lrte_pmd_mlx4 -lrte_common_dpaax -lrte_pmd_skeleton_rawdev -lrte_pmd_ena -lrte_pmd_opdl_event -lrte_bus_fslmc -lnuma -lrte_pmd_avf -lrte_pmd_crypto_scheduler -lrte_pmd_vhost -lrte_bus_dpaa -lrte_mempool_dpaa2 -lrte_common_octeontx -Wl,--no-as-needed -lcrypto -lrte_pmd_ifc -lrte_pmd_liquidio -lrte_pmd_enic -lrte_common_cpt -Wl,--no-whole-archive -lrte_bus_vmbus -lrte_pmd_octeontx_crypto -lrte_pmd_qat -lrte_pmd_ifpga_rawdev -lrte_pmd_dpaa -lrte_bus_vdev -lrte_pmd_bbdev_null -lrte_pmd_dpaa2 -lrte_pmd_skeleton_event -lrte_pmd_ring -lrte_pmd_af_packet -lrte_pmd_thunderx -lrte_pmd_dpaa_event -lrte_pmd_octeontx_compress -lrte_pmd_dpaa_sec -lrte_pmd_sw_event -Wl,--whole-archive -lrte_pmd_ark -lrte_mempool_octeontx -lrte_pmd_ixgbe -lrte_mempool_ring -lrte_pmd_kni -lrte_pmd_vmxnet3 -lrte_mempool_dpaa -lrte_bus_pci -lrte_pmd_dpaa2_cmdif -lrte_pmd_mlx5 -lrte_pmd_tap -lrte_pmd_caam_jr -lrte_pmd_dpaa2_sec -lm -lrte_pmd_dpaa2_qdma -lrte_pmd_enetc -lrte_pmd_virtio -lrte_pmd_bnxt -lrte_pmd_dpaa2_event -lrte_pmd_sfc -lrte_pmd_cxgbe -pthread -lrte_pmd_e1000 -lrte_pmd_softnic -ldl -lrte_pmd_null -lrte_pmd_bond -lrte_pmd_fm10k -lrte_pmd_i40e -lrte_pmd_nfp -lrte_pmd_qede -lrte_pmd_axgbe -Wl,-Bdynamic -lrte_pmd_openssl -lrte_pmd_octeontx -lrte_pmd_dsw_event -lrte_mempool_stack -lrte_pmd_virtio_crypto -lrte_pmd_vdev_netvsc

Is this with latest DPDK from git? because I see the exact same as Luca:

$ PKG_CONFIG_PATH=build/meson-private/ pkg-config --libs --static libdpdk | grep mlx
-L/usr/local/lib64 -lrte_telemetry -lrte_bpf -lrte_flow_classify -lrte_pipeline -lrte_table -lrte_port -lrte_vhost -lrte_security -lrte_sched -lrte_reorder -lrte_rawdev -lrte_pdump -lrte_power -lrte_meter -lrte_member -lrte_lpm -lrte_latencystats -lrte_kni -lrte_jobstats -lrte_ip_frag -lrte_gso -lrte_gro -lrte_eventdev -lrte_efd -lrte_distributor -lrte_cryptodev -lrte_compressdev -lrte_cfgfile -lrte_bitratestats -lrte_bbdev -lrte_acl -lrte_timer -lrte_hash -lrte_metrics -lrte_pci -lrte_ethdev -lrte_net -lrte_mbuf -lrte_mempool -lrte_ring -lrte_eal -lrte_kvargs -lrte_cmdline -L/usr/local/lib64 -lrte_kvargs -lrte_eal -lrte_ring -lrte_mempool -lrte_mbuf -lrte_pci -lrte_cryptodev -lrte_net -lrte_cmdline -lrte_ethdev -lrte_hash -lrte_timer -lrte_common_dpaax -lrte_eventdev -lrte_rawdev -lrte_bus_dpaa -lrte_bus_fslmc -lrte_bus_pci -lrte_common_octeontx -lrte_bus_vdev -lrte_meter -lrte_sched -lrte_ip_frag -lz -lrte_mempool_dpaa -lrte_mempool_dpaa2 -lrte_vhost -lrte_security -lrte_kni -lmnl -lmlx4 -libverbs -lmnl -lmlx5 -libverbs -lrte_bus_vmbus -lrte_mempool_octeontx -lpcap -lrte_port -lrte_lpm -lrte_acl -lrte_table -lrte_pipeline -lsze2 -lrte_gso -lIPSec_MB -lIPSec_MB -lrte_common_cpt -lrte_reorder -lrte_compressdev -lrte_pmd_dpaa -lrte_pmd_dpaa2 -lrte_pmd_dpaa2_sec -lrte_pmd_octeontx -lrte_bbdev -lrte_bus_ifpga -Wl,--whole-archive -lrte_mempool_bucket -lrte_mempool_ring -lrte_mempool_stack -lrte_pmd_af_packet -lrte_pmd_ark -lrte_pmd_atlantic -lrte_pmd_avf -lrte_pmd_avp -lrte_pmd_axgbe -lrte_pmd_bond -lrte_pmd_bnx2x -lrte_pmd_bnxt -lrte_pmd_cxgbe -lrte_pmd_e1000 -lrte_pmd_ena -lrte_pmd_enetc -lrte_pmd_enic -lrte_pmd_failsafe -lrte_pmd_fm10k -lrte_pmd_i40e -lrte_pmd_ifc -lrte_pmd_ixgbe -lrte_pmd_kmod -lrte_pmd_kni -lrte_pmd_liquidio -lrte_pmd_mlx4 -lrte_pmd_mlx5 -lrte_pmd_netvsc -lrte_pmd_nfp -lrte_pmd_null -lrte_pmd_pcap -lrte_pmd_qede -lrte_pmd_ring -lrte_pmd_sfc -lrte_pmd_softnic -lrte_pmd_szedata2 -lrte_pmd_tap -lrte_pmd_thunderx -lrte_pmd_vdev_netvsc -lrte_pmd_vhost -lrte_pmd_virtio -lrte_pmd_vmxnet3 -lrte_pmd_aesni_gcm -lrte_pmd_aesni_mb -lrte_pmd_caam_jr -lrte_pmd_ccp -lrte_pmd_dpaa_sec -lrte_pmd_null_crypto -lrte_pmd_octeontx_crypto -lrte_pmd_openssl -lrte_pmd_crypto_scheduler -lrte_pmd_virtio_crypto -lrte_pmd_octeontx_compress -lrte_pmd_qat -lrte_pmd_zlib -lrte_pmd_dpaa_event -lrte_pmd_dpaa2_event -lrte_pmd_octeontx_event -lrte_pmd_opdl_event -lrte_pmd_skeleton_event -lrte_pmd_sw_event -lrte_pmd_dsw_event -lrte_pmd_bbdev_null -lrte_pmd_skeleton_rawdev -lrte_pmd_dpaa2_cmdif -lrte_pmd_dpaa2_qdma -lrte_pmd_ifpga_rawdev -lrte_pmd_ioat_copy -Wl,--no-whole-archive -Wl,-Bdynamic -Wl,--no-as-needed -pthread -lm -ldl -lnuma -lbsd -lpcap -lcrypto -lcrypto -lz -lcrypto -ldl -pthread -lz