From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.2 required=3.0 tests=HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_PATCH,MAILING_LIST_MULTI, MENTIONS_GIT_HOSTING,SIGNED_OFF_BY,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_SANE_1 autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id B3389C34022 for ; Mon, 17 Feb 2020 15:49:23 +0000 (UTC) Received: from dpdk.org (dpdk.org [92.243.14.124]) by mail.kernel.org (Postfix) with ESMTP id 3887F20801 for ; Mon, 17 Feb 2020 15:49:23 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 3887F20801 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=ashroe.eu Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=dev-bounces@dpdk.org Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 5B9501DA9F; Mon, 17 Feb 2020 16:49:22 +0100 (CET) Received: from qrelay241.mxroute.com (qrelay241.mxroute.com [172.82.139.241]) by dpdk.org (Postfix) with ESMTP id BE7D034F3 for ; Mon, 17 Feb 2020 16:49:20 +0100 (CET) Received: from filter003.mxroute.com ([168.235.111.26] 168-235-111-26.cloud.ramnode.com) (Authenticated sender: mN4UYu2MZsgR) by qrelay241.mxroute.com (ZoneMTA) with ESMTPA id 17053d6e099000088e.001 for ; Mon, 17 Feb 2020 15:49:16 +0000 X-Zone-Loop: c2ff9f861dd804421ff53b04502dfbe7ff72fa8b7a1d X-Originating-IP: [168.235.111.26] Received: from galaxy.mxroute.com (unknown [23.92.70.113]) by filter003.mxroute.com (Postfix) with ESMTPS id 6F2966002C; Mon, 17 Feb 2020 15:49:14 +0000 (UTC) Received: from [134.191.227.39] by galaxy.mxroute.com with esmtpsa (TLSv1.2:ECDHE-RSA-AES128-GCM-SHA256:128) (Exim 4.91) (envelope-from ) id 1j3iJc-0004DD-DJ; Mon, 17 Feb 2020 10:28:08 -0500 To: Ye Xiaolong Cc: John McNamara , Marko Kovacevic , dev@dpdk.org, Bruce Richardson , Stephen Hemminger , Thomas Monjalon References: <20191129081911.92988-1-xiaolong.ye@intel.com> <20200217024703.80451-1-xiaolong.ye@intel.com> <20200217154406.GA85737@intel.com> From: Ray Kinsella Autocrypt: addr=mdr@ashroe.eu; keydata= mQINBFv8B3wBEAC+5ImcgbIvadt3axrTnt7Sxch3FsmWTTomXfB8YiuHT8KL8L/bFRQSL1f6 ASCHu3M89EjYazlY+vJUWLr0BhK5t/YI7bQzrOuYrl9K94vlLwzD19s/zB/g5YGGR5plJr0s JtJsFGEvF9LL3e+FKMRXveQxBB8A51nAHfwG0WSyx53d61DYz7lp4/Y4RagxaJoHp9lakn8j HV2N6rrnF+qt5ukj5SbbKWSzGg5HQF2t0QQ5tzWhCAKTfcPlnP0GymTBfNMGOReWivi3Qqzr S51Xo7hoGujUgNAM41sxpxmhx8xSwcQ5WzmxgAhJ/StNV9cb3HWIoE5StCwQ4uXOLplZNGnS uxNdegvKB95NHZjRVRChg/uMTGpg9PqYbTIFoPXjuk27sxZLRJRrueg4tLbb3HM39CJwSB++ YICcqf2N+GVD48STfcIlpp12/HI+EcDSThzfWFhaHDC0hyirHxJyHXjnZ8bUexI/5zATn/ux TpMbc/vicJxeN+qfaVqPkCbkS71cHKuPluM3jE8aNCIBNQY1/j87k5ELzg3qaesLo2n1krBH bKvFfAmQuUuJT84/IqfdVtrSCTabvDuNBDpYBV0dGbTwaRfE7i+LiJJclUr8lOvHUpJ4Y6a5 0cxEPxm498G12Z3NoY/mP5soItPIPtLR0rA0fage44zSPwp6cQARAQABtBxSYXkgS2luc2Vs bGEgPG1kckBhc2hyb2UuZXU+iQJUBBMBCAA+FiEEcDUDlKDJaDuJlfZfdJdaH/sCCpsFAlv8 B3wCGyMFCQlmAYAFCwkIBwIGFQoJCAsCBBYCAwECHgECF4AACgkQdJdaH/sCCptdtRAAl0oE msa+djBVYLIsax+0f8acidtWg2l9f7kc2hEjp9h9aZCpPchQvhhemtew/nKavik3RSnLTAyn B3C/0GNlmvI1l5PFROOgPZwz4xhJKGN7jOsRrbkJa23a8ly5UXwF3Vqnlny7D3z+7cu1qq/f VRK8qFyWkAb+xgqeZ/hTcbJUWtW+l5Zb+68WGEp8hB7TuJLEWb4+VKgHTpQ4vElYj8H3Z94a 04s2PJMbLIZSgmKDASnyrKY0CzTpPXx5rSJ1q+B1FCsfepHLqt3vKSALa3ld6bJ8fSJtDUJ7 JLiU8dFZrywgDIVme01jPbjJtUScW6jONLvhI8Z2sheR71UoKqGomMHNQpZ03ViVWBEALzEt TcjWgJFn8yAmxqM4nBnZ+hE3LbMo34KCHJD4eg18ojDt3s9VrDLa+V9fNxUHPSib9FD9UX/1 +nGfU/ZABmiTuUDM7WZdXri7HaMpzDRJUKI6b+/uunF8xH/h/MHW16VuMzgI5dkOKKv1LejD dT5mA4R+2zBS+GsM0oa2hUeX9E5WwjaDzXtVDg6kYq8YvEd+m0z3M4e6diFeLS77/sAOgaYL 92UcoKD+Beym/fVuC6/55a0e12ksTmgk5/ZoEdoNQLlVgd2INtvnO+0k5BJcn66ZjKn3GbEC VqFbrnv1GnA58nEInRCTzR1k26h9nmS5Ag0EW/wHfAEQAMth1vHr3fOZkVOPfod3M6DkQir5 xJvUW5EHgYUjYCPIa2qzgIVVuLDqZgSCCinyooG5dUJONVHj3nCbITCpJp4eB3PI84RPfDcC hf/V34N/Gx5mTeoymSZDBmXT8YtvV/uJvn+LvHLO4ZJdvq5ZxmDyxfXFmkm3/lLw0+rrNdK5 pt6OnVlCqEU9tcDBezjUwDtOahyV20XqxtUttN4kQWbDRkhT+HrA9WN9l2HX91yEYC+zmF1S OhBqRoTPLrR6g4sCWgFywqztpvZWhyIicJipnjac7qL/wRS+wrWfsYy6qWLIV80beN7yoa6v ccnuy4pu2uiuhk9/edtlmFE4dNdoRf7843CV9k1yRASTlmPkU59n0TJbw+okTa9fbbQgbIb1 pWsAuicRHyLUIUz4f6kPgdgty2FgTKuPuIzJd1s8s6p2aC1qo+Obm2gnBTduB+/n1Jw+vKpt 07d+CKEKu4CWwvZZ8ktJJLeofi4hMupTYiq+oMzqH+V1k6QgNm0Da489gXllU+3EFC6W1qKj tkvQzg2rYoWeYD1Qn8iXcO4Fpk6wzylclvatBMddVlQ6qrYeTmSbCsk+m2KVrz5vIyja0o5Y yfeN29s9emXnikmNfv/dA5fpi8XCANNnz3zOfA93DOB9DBf0TQ2/OrSPGjB3op7RCfoPBZ7u AjJ9dM7VABEBAAGJAjwEGAEIACYWIQRwNQOUoMloO4mV9l90l1of+wIKmwUCW/wHfAIbDAUJ CWYBgAAKCRB0l1of+wIKm3KlD/9w/LOG5rtgtCUWPl4B3pZvGpNym6XdK8cop9saOnE85zWf u+sKWCrxNgYkYP7aZrYMPwqDvilxhbTsIJl5HhPgpTO1b0i+c0n1Tij3EElj5UCg3q8mEc17 c+5jRrY3oz77g7E3oPftAjaq1ybbXjY4K32o3JHFR6I8wX3m9wJZJe1+Y+UVrrjY65gZFxcA thNVnWKErarVQGjeNgHV4N1uF3pIx3kT1N4GSnxhoz4Bki91kvkbBhUgYfNflGURfZT3wIKK +d50jd7kqRouXUCzTdzmDh7jnYrcEFM4nvyaYu0JjSS5R672d9SK5LVIfWmoUGzqD4AVmUW8 pcv461+PXchuS8+zpltR9zajl72Q3ymlT4BTAQOlCWkD0snBoKNUB5d2EXPNV13nA0qlm4U2 GpROfJMQXjV6fyYRvttKYfM5xYKgRgtP0z5lTAbsjg9WFKq0Fndh7kUlmHjuAIwKIV4Tzo75 QO2zC0/NTaTjmrtiXhP+vkC4pcrOGNsbHuaqvsc/ZZ0siXyYsqbctj/sCd8ka2r94u+c7o4l BGaAm+FtwAfEAkXHu4y5Phuv2IRR+x1wTey1U1RaEPgN8xq0LQ1OitX4t2mQwjdPihZQBCnZ wzOrkbzlJMNrMKJpEgulmxAHmYJKgvZHXZXtLJSejFjR0GdHJcL5rwVOMWB8cg== Message-ID: <11c0e3ff-f0a4-e252-cf2b-129b9bed1ac4@ashroe.eu> Date: Mon, 17 Feb 2020 15:49:11 +0000 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:68.0) Gecko/20100101 Thunderbird/68.5.0 MIME-Version: 1.0 In-Reply-To: <20200217154406.GA85737@intel.com> Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 7bit X-AuthUser: mdr@ashroe.eu Subject: Re: [dpdk-dev] [PATCH v6] doc: introduce openwrt how-to guide X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" On 17/02/2020 15:44, Ye Xiaolong wrote: > Hi, Ray > > On 02/17, Ray Kinsella wrote: >> On 17/02/2020 02:47, Xiaolong Ye wrote: >>> This doc describes how to enable DPDK on Openwrt in both virtual and >>> physical x86 environment. >>> >>> Signed-off-by: Xiaolong Ye >>> Acked-by: John McNamara >>> --- >>> >>> V6 changes: >>> >>> 1. addressed review comments raised by Thomas >>> >>> V5 changes: >>> >>> 1. improve the doc's grammar and wording according to John's >>> suggestions. >>> >>> V4 changes: >>> >>> 1. add release notes >>> >>> V3 changes: >>> >>> 1. emphasize target select in `OpenWrt configuration` section >>> >>> V2 changes: >>> >>> 1. add meson build steps for dpdk >>> 2. replace steps about build openwrt and running dpdk application with >>> links >>> >>> >>> doc/guides/howto/index.rst | 1 + >>> doc/guides/howto/openwrt.rst | 163 +++++++++++++++++++++++++ >>> doc/guides/rel_notes/release_20_02.rst | 5 + >>> 3 files changed, 169 insertions(+) >>> create mode 100644 doc/guides/howto/openwrt.rst >>> >>> diff --git a/doc/guides/howto/index.rst b/doc/guides/howto/index.rst >>> index a4c131652..5a97ea508 100644 >>> --- a/doc/guides/howto/index.rst >>> +++ b/doc/guides/howto/index.rst >>> @@ -19,3 +19,4 @@ HowTo Guides >>> packet_capture_framework >>> telemetry >>> debug_troubleshoot >>> + openwrt >>> diff --git a/doc/guides/howto/openwrt.rst b/doc/guides/howto/openwrt.rst >>> new file mode 100644 >>> index 000000000..6081f057b >>> --- /dev/null >>> +++ b/doc/guides/howto/openwrt.rst >>> @@ -0,0 +1,163 @@ >>> +.. SPDX-License-Identifier: BSD-3-Clause >>> + Copyright(c) 2019 Intel Corporation. >>> + >>> +Enable DPDK on OpenWrt >>> +====================== >>> + >>> +This document describes how to enable Data Plane Development Kit (DPDK) on >>> +OpenWrt in both a virtual and physical x86 environment. >>> + >>> +Introduction >>> +------------ >>> + >>> +The OpenWrt project is a well-known source-based router OS which provides a >>> +fully writable filesystem with package management. >>> + >>> +Build OpenWrt >>> +------------- >>> + >>> +You can obtain OpenWrt image through https://downloads.openwrt.org/releases. >>> +To fully customize your own OpenWrt, it is highly recommended to build it from >>> +the source code. You can clone the OpenWrt source code as follows: >>> + >>> +.. code-block:: console >>> + >>> + git clone https://git.openwrt.org/openwrt/openwrt.git >>> + >>> +OpenWrt configuration >>> +~~~~~~~~~~~~~~~~~~~~~ >>> + >>> +* Select ``x86`` in ``Target System`` >>> +* Select ``x86_64`` in ``Subtarget`` >>> +* Select ``Build the OpenWrt SDK`` for cross-compilation environment >>> +* Select ``Use glibc`` in ``Advanced configuration options (for developers)`` >>> + then ``ToolChain Options`` and ``C Library implementation`` >>> + >>> +Kernel configuration >>> +~~~~~~~~~~~~~~~~~~~~ >>> + >>> +The following configurations should be enabled: >>> + >>> +* ``CONFIG_VFIO_IOMMU_TYPE1=y`` >>> +* ``CONFIG_VFIO_VIRQFD=y`` >>> +* ``CONFIG_VFIO=y`` >>> +* ``CONFIG_VFIO_NOIOMMU=y`` >>> +* ``CONFIG_VFIO_PCI=y`` >>> +* ``CONFIG_VFIO_PCI_MMAP=y`` >>> +* ``CONFIG_HUGETLBFS=y`` >>> +* ``CONFIG_HUGETLB_PAGE=y`` >>> +* ``CONFIG_PROC_PAGE_MONITOR=y`` >>> + >> >> My 2c is that this is only ok as a short term solution. >> However we really ought to be aiming for upstream support in OpenWRT, right? > > Thanks for the input, I agree that the perfect way would be getting everything > (kernel support, numa lib) ready in OpenWrt, and we can just install DPDK as > shared libraries in OpenWrt. This could be our next step. > > while currently, this is the only way of enabling DPDK on OpenWrt by what > we have in hands, we can update the guide once Openwrt side support is ready. > > Thanks, > Xiaolong Thanks Xiaolong, So to be clear. I see that documentation like this is brittle and has a limited lifespan. I can almost guarantee the next version of DPDK or OpenWRT will break it. (and how would we know?) Much better to do this in the right place - OpenWRT. > >> >> For instance, why would enabling VFIO in OpenWRT necessitate a kernel rebuild. >> We have had Intel IOMMU's for more than 10 years, why wouldn't this be enabled by default? >> I imagine hugepages have been around a similar period. >> >>> +Build steps >>> +~~~~~~~~~~~ >>> + >>> +For detailed OpenWrt build steps and prerequisites, please refer to the >>> +`OpenWrt build guide >>> +`_. >>> + >>> +After the build is completed, you can find the images and SDK in >>> +``/bin/targets/x86/64-glibc/``. >>> + >>> + >>> +DPDK Cross Compilation for OpenWrt >>> +---------------------------------- >> >>> +Pre-requisites >>> +~~~~~~~~~~~~~~ >>> + >>> +NUMA is required to run DPDK in x86. >>> + >>> +.. note:: >>> + >>> + For compiling the NUMA lib, run ``libtool --version`` to ensure the libtool >>> + version >= 2.2, otherwise the compilation will fail with errors. >>> + >>> +.. code-block:: console >>> + >>> + git clone https://github.com/numactl/numactl.git >>> + cd numactl >>> + git checkout v2.0.13 -b v2.0.13 >>> + ./autogen.sh >>> + autoconf -i >>> + export PATH=/glibc/openwrt-sdk-x86-64_gcc-8.3.0_glibc.Linux-x86_64/staging_dir/toolchain-x86_64_gcc-8.3.0_glibc/bin/:$PATH >>> + ./configure CC=x86_64-openwrt-linux-gnu-gcc --prefix= >>> + make install >>> + >>> +The numa header files and lib file is generated in the include and lib folder >>> +respectively under . >> >> Why not an upstream numactl package in OpenWRT, >> that just installs the shared libraries and development headers instead of these gymnastics? >> >>> +Build DPDK >>> +~~~~~~~~~~ >>> + >>> +To cross compile with meson build, you need to write a customized cross file >>> +first. >>> + >>> +.. code-block:: console >>> + >>> + [binaries] >>> + c = 'x86_64-openwrt-linux-gcc' >>> + cpp = 'x86_64-openwrt-linux-cpp' >>> + ar = 'x86_64-openwrt-linux-ar' >>> + strip = 'x86_64-openwrt-linux-strip' >>> + >>> + meson builddir --cross-file openwrt-cross >>> + ninja -C builddir >>> + >>> +.. note:: >>> + >>> + For compiling the igb_uio with the kernel version used in target machine, >>> + you need to explicitly specify kernel_dir in meson_options.txt. >> >> As above, why not have an upstream DPDK package in OpenWRT, >> that just installs the shared libraries and development headers instead of these gymnastics? >> >>> +Running DPDK application on OpenWrt >>> +----------------------------------- >>> + >>> +Virtual machine >>> +~~~~~~~~~~~~~~~ >>> + >>> +* Extract the boot image >>> + >>> +.. code-block:: console >>> + >>> + gzip -d openwrt-x86-64-combined-ext4.img.gz >>> + >>> +* Launch Qemu >>> + >>> +.. code-block:: console >>> + >>> + qemu-system-x86_64 \ >>> + -cpu host \ >>> + -smp 8 \ >>> + -enable-kvm \ >>> + -M q35 \ >>> + -m 2048M \ >>> + -object memory-backend-file,id=mem,size=2048M,mem-path=/tmp/hugepages,share=on \ >>> + -drive file=/openwrt-x86-64-combined-ext4.img,id=d0,if=none,bus=0,unit=0 \ >>> + -device ide-hd,drive=d0,bus=ide.0 \ >>> + -net nic,vlan=0 \ >>> + -net nic,vlan=1 \ >>> + -net user,vlan=1 \ >>> + -display none \ >>> + >>> + >>> +Physical machine >>> +~~~~~~~~~~~~~~~~ >>> + >>> +You can use the ``dd`` tool to write the OpenWrt image to the drive you >>> +want to write the image on. >>> + >>> +.. code-block:: console >>> + >>> + dd if=openwrt-18.06.1-x86-64-combined-squashfs.img of=/dev/sdX >>> + >>> +Where sdX is name of the drive. (You can find it though ``fdisk -l``) >>> + >>> +Running DPDK >>> +~~~~~~~~~~~~ >>> + >>> +More detailed info about how to run a DPDK application please refer to >>> +``Running DPDK Applications`` section of :ref:`the DPDK documentation `. >>> + >>> +.. note:: >>> + >>> + You need to install pre-built NUMA libraries (including soft link) >>> + to /usr/lib64 in OpenWrt. >>> diff --git a/doc/guides/rel_notes/release_20_02.rst b/doc/guides/rel_notes/release_20_02.rst >>> index 78dab7cfc..25df894a9 100644 >>> --- a/doc/guides/rel_notes/release_20_02.rst >>> +++ b/doc/guides/rel_notes/release_20_02.rst >>> @@ -229,6 +229,11 @@ New Features >>> Enhanced the compression performance tool by adding a cycle-count mode >>> which can be used to help measure and tune hardware and software PMDs. >>> >>> +* **Added OpenWrt howto guide.** >>> + >>> + Added document describes how to enable DPDK on OpenWrt in both virtual and >>> + physical machine. >>> + >>> >>> Removed Items >>> ------------- >>> >>