From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.2 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,NICE_REPLY_A, SIGNED_OFF_BY,SPF_HELO_NONE,SPF_PASS,UNPARSEABLE_RELAY,USER_AGENT_SANE_1 autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id C2299C4346E for ; Sun, 27 Sep 2020 12:04:16 +0000 (UTC) Received: from dpdk.org (dpdk.org [92.243.14.124]) by mail.kernel.org (Postfix) with ESMTP id 0887A2389F for ; Sun, 27 Sep 2020 12:04:15 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=alibaba-inc.com header.i=@alibaba-inc.com header.b="bcIioZkT" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 0887A2389F Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=alibaba-inc.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=dev-bounces@dpdk.org Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 5B01F1D65A; Sun, 27 Sep 2020 14:04:13 +0200 (CEST) Received: from out0-147.mail.aliyun.com (out0-147.mail.aliyun.com [140.205.0.147]) by dpdk.org (Postfix) with ESMTP id 0F6661D658 for ; Sun, 27 Sep 2020 14:04:07 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=alibaba-inc.com; s=default; t=1601208244; h=Subject:To:From:Message-ID:Date:MIME-Version:Content-Type; bh=mvKeljNV05YuCPj8BbjE5PmUml7akd2yiXqMDymhuU0=; b=bcIioZkThUhv8j04RwJ3UmA0FQ4AOXNkeldaPdnXhvD7rMGne36ajl08lvuRRgTO0TPL1sb0PKFsNOFfJBVCZSzJjrCuJ2/sCkLGPz8Fdui1s0ify57ADWusk6VIXtF0dI5qnYg0LCTESZr4+FbRemys4u4jkQAmj4jm6OQfJEE= X-Alimail-AntiSpam: AC=PASS; BC=-1|-1; BR=01201311R191e4; CH=green; DM=||false|; DS=||; FP=0|-1|-1|-1|0|-1|-1|-1; HT=e01a16384; MF=huawei.xhw@alibaba-inc.com; NM=1; PH=DS; RN=8; SR=0; TI=SMTPD_---.IcoVCUX_1601208242; Received: from 30.43.68.45(mailfrom:huawei.xhw@alibaba-inc.com fp:SMTPD_---.IcoVCUX_1601208242) by smtp.aliyun-inc.com(127.0.0.1); Sun, 27 Sep 2020 20:04:03 +0800 To: Ferruh Yigit , dev Cc: "Wang, Zhihong" , "Xia, Chenbo" , Maxime Coquelin , Anatoly Burakov , David Marchand , Gaetan Rivet References: <19840b6a-b08f-4877-a530-15ced259bedf.huawei.xhw@alibaba-inc.com> <3f5bd3a2-8696-237d-2723-35be5a60bb65@alibaba-inc.com> <6b99b215-df24-10d8-5b10-75848e89e572@intel.com> From: "=?UTF-8?B?6LCi5Y2O5LyfKOatpOaXtuatpOWIu++8iQ==?=" Message-ID: <7a33e5b4-ec0c-dfa0-be89-8a09e491ef46@alibaba-inc.com> Date: Sun, 27 Sep 2020 20:03:41 +0800 User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:78.0) Gecko/20100101 Thunderbird/78.2.2 MIME-Version: 1.0 In-Reply-To: <6b99b215-df24-10d8-5b10-75848e89e572@intel.com> Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit Content-Language: en-US Subject: Re: [dpdk-dev] [PATCH] pci: support both PIO and MMIO BAR for legacy virtio on x86 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" On 2020/9/25 2:02, Ferruh Yigit wrote: > On 9/15/2020 4:21 PM, 谢华伟(此时此刻） wrote: >> Hi Ferruh: >> Legacy virtio only supports PIO BAR resource. As we need to create >> lots of virtio devices and PIO resource on x86 is very limited, we >> expose MMIO BAR. >> Kernel support both PIO and MMIO BAR for legacy virtio device. This >> patch deals with two cases in the similar way. >> > > Thanks Huawei, > > I was about to ask the actual motivation for the patch, better to put > this explenation into the commit log. > OK. > If legacy virtio device only supports PIO BAR, how exposing MMIO BAR > helps? Or if there is MMIO BAR available in the device, why not > mapping them instead of using PIO? > Most probably I am missing something but can you please help to > understand. > DPDK virtio-pmd for legacy device only looks for PIO resource, which doesn't work if we expose MMIO bar. virtio-pci kernel driver doesn't have this requirement. It gets IO address either from PIO port or mapped bar. It uses address range to distinguish PIO bar and MMIO bar. That is what this patch fixes. > Also below code unifies vfio & uio cases for PIO into same function. > Are you using igb_uio or both? I am not sure about unifiying the vfio > case, what do you think not touching that case in this patch? All virtio PMD need is to get bar address from pci resouce attribute under sysfs like what kernel driver does. UIO/IGB_UIO/VFIO driver path for virtio bar mapping is too complicated. Actually i think we shouldn't put those mapping functions into PCI layer, because they are only for virtio legacy device, not generic. I don't remember if it is me who put them there. Besides, VFIO path uses vfio fd to read/write IO bar which has performance issue. Syscall is needed to notify virtio backend. I see IGB_UIO is to be removed in this release.  UIO_PCI_GENERIC doesn't support msix. If we expose MMIO bar, we have to use VFIO driver before uio_pci_generic kernel driver is enhanced. So i have to fix vfio path. > > There are a few more comments inline, please check. > > Btw, the patch doesn't apply cleanly, can you please send a new > version on top of latest head of repo? > > I will check if we recently applied some pci related patches. > >> On 2020/9/15 16:18, 谢华伟(此时此刻） wrote: >>>  From d0138f24037d8df14cac04c2c24831e4b5d27b8c Mon Sep 17 00:00:00 2001 >>> From: "huawei.xhw" >>> Date: Mon, 14 Sep 2020 23:44:56 +0800 >>> Subject: [PATCH] pci:  support both PIO and MMIO BAR for legacy >>> virtio on x86 >>> >>> In previous implementation, with igb_uio we get PIO address from >>> igb_uio >>> sysfs entry; with uio_pci_generic, we get PIO address from >>> /proc/ioports. >>> For PIO/MMIO RW, there is different path for different drivers and >>> arch. >>> For VFIO, PIO/MMIO RW is through syscall, which has big performance >>> issue. >>> On X86, it assumes only PIO is supported. >>> >>> This is too much twisted. >>> This patch unifies the way to get both PIO and MMIO address for >>> different driver >>> and arch, all from standard resource attr under pci sysfs. >>> >>> We distinguish PIO and MMIO by their address. It is ugly but works. >>> >>> Signed-off-by: huawei.xhw > > <...> > >>> --- a/drivers/bus/pci/linux/pci_uio.c >>> +++ b/drivers/bus/pci/linux/pci_uio.c >>> @@ -372,52 +372,82 @@ >>>   pci_uio_ioport_map(struct rte_pci_device *dev, int bar, >>>              struct rte_pci_ioport *p) >>>   { > > There are two 'pci_uio_ioport_map()', one for x86 and other for rest. > This one is for x86, since read/write functions for multiple arch > combined together, should non-x86 version of this function updated or > removed completely? > non x86 version has weird behavior. It requires bar with  IO attribute but does the mmap. Besides, it adds mmap address with physical address. Fortunately, all other devices on non x86 platform seems to use MMIO bar. > <...> > >>> +    if (flags & IORESOURCE_IO) { >>> +        iobar = 1; >>> +        base = (unsigned long)phys_addr; >>> +        RTE_LOG(INFO, EAL, "%s(): PIO BAR %08lx detected\n", >>> __func__, base); >>> +    } else if (flags & IORESOURCE_MEM) { >>> +        iobar = 0; >>> +        base = (unsigned long)dev->mem_resource[bar].addr; > > Isn't the 'mem_resource[bar].addr' value set only when it is mapped? > For this case I guess it will be 0x0. If not who maps it first? Yes, it is set only when it is MMIO bar which gets mapped. So when it is MMIO bar, we use the already maped address in else path. > >>> +        RTE_LOG(INFO, EAL, "%s(): MMIO BAR %08lx detected\n", >>> __func__, base); >>> +    } else { >>> +        RTE_LOG(ERR, EAL, "%s(): unknown BAR type\n", __func__); >>> +        goto error; >>> +    } >>> + >>> +    if (iobar && rte_eal_iopl_init() != 0) { >>> +        RTE_LOG(ERR, EAL, "%s(): insufficient ioport permissions >>> for PCI device %s\n", >>> +            __func__, dev->name); >>> +        goto error; >>>       } >>> -    /* ensure we don't get anything funny here, read/write will >>> cast to >>> -     * uin16_t */ >>> -    if (start > UINT16_MAX) >>> -        return -1; >>>       /* FIXME only for primary process ? */ >>>       if (dev->intr_handle.type == RTE_INTR_HANDLE_UNKNOWN) { >>> +        uio_num = pci_get_uio_dev(dev, dirname, sizeof(dirname), 0); >>> +        if (uio_num < 0) >>> +            goto error; > > How this will work with vfio? In original implementation, only uio fixes intr handle. I could add UIO driver check to keep the same behavior.