From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-0.5 required=3.0 tests=DKIM_INVALID,DKIM_SIGNED, HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7EFFCCA9EB9 for ; Wed, 23 Oct 2019 21:33:22 +0000 (UTC) Received: from dpdk.org (dpdk.org [92.243.14.124]) by mail.kernel.org (Postfix) with ESMTP id 046F02173B for ; Wed, 23 Oct 2019 21:33:21 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=broadcom.com header.i=@broadcom.com header.b="Xn0hMnXP" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 046F02173B Authentication-Results: mail.kernel.org; dmarc=fail (p=quarantine dis=none) header.from=broadcom.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=dev-bounces@dpdk.org Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 477D11C236; Wed, 23 Oct 2019 23:33:21 +0200 (CEST) Received: from mail-il1-f193.google.com (mail-il1-f193.google.com [209.85.166.193]) by dpdk.org (Postfix) with ESMTP id F14A21BF81 for ; Mon, 21 Oct 2019 17:46:40 +0200 (CEST) Received: by mail-il1-f193.google.com with SMTP id z10so12449061ilo.8 for ; Mon, 21 Oct 2019 08:46:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=broadcom.com; s=google; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=4fGpFX0J6vTQHUoGsw2P1ORYZNJkRggsNNMnrN7GsTg=; b=Xn0hMnXP2JD9e2IanhyThV6TNdUJ5bAL7XA5YneTmVTddLkKh5FACCq6WoMBJqT0QX ZfpUcx3YQ6RV6Dks6ovCCYYObKUV1R6VIITKdNzXXQLCvvGo9XYsGH9PZdKkJR0jrIj5 k9Hknay3F3xvb5DL4JNPCBfNDJJJaxn92t+yw= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=4fGpFX0J6vTQHUoGsw2P1ORYZNJkRggsNNMnrN7GsTg=; b=c9L6xH1CQ8qDnEZWdvCizqjcZRADOKDgHUivW4aiWK4fsuy9dRZWwNUmZ/4joER4Bz wkeP+9MFVxgNPg3LVDcCRN4HHLPtziPQYdATVWeD+H515qsmFc9P9F1eNvDbN8LR2EDn pTM0vqL+XGaR9eIhExJhVnYyP0jhdJswpUKdBh4FC3m+4UZThKGSrMk40hcVfoF06P0q DLa67iSrHdbv7hIi78cE1Ja+hLYB8iHI2EX60sXdBUQNv0Uzw/oBPHJREKFSY2NOGYG3 liVkXJJ0ThGcoYSYvrahrMPrt9thCU3gyQ/rd0xdqw5gjB1fJMV2NZ31rm+IwoVnZdSV zWXg== X-Gm-Message-State: APjAAAX/aBVZXDyorIYlfCUhUJ4l1i7UF/8hgBT6CN8sBMF3CkhKI6FW 9T4xOLmOwfRUWurBz6JoGjGmREuHZ/XrPqnNfyIppg== X-Google-Smtp-Source: APXvYqxpJuuebwuqt4qWjZfg9eUUZ18zhga8BjTh+Je4df8cJ/5WTh6mTrX9oPhjamgKF+IOJ5vbUpz9cKYD0v0C2rA= X-Received: by 2002:a92:5819:: with SMTP id m25mr8694340ilb.35.1571672800174; Mon, 21 Oct 2019 08:46:40 -0700 (PDT) MIME-Version: 1.0 References: <20191015053047.52260-1-ajit.khaparde@broadcom.com> <83009bb3-1e0c-a22e-eff8-41a437817cb7@intel.com> In-Reply-To: <83009bb3-1e0c-a22e-eff8-41a437817cb7@intel.com> From: Rajesh Ravi Date: Mon, 21 Oct 2019 21:16:03 +0530 Message-ID: To: "Burakov, Anatoly" Cc: Ajit Khaparde , dev@dpdk.org, Jonathan Richardson , Scott Branden , Vikram Mysore Prakash , Srinath Mannam X-Mailman-Approved-At: Wed, 23 Oct 2019 23:33:20 +0200 Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.15 Subject: Re: [dpdk-dev] [PATCH] eal: add option --iso-cmem for external custom memory X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Thanks Anatoly for prompt response. Sorry for the delayed response, I took some time to reverify with SPDK. Infact, I do want the iommu mapping to be performed. I don't want it to be bypassed by type1_map() [lib/librte_eal/linuxapp/eal/eal_vfio.c] for external memory. Then, if I understood correctly, you suggested to call rte_vfio_dma_map() as an alternative. *Context & clarification* 1) We 're using DPDK to manage/allocate memory for SPDK through heap API. The only DPDK APIs we 're calling are: A)rte_malloc_heap_memory_add() to add external memory to heap. B)rte_malloc_heap_get_socket() & rte_malloc_socket() to allocate memory * Are you suggesting to make a call to rte_vfio_dma_map() from spdk, in addition to the APIs listed under 1)A & 1)B instead of modifying DPDK vfio functions?* Please confirm, Probably I missed the context and might not have understood fully. 2) .dma_user_map_func=vfio_type1_dma_mem_map() is called from 2 paths in dpdk. In either case call to dma_user_map_func() is skipped. A) *During the startup, as you said before:* rte_vfio_setup_device() --> type1_map() B)During allocation event: vfio_mem_event_callback() (type=RTE_MEM_EVENT_ALLOC,..) -->vfio_dma_mem_map() -->dma_user_map_func() ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- *Conclusion* So we have 2 alternatives: A) make additional call to rte_vfio_dma_map() API after adding external memory using rte_malloc_heap_memory_add() API. B) remove msl->external check which bypasses call to dma_user_map_func() in DPDK. I modified DPDK functions [Option B) ]. I guess you 're suggesting option A) Please confirm. Regards, Rajesh On Fri, Oct 18, 2019 at 10:23 PM Burakov, Anatoly wrote: > On 18-Oct-19 11:54 AM, Rajesh Ravi wrote: > > + Srinath > > > > Thanks Anatoly for reviewing this. Please find my reply inline below: > > > > [Anatoly] First of all, what is "iso-cmem"? It doesn't seem to have any > > defined > > meaning nor any relation to any existing functionality, and it's not > > explained anywhere what is "isolated cmem". > > [Rajesh] I 'll correct commit message to include clearer explanation. > > _ > > _ > > _Context & purpose_ > > We 're using this to improve SPDK performance. When NVMe completion > > queues are allocated from a certain memory range, > > out of order competions completions are enabled with our PCIe and it > > improves performance. > > > > _Usage_ > > spdk_env_init()-->(calls rte_eal_init(), and then calls > > )spdk_env_dpdk_post_init() [file: spdk/lib/env_dpdk/init.c] > > --> spdk_mem_map_init() > > [lib/env_dpdk/memory.c]-->map_cmem_virtual_area() [this is our custom > > function] > > In map_cmem_virtual_area(): we 're mmaping anonymous & private region > > and then creating iova table which makes iova addresses which fall in > > custom memory region. > > Then we call rte_malloc_heap_memory_add() and then allocate NVMe > > completion queues from this heap. > > > > [Anatoly] More importantly, why is this necessary? Type1 map only > bypasses > > external segments when adding memory at startup - it doesn't stop you > > from calling rte_vfio_dma_map() to map the memory with VFIO when you > > create the segment. > > [Rajesh] Please correct me if I'm wrong or missing some thing here. > > A) We wanted to create a heap from which we can allocate dynamically > > > > B) I see that type1_map() > > [file:dpdk/lib/librte_eal/linuxapp/eal/eal_vfio.c] getting called once > > rte_malloc_heap_memory_add()() was invoked. > > SPDK uses type1 dma map [spdk/lib/env_dpdk/memory.c] > > vfio_type1_dma_map() is registered for .dma_map_func member > > [dpdk/lib/librte_eal/linuxapp/eal/eal_vfio.c] > > So type1_map is called. Not sure whether we can change this. > > > > I'm not sure i'm following. You mean you *don't* want this mapping to be > performed? > > -- > Thanks, > Anatoly > -- Regards, Rajesh