From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_HELO_NONE, SPF_PASS autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id BE37DC43214 for ; Tue, 10 Aug 2021 07:44:14 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 9426760F38 for ; Tue, 10 Aug 2021 07:44:14 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236932AbhHJHoe (ORCPT ); Tue, 10 Aug 2021 03:44:34 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:53132 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236488AbhHJHoa (ORCPT ); Tue, 10 Aug 2021 03:44:30 -0400 Received: from mail-ej1-x636.google.com (mail-ej1-x636.google.com [IPv6:2a00:1450:4864:20::636]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B0524C061799 for ; Tue, 10 Aug 2021 00:44:08 -0700 (PDT) Received: by mail-ej1-x636.google.com with SMTP id b15so9637810ejg.10 for ; Tue, 10 Aug 2021 00:44:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20150623.gappssmtp.com; s=20150623; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc:content-transfer-encoding; bh=elqUT9stw2J3iR2l4RxHnGypplqsV8TkYIJkIMW8hsY=; b=Dw2anpOlrs9rV2RbKN404Zn18rw9HV6mvfkE8M1kGMV2Ah+OI2rfEhpzsA9qdOawIF k8THbB9AABfuNY1MbNdd0AcLrxpPD246ArW/bPUC+pr03hqytJJD7OFvzo3N0q/tpZGO tH48kCvdWiYFfAcIV1f5yS5oQiNePxkq4fi3pXrnB2oNtBTWdXNxRQpFSbu0OmyXCusR +W05Ly+4a2nE2bv8/HwAQbH8fmJcv8+1juWLj995sdhCvh4wbwIAVnm3NRBqnxRlLOyy KEmSTLyK4NbrCSV7j939f/4v7zlRHpFqgOALTvOj2N1xjrfa63r8X64WZ/4zirZ1FwSG ug5Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc:content-transfer-encoding; bh=elqUT9stw2J3iR2l4RxHnGypplqsV8TkYIJkIMW8hsY=; b=CFmawEo7+bNl7nPRAJxHjAWWTps1qLh++uSfWIJM3aLmIlu+Cij7v4+IPPIlXtXFeh k+7uCQNc+GUHoTyQUB4pyeGTrHw2aEyDu1vXqNN7QTmYlK97QiOgbCHMe9+qsZO3LMM1 PDLH4VS5LUFAy8g7pzR5rxWQA/EW8ZrBBVJnTc/XDPTJpDlLf8wpMfAsnZ8B3zzD58MX G9/5cjT2NaJG6IA1SKdzePFd3g2QA4RzNaTxr7mD1+qk4CWFsj8Nw4U0ImXYB/lON0w0 BXysCNdG0e7YvodVh+C356qzYB/3UF/s6HuTeRm/cyGCNRN2kaEUU6wz+XCTEWyFWK0G aMyQ== X-Gm-Message-State: AOAM532IqUvrDpWMH/FYviCq26DAuaAKqd6gJ9gz75xlAnWwpQnLioi8 PyWBoPmgK4rCDrCPmDdLdenDiAQotgz0C9JiBR5O X-Google-Smtp-Source: ABdhPJx/TbMGjhieQzq1YqRjJBN7a9WTaXbFyhJM7EzEX+MXvLMAXR31L8+X5gM3dqLz8GqZFLUy0w3uIoYuvmPOUqE= X-Received: by 2002:a17:906:8606:: with SMTP id o6mr26642389ejx.247.1628581447154; Tue, 10 Aug 2021 00:44:07 -0700 (PDT) MIME-Version: 1.0 References: <20210729073503.187-1-xieyongji@bytedance.com> <20210729073503.187-2-xieyongji@bytedance.com> <43d88942-1cd3-c840-6fec-4155fd544d80@redhat.com> <6e05e25e-e569-402e-d81b-8ac2cff1c0e8@arm.com> <417ce5af-4deb-5319-78ce-b74fb4dd0582@arm.com> <8c381d3d-9bbd-73d6-9733-0f0b15c40820@redhat.com> In-Reply-To: From: Yongji Xie Date: Tue, 10 Aug 2021 15:43:56 +0800 Message-ID: Subject: Re: [PATCH v10 01/17] iova: Export alloc_iova_fast() and free_iova_fast() To: Jason Wang , Robin Murphy Cc: kvm , "Michael S. Tsirkin" , virtualization , Christian Brauner , Jonathan Corbet , Matthew Wilcox , Christoph Hellwig , Dan Carpenter , Stefano Garzarella , Liu Xiaodong , Joe Perches , Al Viro , Stefan Hajnoczi , songmuchun@bytedance.com, Jens Axboe , He Zhe , Greg KH , Randy Dunlap , linux-kernel , iommu@lists.linux-foundation.org, bcrl@kvack.org, netdev@vger.kernel.org, linux-fsdevel@vger.kernel.org, =?UTF-8?Q?Mika_Penttil=C3=A4?= Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Tue, Aug 10, 2021 at 11:02 AM Jason Wang wrote: > > > =E5=9C=A8 2021/8/9 =E4=B8=8B=E5=8D=881:56, Yongji Xie =E5=86=99=E9=81=93: > > On Thu, Aug 5, 2021 at 9:31 PM Jason Wang wrote: > >> > >> =E5=9C=A8 2021/8/5 =E4=B8=8B=E5=8D=888:34, Yongji Xie =E5=86=99=E9=81= =93: > >>>> My main point, though, is that if you've already got something else > >>>> keeping track of the actual addresses, then the way you're using an > >>>> iova_domain appears to be something you could do with a trivial bitm= ap > >>>> allocator. That's why I don't buy the efficiency argument. The main > >>>> design points of the IOVA allocator are to manage large address spac= es > >>>> while trying to maximise spatial locality to minimise the underlying > >>>> pagetable usage, and allocating with a flexible limit to support > >>>> multiple devices with different addressing capabilities in the same > >>>> address space. If none of those aspects are relevant to the use-case= - > >>>> which AFAICS appears to be true here - then as a general-purpose > >>>> resource allocator it's rubbish and has an unreasonably massive memo= ry > >>>> overhead and there are many, many better choices. > >>>> > >>> OK, I get your point. Actually we used the genpool allocator in the > >>> early version. Maybe we can fall back to using it. > >> > >> I think maybe you can share some perf numbers to see how much > >> alloc_iova_fast() can help. > >> > > I did some fio tests[1] with a ram-backend vduse block device[2]. > > > > Following are some performance data: > > > > numjobs=3D1 numjobs=3D2 numjobs=3D4 = numjobs=3D8 > > iova_alloc_fast 145k iops 265k iops 514k iops 758k io= ps > > > > iova_alloc 137k iops 170k iops 128k iops 113k = iops > > > > gen_pool_alloc 143k iops 270k iops 458k iops 521k iops > > > > The iova_alloc_fast() has the best performance since we always hit the > > per-cpu cache. Regardless of the per-cpu cache, the genpool allocator > > should be better than the iova allocator. > > > I think we see convincing numbers for using iova_alloc_fast() than the > gen_poll_alloc() (45% improvement on job=3D8). > Yes, so alloc_iova_fast() still seems to be the best choice based on performance considerations. Hi Robin, any comments? Thanks, Yongji