From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, MENTIONS_GIT_HOSTING,SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3F14AC4320A for ; Mon, 9 Aug 2021 05:56:58 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 1B26D61057 for ; Mon, 9 Aug 2021 05:56:58 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233176AbhHIF5F (ORCPT ); Mon, 9 Aug 2021 01:57:05 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:40608 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233022AbhHIF5D (ORCPT ); Mon, 9 Aug 2021 01:57:03 -0400 Received: from mail-ej1-x629.google.com (mail-ej1-x629.google.com [IPv6:2a00:1450:4864:20::629]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id AECD0C0613D3 for ; Sun, 8 Aug 2021 22:56:43 -0700 (PDT) Received: by mail-ej1-x629.google.com with SMTP id d11so2538376eja.8 for ; Sun, 08 Aug 2021 22:56:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20150623.gappssmtp.com; s=20150623; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc:content-transfer-encoding; bh=1QKCwJXH7b2bV95pG+j9QIMKAhZHvf1ykyTmniZz1n8=; b=Tz23BY9lPnQ3vIL8d7cY3bYV2IhYyaRUOu6xQBORJN4cz8W0au9Gfbaf8lAUsNVPjB tAe8PYHGpUriBFSopttHCQqPoMMqR1rexL5EnAAhvV7l1kBuiWYPDjMEaayMcy4xi6aA 5uuCDZ/qs9RJhz3WH8weHBsQyMPk7q4EdO+FETqAWaWTUAwpTlufojrgm+S4Uy1A1zXZ ZIxfVRVGOJA3AjSm25fOEwYePUK2HcOeOIwdB6uMCtposnKf/XtfViVOX2BxRLFe5OIJ L4Gu9IBkOOJTcyhBSto5DUifQ4gSNns3GbgdZ0iDqXV+q6HNfQZu9lYsJfT7RE/U4CCP sTvQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc:content-transfer-encoding; bh=1QKCwJXH7b2bV95pG+j9QIMKAhZHvf1ykyTmniZz1n8=; b=WuKFd2QH1LJsm/ByipdmYT+JEbXMQpXv+YkaE5CXnMAtUcAQPS+fNuqcwJho4nO1lX OsM0MWB/Vm9Sb2PhCTWgljbZ3LArHG65ANt/vSTMUr+/ly1oVJJ6e0/tJxlAA4Tpf7Gg ojNyMEYe2HDObYzO2iWfZkIgmaJCaMtC5xmUJkgJfDm5wtwcLJxW+IW9ng0o4lSJGN2n OwUWqJdWGPbaD0cKzLDxiDgCcYZicl/6VWDSALNU2bQsLh3kcP0C6OBCMV5Ll2Q9qyVx GOlF2sBXNqcr75pb//Lxu8plMVjjHE5WzTdifJ8f3zLuph2Yz69pcK6ZH//ldODmckkr 0v6Q== X-Gm-Message-State: AOAM5312K6qFaO22eD/nLN9Qr/+kz0wtr2KaKuo8x+JWdTuE29GPe+wv YW5HVrSwQv1kUbYiZj7pkvHKUBhVG1GUsR6gTsFG X-Google-Smtp-Source: ABdhPJytSHJOcxF9+rpBixPa7fMmae2NjP3sPUJopLOrs0pFpyMSkYVK2NfzI4kxhNLm4dqCFsDdPFoJRqjc++Vk9j4= X-Received: by 2002:a17:907:3e0d:: with SMTP id hp13mr20504923ejc.372.1628488602064; Sun, 08 Aug 2021 22:56:42 -0700 (PDT) MIME-Version: 1.0 References: <20210729073503.187-1-xieyongji@bytedance.com> <20210729073503.187-2-xieyongji@bytedance.com> <43d88942-1cd3-c840-6fec-4155fd544d80@redhat.com> <6e05e25e-e569-402e-d81b-8ac2cff1c0e8@arm.com> <417ce5af-4deb-5319-78ce-b74fb4dd0582@arm.com> <8c381d3d-9bbd-73d6-9733-0f0b15c40820@redhat.com> In-Reply-To: <8c381d3d-9bbd-73d6-9733-0f0b15c40820@redhat.com> From: Yongji Xie Date: Mon, 9 Aug 2021 13:56:31 +0800 Message-ID: Subject: Re: [PATCH v10 01/17] iova: Export alloc_iova_fast() and free_iova_fast() To: Jason Wang , Robin Murphy Cc: kvm , "Michael S. Tsirkin" , virtualization , Christian Brauner , Jonathan Corbet , Matthew Wilcox , Christoph Hellwig , Dan Carpenter , Stefano Garzarella , Liu Xiaodong , Joe Perches , Al Viro , Stefan Hajnoczi , songmuchun@bytedance.com, Jens Axboe , He Zhe , Greg KH , Randy Dunlap , linux-kernel , iommu@lists.linux-foundation.org, bcrl@kvack.org, netdev@vger.kernel.org, linux-fsdevel@vger.kernel.org, =?UTF-8?Q?Mika_Penttil=C3=A4?= Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Thu, Aug 5, 2021 at 9:31 PM Jason Wang wrote: > > > =E5=9C=A8 2021/8/5 =E4=B8=8B=E5=8D=888:34, Yongji Xie =E5=86=99=E9=81=93: > >> My main point, though, is that if you've already got something else > >> keeping track of the actual addresses, then the way you're using an > >> iova_domain appears to be something you could do with a trivial bitmap > >> allocator. That's why I don't buy the efficiency argument. The main > >> design points of the IOVA allocator are to manage large address spaces > >> while trying to maximise spatial locality to minimise the underlying > >> pagetable usage, and allocating with a flexible limit to support > >> multiple devices with different addressing capabilities in the same > >> address space. If none of those aspects are relevant to the use-case - > >> which AFAICS appears to be true here - then as a general-purpose > >> resource allocator it's rubbish and has an unreasonably massive memory > >> overhead and there are many, many better choices. > >> > > OK, I get your point. Actually we used the genpool allocator in the > > early version. Maybe we can fall back to using it. > > > I think maybe you can share some perf numbers to see how much > alloc_iova_fast() can help. > I did some fio tests[1] with a ram-backend vduse block device[2]. Following are some performance data: numjobs=3D1 numjobs=3D2 numjobs=3D4 numj= obs=3D8 iova_alloc_fast 145k iops 265k iops 514k iops 758k iops iova_alloc 137k iops 170k iops 128k iops 113k iops gen_pool_alloc 143k iops 270k iops 458k iops 521k iops The iova_alloc_fast() has the best performance since we always hit the per-cpu cache. Regardless of the per-cpu cache, the genpool allocator should be better than the iova allocator. [1] fio jobfile: [global] rw=3Drandread direct=3D1 ioengine=3Dlibaio iodepth=3D16 time_based=3D1 runtime=3D60s group_reporting bs=3D4k filename=3D/dev/vda [job] numjobs=3D.. [2] $ qemu-storage-daemon \ --chardev socket,id=3Dcharmonitor,path=3D/tmp/qmp.sock,server,nowait = \ --monitor chardev=3Dcharmonitor \ --blockdev driver=3Dhost_device,cache.direct=3Don,aio=3Dnative,filename=3D/dev/nullb0,= node-name=3Ddisk0 \ --export type=3Dvduse-blk,id=3Dtest,node-name=3Ddisk0,writable=3Don,n= ame=3Dvduse-null,num-queues=3D16,queue-size=3D128 The qemu-storage-daemon can be builded based on the repo: https://github.com/bytedance/qemu/tree/vduse-test. Thanks, Yongji