From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.0 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,MENTIONS_GIT_HOSTING,NICE_REPLY_A,SPF_HELO_NONE,SPF_PASS, USER_AGENT_SANE_1 autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4EAE3C432BE for ; Tue, 10 Aug 2021 03:02:33 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 3237F60FE6 for ; Tue, 10 Aug 2021 03:02:33 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236454AbhHJDCw (ORCPT ); Mon, 9 Aug 2021 23:02:52 -0400 Received: from us-smtp-delivery-124.mimecast.com ([216.205.24.124]:24786 "EHLO us-smtp-delivery-124.mimecast.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236243AbhHJDCu (ORCPT ); Mon, 9 Aug 2021 23:02:50 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1628564548; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=oaO6ycaF20ZgcRI3wGSVNh4K2MMhBW7g6tA5PypVq9o=; b=bagKjRKoJldUOmEUH9H0R9LTfci0nvSg95s7XZjN/aACXXcxKIHgw0Iy7CJvp8TpZF3jLc EPR7W9SBNyzTsCdtkaJbHnmikE/evwmXD2/dBr8L7et4aTThsYgzgSuCSd1J6qqn9sWIt0 Gsd9BTE3qz+AMdfr5GIInToyNWoXxTQ= Received: from mail-pl1-f197.google.com (mail-pl1-f197.google.com [209.85.214.197]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-293-e3WMgQtMMxST4LCHn38iiw-1; Mon, 09 Aug 2021 23:02:27 -0400 X-MC-Unique: e3WMgQtMMxST4LCHn38iiw-1 Received: by mail-pl1-f197.google.com with SMTP id s3-20020a1709029883b029012b41197000so1135633plp.16 for ; Mon, 09 Aug 2021 20:02:27 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:cc:references:from:message-id:date :user-agent:mime-version:in-reply-to:content-transfer-encoding :content-language; bh=oaO6ycaF20ZgcRI3wGSVNh4K2MMhBW7g6tA5PypVq9o=; b=ryonC3DkV5mc1/DhwryhFTtom0UbW/cazbTAWOaY6HsnejWXNTCWc/TsNaZzzBC3j9 dekJDw2MT/sG32RjZBbdGIYl5aXrljlIvEuMWrelt4g62tdcfSBmEP3PJ0jwCLDOeP9b De5YdYyAtjMpK+eo8CR0x25xwqPA4sZjOVssySGF8aeC7oUKwo7hgvDNYZ9J6pcmUqnm SW/z2/VsWKIixxJB3hY5eghoW37A8qQzYc5PdtsKohEGfvY3J4cORApzNAIsmSKXCLSJ 7RerHexRLOrSRLrbT24yI8QX/ojB1LFJILx6aScHile7LpXF63a6peVtM7339pAvBi5V qUqg== X-Gm-Message-State: AOAM533307PdQLe1wXRUfru/IDkAF9y4SbqwGkOxcRYgT48yZUgFratQ 9EhSUJpEF6UNmIavdQJQ7P8SPoIHp3qdkD9EMta7OtzgnekqMOMhLdgmQZJDTMFx1R4aWxCs7hI 6VD48VQFV3a69WHnjmaeNWjH0 X-Received: by 2002:a63:f754:: with SMTP id f20mr131581pgk.385.1628564546404; Mon, 09 Aug 2021 20:02:26 -0700 (PDT) X-Google-Smtp-Source: ABdhPJz4iIBidP48y9gv61bBqE2YoRFI1FZjnrIQXCCS9iv1FkdxvVNARb2MOe/u6xBBvn1K8bmYbw== X-Received: by 2002:a63:f754:: with SMTP id f20mr131560pgk.385.1628564546126; Mon, 09 Aug 2021 20:02:26 -0700 (PDT) Received: from wangxiaodeMacBook-Air.local ([209.132.188.80]) by smtp.gmail.com with ESMTPSA id z18sm17386165pfn.88.2021.08.09.20.02.18 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Mon, 09 Aug 2021 20:02:25 -0700 (PDT) Subject: Re: [PATCH v10 01/17] iova: Export alloc_iova_fast() and free_iova_fast() To: Yongji Xie , Robin Murphy Cc: kvm , "Michael S. Tsirkin" , virtualization , Christian Brauner , Jonathan Corbet , Matthew Wilcox , Christoph Hellwig , Dan Carpenter , Stefano Garzarella , Liu Xiaodong , Joe Perches , Al Viro , Stefan Hajnoczi , songmuchun@bytedance.com, Jens Axboe , He Zhe , Greg KH , Randy Dunlap , linux-kernel , iommu@lists.linux-foundation.org, bcrl@kvack.org, netdev@vger.kernel.org, linux-fsdevel@vger.kernel.org, =?UTF-8?Q?Mika_Penttil=c3=a4?= References: <20210729073503.187-1-xieyongji@bytedance.com> <20210729073503.187-2-xieyongji@bytedance.com> <43d88942-1cd3-c840-6fec-4155fd544d80@redhat.com> <6e05e25e-e569-402e-d81b-8ac2cff1c0e8@arm.com> <417ce5af-4deb-5319-78ce-b74fb4dd0582@arm.com> <8c381d3d-9bbd-73d6-9733-0f0b15c40820@redhat.com> From: Jason Wang Message-ID: Date: Tue, 10 Aug 2021 11:02:14 +0800 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:78.0) Gecko/20100101 Thunderbird/78.12.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit Content-Language: en-US Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org 在 2021/8/9 下午1:56, Yongji Xie 写道: > On Thu, Aug 5, 2021 at 9:31 PM Jason Wang wrote: >> >> 在 2021/8/5 下午8:34, Yongji Xie 写道: >>>> My main point, though, is that if you've already got something else >>>> keeping track of the actual addresses, then the way you're using an >>>> iova_domain appears to be something you could do with a trivial bitmap >>>> allocator. That's why I don't buy the efficiency argument. The main >>>> design points of the IOVA allocator are to manage large address spaces >>>> while trying to maximise spatial locality to minimise the underlying >>>> pagetable usage, and allocating with a flexible limit to support >>>> multiple devices with different addressing capabilities in the same >>>> address space. If none of those aspects are relevant to the use-case - >>>> which AFAICS appears to be true here - then as a general-purpose >>>> resource allocator it's rubbish and has an unreasonably massive memory >>>> overhead and there are many, many better choices. >>>> >>> OK, I get your point. Actually we used the genpool allocator in the >>> early version. Maybe we can fall back to using it. >> >> I think maybe you can share some perf numbers to see how much >> alloc_iova_fast() can help. >> > I did some fio tests[1] with a ram-backend vduse block device[2]. > > Following are some performance data: > > numjobs=1 numjobs=2 numjobs=4 numjobs=8 > iova_alloc_fast 145k iops 265k iops 514k iops 758k iops > > iova_alloc 137k iops 170k iops 128k iops 113k iops > > gen_pool_alloc 143k iops 270k iops 458k iops 521k iops > > The iova_alloc_fast() has the best performance since we always hit the > per-cpu cache. Regardless of the per-cpu cache, the genpool allocator > should be better than the iova allocator. I think we see convincing numbers for using iova_alloc_fast() than the gen_poll_alloc() (45% improvement on job=8). Thanks > > [1] fio jobfile: > > [global] > rw=randread > direct=1 > ioengine=libaio > iodepth=16 > time_based=1 > runtime=60s > group_reporting > bs=4k > filename=/dev/vda > [job] > numjobs=.. > > [2] $ qemu-storage-daemon \ > --chardev socket,id=charmonitor,path=/tmp/qmp.sock,server,nowait \ > --monitor chardev=charmonitor \ > --blockdev > driver=host_device,cache.direct=on,aio=native,filename=/dev/nullb0,node-name=disk0 > \ > --export type=vduse-blk,id=test,node-name=disk0,writable=on,name=vduse-null,num-queues=16,queue-size=128 > > The qemu-storage-daemon can be builded based on the repo: > https://github.com/bytedance/qemu/tree/vduse-test. > > Thanks, > Yongji >