From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.9 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 303F6C2D0A3 for ; Tue, 3 Nov 2020 19:46:27 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id D9B3120732 for ; Tue, 3 Nov 2020 19:46:26 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="chwpTUtq" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1729686AbgKCTqV (ORCPT ); Tue, 3 Nov 2020 14:46:21 -0500 Received: from us-smtp-delivery-124.mimecast.com ([63.128.21.124]:58684 "EHLO us-smtp-delivery-124.mimecast.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726232AbgKCTqU (ORCPT ); Tue, 3 Nov 2020 14:46:20 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1604432778; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=hXGHvDYwh9NMkyXNB/mXiydEr8BiB4sd5uCcFnHHQZg=; b=chwpTUtqHwO02E0vi3/Q7iXNI88a/J1qSQNV/ZpXJ8DOipuh0xSFZjIo4qlN9FlCW1PLof P9mmx6hwaluMP2xkOYijiBlxSyka+bF+ZDW60DaeVcnBWPyeOCMlbg1ZjdPcII361h0IW2 EyJRna6Q14KM5jGfCYZJuhUGVamgCQ8= Received: from mail-qv1-f71.google.com (mail-qv1-f71.google.com [209.85.219.71]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-48-ROtvMN9jOk6bxa9SIgiIdA-1; Tue, 03 Nov 2020 14:46:16 -0500 X-MC-Unique: ROtvMN9jOk6bxa9SIgiIdA-1 Received: by mail-qv1-f71.google.com with SMTP id d14so2021961qvz.16 for ; Tue, 03 Nov 2020 11:46:16 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:content-transfer-encoding :in-reply-to; bh=hXGHvDYwh9NMkyXNB/mXiydEr8BiB4sd5uCcFnHHQZg=; b=nISvc0kGL3+oLYNkYZJ4swq7G/gaYXp41ekIodmUxcyIVi0F2EdvSTlm6G7vKBszqy v5723KTPs4Az65C1QrWTkIElzZWJs1WL7xwwqgoJsPbOnaWyIpxrrSwEXoySLoCjsZ6l UQXW4dWOwt6SUfkL6EKjsY2Ae/IEgmC+ArHpOoU3KPgR4djiZaLoPBxy6HWZDnf4AqQ/ tplG/2WqNbdPbJ9i6aJoAou51vwkhtinp5FqS2r0niwY2pYNGONRWBvpTDN0YfL2RaXH hhTFAaxktY/8/ua2w7kSDTtd5RLbmFftgKWvQBtCqdNOSg4aOH6Ctk2rxvVYIV6cRd05 QChQ== X-Gm-Message-State: AOAM531EjaOkevXjXnAK83AoXABGQTday57lHgheQP4alEU39x2PWS1F e53pOFZ+qazlyHOJMCc1XhdC0vUgBdBRemaKjKXoPKM4ep49/tCedCPkaGsp2tvGwFxItC0yVoG hIOrjJ7ZnrIeI X-Received: by 2002:ae9:f402:: with SMTP id y2mr20621900qkl.459.1604432776433; Tue, 03 Nov 2020 11:46:16 -0800 (PST) X-Google-Smtp-Source: ABdhPJyHrx/4K1ur2emSikIwCSW6QoSh0HGIlNgos3cwaHoIzJ12L9uFjx7qLeKQUNbiJ0RTQKVuVA== X-Received: by 2002:ae9:f402:: with SMTP id y2mr20621874qkl.459.1604432776110; Tue, 03 Nov 2020 11:46:16 -0800 (PST) Received: from xz-x1 (bras-vprn-toroon474qw-lp130-20-174-93-89-196.dsl.bell.ca. [174.93.89.196]) by smtp.gmail.com with ESMTPSA id i70sm11572985qke.11.2020.11.03.11.46.14 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 03 Nov 2020 11:46:15 -0800 (PST) Date: Tue, 3 Nov 2020 14:46:13 -0500 From: Peter Xu To: Jason Wang Cc: Stefano Garzarella , mst@redhat.com, netdev@vger.kernel.org, Stefan Hajnoczi , kvm@vger.kernel.org, virtualization@lists.linux-foundation.org, linux-kernel@vger.kernel.org Subject: Re: [PATCH] vhost/vsock: add IOTLB API support Message-ID: <20201103194613.GK20600@xz-x1> References: <20201029174351.134173-1-sgarzare@redhat.com> <751cc074-ae68-72c8-71de-a42458058761@redhat.com> <20201030105422.ju2aj2bmwsckdufh@steredhat> <278f4732-e561-2b4f-03ee-b26455760b01@redhat.com> <20201102171104.eiovmkj23fle5ioj@steredhat> <8648a2e3-1052-3b5b-11ce-87628ac8dd33@redhat.com> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <8648a2e3-1052-3b5b-11ce-87628ac8dd33@redhat.com> Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org On Tue, Nov 03, 2020 at 05:04:23PM +0800, Jason Wang wrote: > > On 2020/11/3 上午1:11, Stefano Garzarella wrote: > > On Fri, Oct 30, 2020 at 07:44:43PM +0800, Jason Wang wrote: > > > > > > On 2020/10/30 下午6:54, Stefano Garzarella wrote: > > > > On Fri, Oct 30, 2020 at 06:02:18PM +0800, Jason Wang wrote: > > > > > > > > > > On 2020/10/30 上午1:43, Stefano Garzarella wrote: > > > > > > This patch enables the IOTLB API support for vhost-vsock devices, > > > > > > allowing the userspace to emulate an IOMMU for the guest. > > > > > > > > > > > > These changes were made following vhost-net, in details this patch: > > > > > > - exposes VIRTIO_F_ACCESS_PLATFORM feature and inits the iotlb > > > > > >   device if the feature is acked > > > > > > - implements VHOST_GET_BACKEND_FEATURES and > > > > > >   VHOST_SET_BACKEND_FEATURES ioctls > > > > > > - calls vq_meta_prefetch() before vq processing to prefetch vq > > > > > >   metadata address in IOTLB > > > > > > - provides .read_iter, .write_iter, and .poll callbacks for the > > > > > >   chardev; they are used by the userspace to exchange IOTLB messages > > > > > > > > > > > > This patch was tested with QEMU and a patch applied [1] to fix a > > > > > > simple issue: > > > > > >     $ qemu -M q35,accel=kvm,kernel-irqchip=split \ > > > > > >            -drive file=fedora.qcow2,format=qcow2,if=virtio \ > > > > > >            -device intel-iommu,intremap=on \ > > > > > >            -device vhost-vsock-pci,guest-cid=3,iommu_platform=on > > > > > > > > > > > > > > > Patch looks good, but a question: > > > > > > > > > > It looks to me you don't enable ATS which means vhost won't > > > > > get any invalidation request or did I miss anything? > > > > > > > > > > > > > You're right, I didn't see invalidation requests, only miss and > > > > updates. > > > > Now I have tried to enable 'ats' and 'device-iotlb' but I still > > > > don't see any invalidation. > > > > > > > > How can I test it? (Sorry but I don't have much experience yet > > > > with vIOMMU) > > > > > > > > > I guess it's because the batched unmap. Maybe you can try to use > > > "intel_iommu=strict" in guest kernel command line to see if it > > > works. > > > > > > Btw, make sure the qemu contains the patch [1]. Otherwise ATS won't > > > be enabled for recent Linux Kernel in the guest. > > > > The problem was my kernel, it was built with a tiny configuration. > > Using fedora stock kernel I can see the 'invalidate' requests, but I > > also had the following issues. > > > > Do they make you ring any bells? > > > > $ ./qemu -m 4G -smp 4 -M q35,accel=kvm,kernel-irqchip=split \ > >     -drive file=fedora.qcow2,format=qcow2,if=virtio \ > >     -device intel-iommu,intremap=on,device-iotlb=on \ > >     -device vhost-vsock-pci,guest-cid=6,iommu_platform=on,ats=on,id=v1 > > > >     qemu-system-x86_64: vtd_iova_to_slpte: detected IOVA overflow     > > (iova=0x1d40000030c0) > > > It's a hint that IOVA exceeds the AW. It might be worth to check whether the > missed IOVA reported from IOTLB is legal. Yeah. By default the QEMU vIOMMU should only support 39bits width for guest iova address space. To extend it, we can use: -device intel-iommu,aw-bits=48 So we'll enable 4-level iommu pgtable. Here the iova is obvious longer than this, so it'll be interesting to know why that iova is allocated in the guest driver since the driver should know somehow that this iova is beyond what's supported (guest iommu driver should be able to probe viommu capability on this width information too). -- Peter Xu