From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2E098C433EF for ; Tue, 26 Apr 2022 16:35:17 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1348540AbiDZQiX (ORCPT ); Tue, 26 Apr 2022 12:38:23 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:59928 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230295AbiDZQiW (ORCPT ); Tue, 26 Apr 2022 12:38:22 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTP id 440EE205C5 for ; Tue, 26 Apr 2022 09:35:14 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1650990913; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=LfsfOstOihKjpZcpPA+lYDNuveMEExmShKa1N/TShik=; b=CzcatVpoHbnyg9FtQ1X6fi6qpL7UYv6tPq3r+lrDOlFGpIZ4RxQvf50GDWv6uyqT9n1dsK L3lHvj87nr5o2tiowytHuCGN3T2YrnFRU7qiLEi67pqNJC6ZEphBKhOifg0RyaG7Tsonm8 umOVfZSZwul0EC2Xbzq8nE2FLjFyAiw= Received: from mail-io1-f72.google.com (mail-io1-f72.google.com [209.85.166.72]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-523-vReuIY8RMgCr6VO7mTX3Jg-1; Tue, 26 Apr 2022 12:35:11 -0400 X-MC-Unique: vReuIY8RMgCr6VO7mTX3Jg-1 Received: by mail-io1-f72.google.com with SMTP id k2-20020a0566022a4200b00654c0f121a9so14590597iov.1 for ; Tue, 26 Apr 2022 09:35:10 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:from:to:cc:subject:message-id:in-reply-to :references:organization:mime-version:content-transfer-encoding; bh=LfsfOstOihKjpZcpPA+lYDNuveMEExmShKa1N/TShik=; b=Y5ZQRThM8VIjbL9aKOaVPqJOSkUEQDvGwxF0OrPBZwDTr8i96kq2IKpLTkGzm/8JPp 67IAK3Rv/r/6CX5m2vgWy9O6jN2pZpT8Aiilfw0RQ+xCTpK5fGUdTz17oeuFu4QW8tcT 4WKQhnBt/Pq/ipsao/LPSK8ZBqWX4krkh2ghZKgQfLxNFGR7i7bSA46W071+kcaPuiHe wdgfuvw/ZtJUNYN615hAjOYK91VV5TxqyH+ljWTiTQGFI2rwqz+NT/lAR90fzLXZxq/g 5tdhXgr+W1hK969CUOPB1RlNViOXrW+ksTswxXZk1SXQPYi/O59XU5CzThe+EsmmBU3v 2/tw== X-Gm-Message-State: AOAM531tEy0V6mWVF8HS/U2mjLjMVPVrjrXyr6ZOJ01t6GYtqXFdVMME q58Z5uk3HPSMJ27myGHkXtTBVKFHc3h+sOd1XCpSS4rLWFw/GmHTzN2wb/T91XsKPFCMxu2D9pD mGhJmC2qrzz7k X-Received: by 2002:a05:6e02:19c8:b0:2cd:6c70:1090 with SMTP id r8-20020a056e0219c800b002cd6c701090mr9792465ill.212.1650990909515; Tue, 26 Apr 2022 09:35:09 -0700 (PDT) X-Google-Smtp-Source: ABdhPJxS/wvN70OT99iYPeBPbnWvoQ3P7FmUM0RaRI3UQnFPCLKb1fMCWp9FKCRoz5y2K67IFl9DgQ== X-Received: by 2002:a05:6e02:19c8:b0:2cd:6c70:1090 with SMTP id r8-20020a056e0219c800b002cd6c701090mr9792439ill.212.1650990909065; Tue, 26 Apr 2022 09:35:09 -0700 (PDT) Received: from redhat.com ([38.15.36.239]) by smtp.gmail.com with ESMTPSA id k22-20020a6b6f16000000b0065434014fd2sm9559873ioc.18.2022.04.26.09.35.08 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 26 Apr 2022 09:35:08 -0700 (PDT) Date: Tue, 26 Apr 2022 10:35:07 -0600 From: Alex Williamson To: Shameerali Kolothum Thodi Cc: "eric.auger@redhat.com" , Yi Liu , "cohuck@redhat.com" , "qemu-devel@nongnu.org" , "david@gibson.dropbear.id.au" , "thuth@redhat.com" , "farman@linux.ibm.com" , "mjrosato@linux.ibm.com" , "akrowiak@linux.ibm.com" , "pasic@linux.ibm.com" , "jjherne@linux.ibm.com" , "jasowang@redhat.com" , "kvm@vger.kernel.org" , "jgg@nvidia.com" , "nicolinc@nvidia.com" , "eric.auger.pro@gmail.com" , "kevin.tian@intel.com" , "chao.p.peng@intel.com" , "yi.y.sun@intel.com" , "peterx@redhat.com" , Zhangfei Gao Subject: Re: [RFC 00/18] vfio: Adopt iommufd Message-ID: <20220426103507.5693a0ca.alex.williamson@redhat.com> In-Reply-To: <4ac4956cfe344326a805966535c1dc43@huawei.com> References: <20220414104710.28534-1-yi.l.liu@intel.com> <4f920d463ebf414caa96419b625632d5@huawei.com> <4ac4956cfe344326a805966535c1dc43@huawei.com> Organization: Red Hat MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org On Tue, 26 Apr 2022 12:43:35 +0000 Shameerali Kolothum Thodi wrote: > > -----Original Message----- > > From: Eric Auger [mailto:eric.auger@redhat.com] > > Sent: 26 April 2022 12:45 > > To: Shameerali Kolothum Thodi ; Yi > > Liu ; alex.williamson@redhat.com; cohuck@redhat.com; > > qemu-devel@nongnu.org > > Cc: david@gibson.dropbear.id.au; thuth@redhat.com; farman@linux.ibm.com; > > mjrosato@linux.ibm.com; akrowiak@linux.ibm.com; pasic@linux.ibm.com; > > jjherne@linux.ibm.com; jasowang@redhat.com; kvm@vger.kernel.org; > > jgg@nvidia.com; nicolinc@nvidia.com; eric.auger.pro@gmail.com; > > kevin.tian@intel.com; chao.p.peng@intel.com; yi.y.sun@intel.com; > > peterx@redhat.com; Zhangfei Gao > > Subject: Re: [RFC 00/18] vfio: Adopt iommufd > > [...] > > > >> > > https://lore.kernel.org/kvm/0-v1-e79cd8d168e8+6-iommufd_jgg@nvidia.com > > >> / > > >> [2] https://github.com/luxis1999/iommufd/tree/iommufd-v5.17-rc6 > > >> [3] https://github.com/luxis1999/qemu/tree/qemu-for-5.17-rc6-vm-rfcv1 > > > Hi, > > > > > > I had a go with the above branches on our ARM64 platform trying to > > pass-through > > > a VF dev, but Qemu reports an error as below, > > > > > > [ 0.444728] hisi_sec2 0000:00:01.0: enabling device (0000 -> 0002) > > > qemu-system-aarch64-iommufd: IOMMU_IOAS_MAP failed: Bad address > > > qemu-system-aarch64-iommufd: vfio_container_dma_map(0xaaaafeb40ce0, > > 0x8000000000, 0x10000, 0xffffb40ef000) = -14 (Bad address) > > > > > > I think this happens for the dev BAR addr range. I haven't debugged the > > kernel > > > yet to see where it actually reports that. > > Does it prevent your assigned device from working? I have such errors > > too but this is a known issue. This is due to the fact P2P DMA is not > > supported yet. > > > > Yes, the basic tests all good so far. I am still not very clear how it works if > the map() fails though. It looks like it fails in, > > iommufd_ioas_map() > iopt_map_user_pages() > iopt_map_pages() > .. > pfn_reader_pin_pages() > > So does it mean it just works because the page is resident()? No, it just means that you're not triggering any accesses that require peer-to-peer DMA support. Any sort of test where the device is only performing DMA to guest RAM, which is by far the standard use case, will work fine. This also doesn't affect vCPU access to BAR space. It's only a failure of the mappings of the BAR space into the IOAS, which is only used when a device tries to directly target another device's BAR space via DMA. Thanks, Alex From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 132A6C433F5 for ; Tue, 26 Apr 2022 17:21:35 +0000 (UTC) Received: from localhost ([::1]:46642 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1njOsX-00030p-VO for qemu-devel@archiver.kernel.org; Tue, 26 Apr 2022 13:21:33 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:47362) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1njO9o-0004NK-7M for qemu-devel@nongnu.org; Tue, 26 Apr 2022 12:35:20 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]:37749) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1njO9h-0004c4-Bw for qemu-devel@nongnu.org; Tue, 26 Apr 2022 12:35:18 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1650990912; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=LfsfOstOihKjpZcpPA+lYDNuveMEExmShKa1N/TShik=; b=dZ0Jxj78zfzzYzSwQDxLfNo4wQTN+eCs3CPsUonHWNozPaTTa5Fov9IMyP4H+BHNPO6Q85 Exm/9u38sG/nUffITC5KIyGNkD2v0ROuQliKoDgQ4RXjxZv+PNI7Usgt+kFZy47s+3Lqtp TwOP9K4J42nnJVOmzRugWwRnOrJT4i8= Received: from mail-io1-f72.google.com (mail-io1-f72.google.com [209.85.166.72]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-523-T5yAq1TsN7ieZyq39qQI7A-1; Tue, 26 Apr 2022 12:35:10 -0400 X-MC-Unique: T5yAq1TsN7ieZyq39qQI7A-1 Received: by mail-io1-f72.google.com with SMTP id q5-20020a0566022f0500b00654a56b1dfbso14552238iow.8 for ; Tue, 26 Apr 2022 09:35:10 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:from:to:cc:subject:message-id:in-reply-to :references:organization:mime-version:content-transfer-encoding; bh=LfsfOstOihKjpZcpPA+lYDNuveMEExmShKa1N/TShik=; b=2iub0UJWJlqZO5LU+3UfQQyPEPCA901RFCwrvAMJU+TX2mtOyHr/0xgJV7oeyHXBLL oSux+A5xlBPYz1QV3BUncrZ3FjnvYK+KsM76dr2qnHO2tfZ1q9X32k2n/3bRgJcyaSxD KWpJyi7qn/kjrlqwJna2UsaYBkb34ez+cnCpLXazaA4U22xrbL28lzFl0c1GSL+vG9R4 nhddV/Oq/M339RpjDNOPIZk68cyi4v0MHXrOLY4TspkB8tomyXo6q5jOhBE4mZ55N+ag d4TbugDuh1gfBX7P8gIxcIjSw9v8SWRmf9PyBvok851izw5IE/JYAxAxHxLcXKudYRp6 IODA== X-Gm-Message-State: AOAM531tkB9zV9KmpX3pEPN+vFiTcZsCBeVWQC6ddGaZBZFNtasutu9Z Dcu86p3htbeqGJt3gS+VvoOYJvdeWUMWDiM+YObDabMQHBImojz8UXTdc4GGm/hwBbKxit92AV3 sM/GxakJmDHEnwSo= X-Received: by 2002:a05:6e02:19c8:b0:2cd:6c70:1090 with SMTP id r8-20020a056e0219c800b002cd6c701090mr9792476ill.212.1650990909520; Tue, 26 Apr 2022 09:35:09 -0700 (PDT) X-Google-Smtp-Source: ABdhPJxS/wvN70OT99iYPeBPbnWvoQ3P7FmUM0RaRI3UQnFPCLKb1fMCWp9FKCRoz5y2K67IFl9DgQ== X-Received: by 2002:a05:6e02:19c8:b0:2cd:6c70:1090 with SMTP id r8-20020a056e0219c800b002cd6c701090mr9792439ill.212.1650990909065; Tue, 26 Apr 2022 09:35:09 -0700 (PDT) Received: from redhat.com ([38.15.36.239]) by smtp.gmail.com with ESMTPSA id k22-20020a6b6f16000000b0065434014fd2sm9559873ioc.18.2022.04.26.09.35.08 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 26 Apr 2022 09:35:08 -0700 (PDT) Date: Tue, 26 Apr 2022 10:35:07 -0600 From: Alex Williamson To: Shameerali Kolothum Thodi Subject: Re: [RFC 00/18] vfio: Adopt iommufd Message-ID: <20220426103507.5693a0ca.alex.williamson@redhat.com> In-Reply-To: <4ac4956cfe344326a805966535c1dc43@huawei.com> References: <20220414104710.28534-1-yi.l.liu@intel.com> <4f920d463ebf414caa96419b625632d5@huawei.com> <4ac4956cfe344326a805966535c1dc43@huawei.com> Organization: Red Hat MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Received-SPF: pass client-ip=170.10.129.124; envelope-from=alex.williamson@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -28 X-Spam_score: -2.9 X-Spam_bar: -- X-Spam_report: (-2.9 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.082, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: "akrowiak@linux.ibm.com" , "jjherne@linux.ibm.com" , "thuth@redhat.com" , Yi Liu , "kvm@vger.kernel.org" , "mjrosato@linux.ibm.com" , "farman@linux.ibm.com" , "jasowang@redhat.com" , "cohuck@redhat.com" , "qemu-devel@nongnu.org" , "peterx@redhat.com" , "pasic@linux.ibm.com" , "eric.auger@redhat.com" , "yi.y.sun@intel.com" , "chao.p.peng@intel.com" , "nicolinc@nvidia.com" , "kevin.tian@intel.com" , "jgg@nvidia.com" , Zhangfei Gao , "eric.auger.pro@gmail.com" , "david@gibson.dropbear.id.au" Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" On Tue, 26 Apr 2022 12:43:35 +0000 Shameerali Kolothum Thodi wrote: > > -----Original Message----- > > From: Eric Auger [mailto:eric.auger@redhat.com] > > Sent: 26 April 2022 12:45 > > To: Shameerali Kolothum Thodi ; Yi > > Liu ; alex.williamson@redhat.com; cohuck@redhat.com; > > qemu-devel@nongnu.org > > Cc: david@gibson.dropbear.id.au; thuth@redhat.com; farman@linux.ibm.com; > > mjrosato@linux.ibm.com; akrowiak@linux.ibm.com; pasic@linux.ibm.com; > > jjherne@linux.ibm.com; jasowang@redhat.com; kvm@vger.kernel.org; > > jgg@nvidia.com; nicolinc@nvidia.com; eric.auger.pro@gmail.com; > > kevin.tian@intel.com; chao.p.peng@intel.com; yi.y.sun@intel.com; > > peterx@redhat.com; Zhangfei Gao > > Subject: Re: [RFC 00/18] vfio: Adopt iommufd > > [...] > > > >> > > https://lore.kernel.org/kvm/0-v1-e79cd8d168e8+6-iommufd_jgg@nvidia.com > > >> / > > >> [2] https://github.com/luxis1999/iommufd/tree/iommufd-v5.17-rc6 > > >> [3] https://github.com/luxis1999/qemu/tree/qemu-for-5.17-rc6-vm-rfcv1 > > > Hi, > > > > > > I had a go with the above branches on our ARM64 platform trying to > > pass-through > > > a VF dev, but Qemu reports an error as below, > > > > > > [ 0.444728] hisi_sec2 0000:00:01.0: enabling device (0000 -> 0002) > > > qemu-system-aarch64-iommufd: IOMMU_IOAS_MAP failed: Bad address > > > qemu-system-aarch64-iommufd: vfio_container_dma_map(0xaaaafeb40ce0, > > 0x8000000000, 0x10000, 0xffffb40ef000) = -14 (Bad address) > > > > > > I think this happens for the dev BAR addr range. I haven't debugged the > > kernel > > > yet to see where it actually reports that. > > Does it prevent your assigned device from working? I have such errors > > too but this is a known issue. This is due to the fact P2P DMA is not > > supported yet. > > > > Yes, the basic tests all good so far. I am still not very clear how it works if > the map() fails though. It looks like it fails in, > > iommufd_ioas_map() > iopt_map_user_pages() > iopt_map_pages() > .. > pfn_reader_pin_pages() > > So does it mean it just works because the page is resident()? No, it just means that you're not triggering any accesses that require peer-to-peer DMA support. Any sort of test where the device is only performing DMA to guest RAM, which is by far the standard use case, will work fine. This also doesn't affect vCPU access to BAR space. It's only a failure of the mappings of the BAR space into the IOAS, which is only used when a device tries to directly target another device's BAR space via DMA. Thanks, Alex