From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9CAF5C433FE for ; Thu, 30 Sep 2021 10:33:39 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 8B91161882 for ; Thu, 30 Sep 2021 10:33:39 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1350007AbhI3KfU (ORCPT ); Thu, 30 Sep 2021 06:35:20 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:36338 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1349969AbhI3KfT (ORCPT ); Thu, 30 Sep 2021 06:35:19 -0400 Received: from mail-wm1-x334.google.com (mail-wm1-x334.google.com [IPv6:2a00:1450:4864:20::334]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 60435C06176C for ; Thu, 30 Sep 2021 03:33:36 -0700 (PDT) Received: by mail-wm1-x334.google.com with SMTP id z2so4242592wmc.3 for ; Thu, 30 Sep 2021 03:33:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to; bh=ClOTz/czY4oI64rzJtLLTprqersW1WZ3UxUbkRSunI0=; b=q+0y5cUx0pM5xAXd9rFB+2TbqS16LQDpJqNisDwC+zRbUPYdm0x2tN2I4jGADIKWdo Z6Rmg1LBjvD0U+lbZHVncOeHbWEzo6DTx4j/vHIPLAddqpGbfp9B0SrwLQd9BSPJauZr pgi4zLKE3yUk8HInLzgtqnp4yWEeyTDx+hokIuH2Vqw5Gkh+znIfyFVR68RplyMaQqIi 300GvZwMPWNP88uDYzycq2i/slMp88Ydfvw6DtH8LMR9o9fD5Xq5OjGoWssYpYLGJiw+ 6CFym6xccfUdcJypN02A+RKhGaoOT52czvNe45fwXbPLnSqvMMH5VNPfRn4UdjhwHdq2 DNeQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to; bh=ClOTz/czY4oI64rzJtLLTprqersW1WZ3UxUbkRSunI0=; b=G1YGfCsgQiMBpBgXaOkHvo6Fwzs3Yeqeu0XTt+S0wzIY+EgvE1dCJWHgRmmVxmEOPd v6WQagJ+Eglg1NynWmuAd7XXUkUwtKa5Kgu24q4OLWN5wBYQCA40KzsuvSdXjUUXl5zE Ag3bzI3Vs3kfpFFvleeM5/a7gT/7M32Lv45OWG/kNLHJY2qx4g0mRuwub8oWahxWbbbZ QJm29YBUJ/GZNo84i01TJelArnaDoLDDklyky2SbyF8d5tWp8fSwxBxBJ2GASPZ8X+Qk bVcGBtHjk0VxsVtcLmz9cY4uXRFKuNFJ8ObMCy2aQL/H0DGXeIs2/4IySMlRPyYMAHlx SIQA== X-Gm-Message-State: AOAM530ETy5AUDOlWyb/7ODSuHHf5MwQc4N7+dukv5vKCEPJTejso2eh I2e0bMDIgM7ncxjbssgv5RAEBA== X-Google-Smtp-Source: ABdhPJwhVvXXZXdlPItKlpNndOzQ9KbEhrdXbLjx7tfpvI/2kBpGRZ4Qz4ldiDgmI+V7wQlqYvW+Dg== X-Received: by 2002:a05:600c:3b83:: with SMTP id n3mr4618919wms.50.1632998014911; Thu, 30 Sep 2021 03:33:34 -0700 (PDT) Received: from myrica (cpc92880-cmbg19-2-0-cust679.5-4.cable.virginm.net. [82.27.106.168]) by smtp.gmail.com with ESMTPSA id m4sm4568115wml.28.2021.09.30.03.33.33 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 30 Sep 2021 03:33:34 -0700 (PDT) Date: Thu, 30 Sep 2021 11:33:13 +0100 From: Jean-Philippe Brucker To: "Tian, Kevin" Cc: Jason Gunthorpe , "kvm@vger.kernel.org" , "jasowang@redhat.com" , "kwankhede@nvidia.com" , "hch@lst.de" , "Jiang, Dave" , "Raj, Ashok" , "corbet@lwn.net" , "parav@mellanox.com" , Alex Williamson , "lkml@metux.net" , "david@gibson.dropbear.id.au" , "dwmw2@infradead.org" , "Tian, Jun J" , "linux-kernel@vger.kernel.org" , "lushenming@huawei.com" , "pbonzini@redhat.com" , "robin.murphy@arm.com" Subject: Re: [RFC 10/20] iommu/iommufd: Add IOMMU_DEVICE_GET_INFO Message-ID: References: <20210922152407.1bfa6ff7.alex.williamson@redhat.com> <20210922234954.GB964074@nvidia.com> <20210923112716.GE964074@nvidia.com> <20210923122220.GL964074@nvidia.com> <20210929123630.GS964074@nvidia.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Thu, Sep 30, 2021 at 08:30:42AM +0000, Tian, Kevin wrote: > > From: Jason Gunthorpe > > Sent: Wednesday, September 29, 2021 8:37 PM > > > > On Wed, Sep 29, 2021 at 08:48:28AM +0000, Tian, Kevin wrote: > > > > > ARM: > > > - set to snoop format if IOMMU_CACHE > > > - set to nonsnoop format if !IOMMU_CACHE > > > (in both cases TLP snoop bit is ignored?) > > > > Where do you see this? I couldn't even find this functionality in the > > ARM HW manual?? > > Honestly speaking I'm getting confused by the complex attribute > transformation control (default, replace, combine, input, output, etc.) > in SMMU manual. Above was my impression after last check, but now > I cannot find necessary info to build the same picture (except below > code). :/ > > > > > What I saw is ARM linking the IOMMU_CACHE to a IO PTE bit that causes > > the cache coherence to be disabled, which is not ignoring no snoop. > > My impression was that snoop is one way of implementing cache > coherency and now since the PTE can explicitly specify cache coherency > like below: > > else if (prot & IOMMU_CACHE) > pte |= ARM_LPAE_PTE_MEMATTR_OIWB; > else > pte |= ARM_LPAE_PTE_MEMATTR_NC; > > This setting in concept overrides the snoop attribute from the device thus > make it sort of ignored? To make sure we're talking about the same thing: "the snoop attribute from the device" is the "No snoop" attribute in the PCI TLP, right? The PTE flags define whether the memory access is cache-coherent or not. * WB is cacheable (short for write-back cacheable. Doesn't matter here what OI or RWA mean.) * NC is non-cacheable. | Normal PCI access | No_snoop PCI access -------+-------------------+------------------- PTE WB | Cacheable | Non-cacheable PTE NC | Non-cacheable | Non-cacheable Cacheable memory access participate in cache coherency. Non-cacheable accesses go directly to memory, do not cause cache allocation. On Arm cache coherency is configured through PTE attributes. I don't think PCI No_snoop should be used because it's not necessarily supported throughout the system and, as far as I understand, software can't discover whether it is. [...] > Maybe I'll get a clearer picture on this after understanding the difference > between cache coherency and snoop on ARM. The architecture uses terms "cacheable" and "coherent". The term "snoop" is used when referring specifically to the PCI "No snoop" attribute. It is also used within the interconnect coherency protocols, which are invisible to software. Thanks, Jean