From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S936654AbdAJLCF (ORCPT ); Tue, 10 Jan 2017 06:02:05 -0500 Received: from mout.kundenserver.de ([212.227.126.187]:62907 "EHLO mout.kundenserver.de" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S932470AbdAJLCD (ORCPT ); Tue, 10 Jan 2017 06:02:03 -0500 From: Arnd Bergmann To: Nikita Yushchenko Cc: Christoph Hellwig , linux-arm-kernel@lists.infradead.org, Catalin Marinas , Will Deacon , linux-kernel@vger.kernel.org, linux-renesas-soc@vger.kernel.org, Simon Horman , linux-pci@vger.kernel.org, Bjorn Helgaas , artemi.ivanov@cogentembedded.com, Keith Busch , Jens Axboe , Sagi Grimberg , linux-nvme@lists.infradead.org Subject: Re: NVMe vs DMA addressing limitations Date: Tue, 10 Jan 2017 12:01:05 +0100 Message-ID: <4137257.d2v87kqLLv@wuerfel> User-Agent: KMail/5.1.3 (Linux/4.4.0-34-generic; KDE/5.18.0; x86_64; ; ) In-Reply-To: References: <1483044304-2085-1-git-send-email-nikita.yoush@cogentembedded.com> <20170110070719.GA17208@lst.de> MIME-Version: 1.0 Content-Transfer-Encoding: 7Bit Content-Type: text/plain; charset="us-ascii" X-Provags-ID: V03:K0:Nr65lJ5IEGVp2lyVh4tQluIZcViHDawpFmE3d7IqME13w9QSWnr WFNt6hZxoloNoHlNkDhsQnUnYAkBUoftRNBM/SgBlmwsRiWuXaeNz3oP7TE225Q2nGuatiX K+8l1pO/L0/LGZAzsSCsmjLcJLoLupGjO5EX/277bxePRrb19Qmyf64aaDxEYsVxD3p8MJ5 7gcYTBctao8y0blhg1d+g== X-UI-Out-Filterresults: notjunk:1;V01:K0:GbZHmksLZVs=:RzDCUBvvSaZUo7LxMv6Ysr v5KjrbZocEnD5LBhBEMFJmXUScYSNMuMyXb2IekLiaToKPMBHqwl/qaD9sEDXQ40+RSi+smym 3EM7g1EKR1MNeEjxgcrj1xRdV3/9m31shk2Nxg6QNhTSbtaTVLnfBSWU5oPvtdf4Hw+jec1WZ 7CGHz1T+luXi2NY7cuH3whoWJGT/w3WlBbK7vilmjzhdviahVhLmQ9z/D1ywRh/YxWJHbsL5C GX4iQaeLSB+0GjifOBTeNlx/d4O0TSLK3LGeDFLIof2/FrTaEVCSOHQe/6PfOI9nTt2Tq8JH2 o4eD0kIWD+Z6P1lcFCsqHAZ00/z8rYK7qxVMD6FPdDBSSksgN8/tBn9uwmSiVoYTy9SwqoRmJ EHq0ZCNqr7vzsBwz2rwMN/KA+RroVF3FJNNHQ9cFp+LYF9bOWbW04ead1DIOrN6cf2OuI1yLp F7QeGMXv5WrhDddbgTI0+wMsq/xQ4ba9AdoFvpPRtH3+XzXSeDkld2yVsUavyScXxtN1BT4kd Q7Dl8/0qzahK99H7GcF68TrTpubP+6r/QpkE0yD7hyPm1ok+E/SCk+3r0eaNr/4kkRRqCKlrv lepW01xtBbtSvSOl45Ga/MDzpEu8jlKTDvGoRV6FeqRzRG4DdLHu46yZ7W9Ecylo2xhHdoC7u XPAD4WAUDB9A5GDuK6fcs3oHAslIQi7lAdcBEC5Dj6a30R1HXh6qvPv+rLfcAaiyxKq2kPo7J zInLe4oiknIQS1+5 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Tuesday, January 10, 2017 10:31:47 AM CET Nikita Yushchenko wrote: > Christoph, thanks for clear input. > > Arnd, I think that given this discussion, best short-term solution is > indeed the patch I've submitted yesterday. That is, your version + > coherent mask support. With that, set_dma_mask(DMA_BIT_MASK(64)) will > succeed and hardware with work with swiotlb. Ok, good. > Possible next step is to teach swiotlb to dynamically allocate bounce > buffers within entire arm64's ZONE_DMA. That seems reasonable, yes. We probably have to do both, as there are cases where a device has dma_mask smaller than ZONE_DMA but the swiotlb bounce area is low enough to work anyway. Another workaround me might need is to limit amount of concurrent DMA in the NVMe driver based on some platform quirk. The way that NVMe works, it can have very large amounts of data that is concurrently mapped into the device. SWIOTLB is one case where this currently fails, but another example would be old PowerPC servers that have a 256MB window of virtual I/O addresses per VM guest in their IOMMU. Those will likely fail the same way that your does. > Also there is some hope that R-Car *can* iommu-translate addresses that > PCIe module issues to system bus. Although previous attempts to make > that working failed. Additional research is needed here. Does this IOMMU support remapping data within a virtual machine? I believe there are some that only do one of the two -- either you can have guest machines with DMA access to their low memory, or you can remap data on the fly in the host. Arnd