From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id D1970C433EF for ; Fri, 22 Jul 2022 10:37:44 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:Subject:References: In-Reply-To:Message-ID:Cc:To:From:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=UNK1/8C+vlEous6kL1i6sLlGVKxkH7C7UhJkkATzSLU=; b=rQmr0dbA0F/xDUR6B6N2thdHPx 4kEW5TftIBrBqD+DrxFIubUn4b0A8qIsVIZ1zV6XN7wXyXCHgR9QIcjKSj4KwMT12UAXoa4Pqz4/D YjObWh3MH6VINnxPFCYXqTnE5DNdYv5kTelPdv96hiwAZTs1CugXPoBfIG1i46mny+zJMtFyNMcVF XZ6RAelHgZBkwEh6p3xse1xl6qkXhxh6SDcVCytF3FQkov9xuGnufHV+LfguhPHOKfoXiSc8GYLnL ezHZ0IooRRvfnBsnVKzT7yR07ACd2er/3Hoo0qr6kCIOstnCauoapAb8dvPM3C1RN/MdUneNp4OpC B8BFS1Xg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1oEq1E-002nLt-Gr; Fri, 22 Jul 2022 10:36:28 +0000 Received: from sender4-op-o13.zoho.com ([136.143.188.13]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1oEpw9-002j8Q-KD for linux-arm-kernel@lists.infradead.org; Fri, 22 Jul 2022 10:31:15 +0000 ARC-Seal: i=1; a=rsa-sha256; t=1658485869; cv=none; d=zohomail.com; s=zohoarc; b=VPvhjhlWjLcRkvNB3KRSCdnOLtOtv/j2D8VDofcEJ3bGK5dXk6lA6MfXMAqM8OcEvgpoWVxDzpPdbTxgK2ex6sypuuxTrvoePAo1DzPDqc+obH30uC2IOtwYcnnulHTOXugB7A7CWYF44mJMA1vRYvGXKEdpybKrTuwLuTPlvu0= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1658485869; h=Content-Type:Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:MIME-Version:Message-ID:References:Subject:To; bh=0WbIFpXBjuptl97r9GbNsRftbwYVRlA6RI7T4f09fxg=; b=BFFPVIttP8mSvwgZLSkWaO2cvH/nvKmJMCUoRVprw2xeKge3ygK/LJJqbmsippWFSvm9H0rYxvlJhBRn2wNlUjGUMV38zt8ix0AXyAQ7CLp1OfmsbHA4j0SyG+oQY5sgDAXqzjMaj3SOrv4Msdoh8KLLW/q5oCJXiToeqPx5EiA= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass header.i=linux.beauty; spf=pass smtp.mailfrom=me@linux.beauty; dmarc=pass header.from= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; t=1658485869; s=zmail; d=linux.beauty; i=me@linux.beauty; h=Date:Date:From:From:To:To:Cc:Cc:Message-ID:In-Reply-To:References:Subject:Subject:MIME-Version:Content-Type:Content-Transfer-Encoding:Message-Id:Reply-To; bh=0WbIFpXBjuptl97r9GbNsRftbwYVRlA6RI7T4f09fxg=; b=FcAJoWNPxqZUCjM1p01AZAXBWaVfHrqnwTo93IfwOhA3VyRQRqhyvduyR9axms4s eKpciQo/eQO3A39oqbvhm1z2puR82+IPQppJeFYBblwl0rc5ja4NmLvSArqI709qW1D 1p/VlWEM0eq4/0S2L/AEcMK9N5vOpsZKIaUdPO90= Received: from mail.zoho.com by mx.zohomail.com with SMTP id 1658485867679991.1001092254609; Fri, 22 Jul 2022 03:31:07 -0700 (PDT) Date: Fri, 22 Jul 2022 18:31:07 +0800 From: Li Chen To: "Arnd Bergmann" Cc: "linux-arm-kernel" Message-ID: <18225760475.1298153e822204.8920710097965354462@linux.beauty> In-Reply-To: References: <1821ecc9da6.1058f84302773949.334602897681819490@linux.beauty> <18223d6f513.b214956e3076389.5726369280729584331@linux.beauty> <18224fd588a.ad8716753132108.1325666730311378678@linux.beauty> Subject: Re: Why dma_alloc_coherent don't return direct mapped vaddr? MIME-Version: 1.0 Importance: Medium User-Agent: Zoho Mail X-Mailer: Zoho Mail X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20220722_033114_132663_CBEEFA16 X-CRM114-Status: GOOD ( 33.34 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org ---- On Fri, 22 Jul 2022 17:06:36 +0800 Arnd Bergmann wrote --- > On Fri, Jul 22, 2022 at 10:19 AM Li Chen wrote: > > ---- On Fri, 22 Jul 2022 14:50:17 +0800 Arnd Bergmann wrote --- > > > On Fri, Jul 22, 2022 at 4:57 AM Li Chen wrote: > > > > ---- On Thu, 21 Jul 2022 15:06:57 +0800 Arnd Bergmann wrote --- > > > > > in between. > > > > > > > > Thanks for your answer! My device is a misc character device, just like > > > > https://lwn.net/ml/linux-kernel/20220711122459.13773-5-me@linux.beauty/ > > > > IIUC, its dma_addr is always the same with phy addr. If I want to alloc from > > > > reserved memory and then mmap to userspace with vm_insert_pages, are > > > > cma_alloc/dma_alloc_contigous/dma_alloc_from_contigous better choices? > > > > > > In the driver, you should only ever use dma_alloc_coherent() for getting > > > a coherent DMA buffer, the other functions are just the implementation > > > details behind that. > > > > > > To map this buffer to user space, your mmap() function should call > > > dma_mmap_coherent(), which in turn does the correct translation > > > from device specific dma_addr_t values into pages and uses the > > > correct caching attributes. > > > > Yeah, dma_mmap_coherent() is best if I don't care about direct IO. > > But if we need **direct I/O**, dma_mmap_cohere cannot be used because it uses > > remap_pfn_range internally, which will set vma to be VM_IO and VM_PFNMAP, > > so I think I still have to go back to get struct page from rmem and use > > vm_insert_pages to insert pages into vma, right? > > I'm not entirely sure, but I suspect that direct I/O on pages that are mapped > uncacheable will cause data corruption somewhere as well: The direct i/o > code expects normal page cache pages, but these are clearly not. direct I/O just bypasses page cache, so I think you want to say "normal pages"? At least from my hundreds of attempts on 512M rmem, the data doesn't get corrupted, crc32 of the resulted file is always correct after direct I/O. > Also, the coherent DMA API is not actually meant for transferring large > amounts of data. Agree, that's why I also tried cma API like cma_alloc/dma_alloc_from_contigous and they also worked fine. > My guess is that what you are doing here is to use the > coherent API to map a large buffer uncached and then try to access the > uncached data in user space, which is inherently slow. Using direct I/o > appears to solve the problem by not actually using the uncached mapping > when sending the data to another device, but this is not the right approach. My case is to do direct IO from this reserved memory to NVME, and the throughput is good, about 2.3GB/s, which is almost the same as fio's direct I/O seq write result (Of course, fio uses cached and non-reserved memory). > Do you have an IOMMU, scatter/gather support or similar to back the > device? No. My misc char device is simply a pseudo device and have no real hardware. Our dsp will writing raw data to this rmem, but that is another story, we can ignore it here. > I think the only way to safely do what you want to achieve in way > that is both safe and efficient would be to use normal page cache pages > allocated from user space, ideally using hugepage mappings, and then > mapping those into the device using the streaming DMA API to assign > them to the DMA master with get_user_pages_fast()/dma_map_sg() > and dma_sync_sg_for_{device,cpu}. Thanks for your advice, but unfortunately, dsp can only write to contiguous physical memory(it doesn't know MMU), and pages allocated from userspace are not contiguous on physical memory. Regards, Li _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel