From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.5 required=3.0 tests=HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_PASS,URIBL_BLOCKED,USER_AGENT_MUTT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id CB24CC43441 for ; Mon, 19 Nov 2018 16:49:07 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 90AEC208E3 for ; Mon, 19 Nov 2018 16:49:07 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 90AEC208E3 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2389160AbeKTDNS (ORCPT ); Mon, 19 Nov 2018 22:13:18 -0500 Received: from mx1.redhat.com ([209.132.183.28]:48672 "EHLO mx1.redhat.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S2389110AbeKTDNQ (ORCPT ); Mon, 19 Nov 2018 22:13:16 -0500 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.phx2.redhat.com [10.5.11.12]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id F18AFC0C18B0; Mon, 19 Nov 2018 16:49:02 +0000 (UTC) Received: from redhat.com (ovpn-124-1.rdu2.redhat.com [10.10.124.1]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 13FF365F56; Mon, 19 Nov 2018 16:48:55 +0000 (UTC) Date: Mon, 19 Nov 2018 11:48:54 -0500 From: Jerome Glisse To: Leon Romanovsky Cc: Kenneth Lee , Tim Sell , linux-doc@vger.kernel.org, Alexander Shishkin , Zaibo Xu , zhangfei.gao@foxmail.com, linuxarm@huawei.com, haojian.zhuang@linaro.org, Christoph Lameter , Hao Fang , Gavin Schenk , RDMA mailing list , Zhou Wang , Jason Gunthorpe , Doug Ledford , Uwe =?iso-8859-1?Q?Kleine-K=F6nig?= , David Kershner , Kenneth Lee , Johan Hovold , Cyrille Pitchen , Sagar Dharia , Jens Axboe , guodong.xu@linaro.org, linux-netdev , Randy Dunlap , linux-kernel@vger.kernel.org, Vinod Koul , linux-crypto@vger.kernel.org, Philippe Ombredanne , Sanyog Kale , "David S. Miller" , linux-accelerators@lists.ozlabs.org Subject: Re: [RFCv3 PATCH 1/6] uacce: Add documents for WarpDrive/uacce Message-ID: <20181119164853.GA4593@redhat.com> References: <20181112075807.9291-1-nek.in.cn@gmail.com> <20181112075807.9291-2-nek.in.cn@gmail.com> <20181113002354.GO3695@mtr-leonro.mtl.com> <95310df4-b32c-42f0-c750-3ad5eb89b3dd@gmail.com> <20181114160017.GI3759@mtr-leonro.mtl.com> <20181115085109.GD157308@Turing-Arch-b> <20181115145455.GN3759@mtr-leonro.mtl.com> <20181119091405.GE157308@Turing-Arch-b> <20181119091910.GF157308@Turing-Arch-b> <20181119104801.GF8268@mtr-leonro.mtl.com> MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <20181119104801.GF8268@mtr-leonro.mtl.com> User-Agent: Mutt/1.10.1 (2018-07-13) X-Scanned-By: MIMEDefang 2.79 on 10.5.11.12 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.32]); Mon, 19 Nov 2018 16:49:04 +0000 (UTC) Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Mon, Nov 19, 2018 at 12:48:01PM +0200, Leon Romanovsky wrote: > On Mon, Nov 19, 2018 at 05:19:10PM +0800, Kenneth Lee wrote: > > On Mon, Nov 19, 2018 at 05:14:05PM +0800, Kenneth Lee wrote: > > > On Thu, Nov 15, 2018 at 04:54:55PM +0200, Leon Romanovsky wrote: > > > > On Thu, Nov 15, 2018 at 04:51:09PM +0800, Kenneth Lee wrote: > > > > > On Wed, Nov 14, 2018 at 06:00:17PM +0200, Leon Romanovsky wrote: > > > > > > On Wed, Nov 14, 2018 at 10:58:09AM +0800, Kenneth Lee wrote: > > > > > > > > On Mon, Nov 12, 2018 at 03:58:02PM +0800, Kenneth Lee wrote: [...] > > > > memory exposed to user is properly protected from security point of view. > > > > 3. "stop using the page for a while for the copying" - I'm not fully > > > > understand this claim, maybe this article will help you to better > > > > describe : https://lwn.net/Articles/753027/ > > > > > > This topic was being discussed in RFCv2. The key problem here is that: > > > > > > The device need to hold the memory for its own calculation, but the CPU/software > > > want to stop it for a while for synchronizing with disk or COW. > > > > > > If the hardware support SVM/SVA (Shared Virtual Memory/Address), it is easy, the > > > device share page table with CPU, the device will raise a page fault when the > > > CPU downgrade the PTE to read-only. > > > > > > If the hardware cannot share page table with the CPU, we then need to have > > > some way to change the device page table. This is what happen in ODP. It > > > invalidates the page table in device upon mmu_notifier call back. But this cannot > > > solve the COW problem: if the user process A share a page P with device, and A > > > forks a new process B, and it continue to write to the page. By COW, the > > > process B will keep the page P, while A will get a new page P'. But you have > > > no way to let the device know it should use P' rather than P. > > I didn't hear about such issue and we supported fork for a long time. > Just to comment on this, any infiniband driver which use umem and do not have ODP (here ODP for me means listening to mmu notifier so all infiniband driver except mlx5) will be affected by same issue AFAICT. AFAICT there is no special thing happening after fork() inside any of those driver. So if parent create a umem mr before fork() and program hardware with it then after fork() the parent might start using new page for the umem range while the old memory is use by the child. The reverse is also true (parent using old memory and child new memory) bottom line you can not predict which memory the child or the parent will use for the range after fork(). So no matter what you consider the child or the parent, what the hw will use for the mr is unlikely to match what the CPU use for the same virtual address. In other word: Before fork: CPU parent: virtual addr ptr1 -> physical address = 0xCAFE HARDWARE: virtual addr ptr1 -> physical address = 0xCAFE Case 1: CPU parent: virtual addr ptr1 -> physical address = 0xCAFE CPU child: virtual addr ptr1 -> physical address = 0xDEAD HARDWARE: virtual addr ptr1 -> physical address = 0xCAFE Case 2: CPU parent: virtual addr ptr1 -> physical address = 0xBEEF CPU child: virtual addr ptr1 -> physical address = 0xCAFE HARDWARE: virtual addr ptr1 -> physical address = 0xCAFE This apply for every single page and is not predictable. This only apply to private memory (mmap() with MAP_PRIVATE) I am not familiar enough with RDMA user space API contract to know if this is an issue or not. Note that this can not be fix, no one should have done umem without ODP like mlx5. For this to work properly you need sane hardware like mlx5. Cheers, Jérôme