From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-4.1 required=3.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,SPF_PASS,USER_AGENT_MUTT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4CEEEC43610 for ; Mon, 19 Nov 2018 19:02:40 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 143CF20870 for ; Mon, 19 Nov 2018 19:02:40 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=kernel.org header.i=@kernel.org header.b="jYzMlTgg" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 143CF20870 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=kernel.org Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727692AbeKTF1c (ORCPT ); Tue, 20 Nov 2018 00:27:32 -0500 Received: from mail.kernel.org ([198.145.29.99]:35836 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1725969AbeKTF1b (ORCPT ); Tue, 20 Nov 2018 00:27:31 -0500 Received: from localhost (unknown [77.138.135.184]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 3113020851; Mon, 19 Nov 2018 19:02:34 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1542654155; bh=zWgFKLjronZiVWnLDTeZ27i548bHz3nIsobsoM0p3rY=; h=Date:From:To:Cc:Subject:References:In-Reply-To:From; b=jYzMlTgg/b4LFGR8t6hTyF1MQYhowTNjqzYnwmrovRBu+gf3dWHbNKHqtX0PL5GOu oKkSStZJCRJx65leTp9WJE9e+qOm2lh9rpi84nqGAYvMBqsSudBElMGD2owZr8AXZc Lf3J7FAE658veK4Ilr9hBXWR40dXR4Z3rNVqYnTk= Date: Mon, 19 Nov 2018 21:02:29 +0200 From: Leon Romanovsky To: Jerome Glisse Cc: Jason Gunthorpe , Kenneth Lee , Tim Sell , linux-doc@vger.kernel.org, Alexander Shishkin , Zaibo Xu , zhangfei.gao@foxmail.com, linuxarm@huawei.com, haojian.zhuang@linaro.org, Christoph Lameter , Hao Fang , Gavin Schenk , RDMA mailing list , Zhou Wang , Doug Ledford , Uwe =?iso-8859-1?Q?Kleine-K=F6nig?= , David Kershner , Kenneth Lee , Johan Hovold , Cyrille Pitchen , Sagar Dharia , Jens Axboe , guodong.xu@linaro.org, linux-netdev , Randy Dunlap , linux-kernel@vger.kernel.org, Vinod Koul , linux-crypto@vger.kernel.org, Philippe Ombredanne , Sanyog Kale , "David S. Miller" , linux-accelerators@lists.ozlabs.org Subject: Re: [RFCv3 PATCH 1/6] uacce: Add documents for WarpDrive/uacce Message-ID: <20181119190229.GB21715@mtr-leonro.mtl.com> References: <95310df4-b32c-42f0-c750-3ad5eb89b3dd@gmail.com> <20181114160017.GI3759@mtr-leonro.mtl.com> <20181115085109.GD157308@Turing-Arch-b> <20181115145455.GN3759@mtr-leonro.mtl.com> <20181119091405.GE157308@Turing-Arch-b> <20181119091910.GF157308@Turing-Arch-b> <20181119104801.GF8268@mtr-leonro.mtl.com> <20181119164853.GA4593@redhat.com> <20181119182752.GA4890@ziepe.ca> <20181119184215.GB4593@redhat.com> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="ZGiS0Q5IWpPtfppv" Content-Disposition: inline In-Reply-To: <20181119184215.GB4593@redhat.com> User-Agent: Mutt/1.10.1 (2018-07-13) Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org --ZGiS0Q5IWpPtfppv Content-Type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Mon, Nov 19, 2018 at 01:42:16PM -0500, Jerome Glisse wrote: > On Mon, Nov 19, 2018 at 11:27:52AM -0700, Jason Gunthorpe wrote: > > On Mon, Nov 19, 2018 at 11:48:54AM -0500, Jerome Glisse wrote: > > > > > Just to comment on this, any infiniband driver which use umem and do > > > not have ODP (here ODP for me means listening to mmu notifier so all > > > infiniband driver except mlx5) will be affected by same issue AFAICT. > > > > > > AFAICT there is no special thing happening after fork() inside any of > > > those driver. So if parent create a umem mr before fork() and program > > > hardware with it then after fork() the parent might start using new > > > page for the umem range while the old memory is use by the child. The > > > reverse is also true (parent using old memory and child new memory) > > > bottom line you can not predict which memory the child or the parent > > > will use for the range after fork(). > > > > > > So no matter what you consider the child or the parent, what the hw > > > will use for the mr is unlikely to match what the CPU use for the > > > same virtual address. In other word: > > > > > > Before fork: > > > CPU parent: virtual addr ptr1 -> physical address =3D 0xCAFE > > > HARDWARE: virtual addr ptr1 -> physical address =3D 0xCAFE > > > > > > Case 1: > > > CPU parent: virtual addr ptr1 -> physical address =3D 0xCAFE > > > CPU child: virtual addr ptr1 -> physical address =3D 0xDEAD > > > HARDWARE: virtual addr ptr1 -> physical address =3D 0xCAFE > > > > > > Case 2: > > > CPU parent: virtual addr ptr1 -> physical address =3D 0xBEEF > > > CPU child: virtual addr ptr1 -> physical address =3D 0xCAFE > > > HARDWARE: virtual addr ptr1 -> physical address =3D 0xCAFE > > > > IIRC this is solved in IB by automatically calling > > madvise(MADV_DONTFORK) before creating the MR. > > > > MADV_DONTFORK > > .. This is useful to prevent copy-on-write semantics from changing the > > physical location of a page if the parent writes to it after a > > fork(2) .. > > This would work around the issue but this is not transparent ie > range marked with DONTFORK no longer behave as expected from the > application point of view. > > Also it relies on userspace doing the right thing (which is not > something i usualy trust :)). The good thing that we didn't see anyone who succeeded to run IB stack without our user space, which does right thing under the hood :). > > Cheers, > J=E9r=F4me --ZGiS0Q5IWpPtfppv Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQIcBAEBAgAGBQJb8wjFAAoJEORje4g2clinsgMP/06W7PP/epn/WsbihGFnNzR7 eSCxWl6pe1bRlT0k3QIUENSn61M04nryTJTErkpGC/Qgk+hDxQMMGk4Rr1sE7PUA Ht9AYxBf9+aijIZM8ttCcandwz+RZ8TjtUH+ejXd1XDc1kbkJFgL+9H8TNbfJu+o 521rbhMoebH+dJnTtGfts82ZrY+/rfyALkXLnRwArh8Hkz7AqTNQEKI/sqaM8jhc dPqtU+xlIbwAejMNKhFKQswXRJfzO7MBSXocRf4wSX5L/U8A+p+s2QmknJTokiiG 061EB/clVZ1u09wuhEBT4IbMQmle2CkRg3m3NtTrTJg3Or1wP6G9lt8NSA73nS6/ o6OAdsPiool8Hnv0h9wGSgaRLuXTc/TUG2xVbBQWPyRJFXqy7aeXNOW1fS5SFIAy kj9PSxrMmpp+gYyEx9DBbpX6oPtJmsv4dvbKmpUQWsqbCIxdtwtKW00gAPTvYbHP jz7mEfGCHDRRIPcEcXxmeASta6BIzJRZR94xviAeCPOst5PuS4QUr098hv46Syn9 gXRYzkcaIIhCjsPhWfsy/mrs+5k+NFPIy7jH4Zb5JE020Qu+Mir8PoY4ml25PiJy 0SAiHLu0vnYtgbxgpHPXPqMqPxTqZAh8kGE+utcGVZeS6VrxBSysKe5fxSlehifS iH8hKkPnI7Z6xPkex8QN =EMEz -----END PGP SIGNATURE----- --ZGiS0Q5IWpPtfppv--