From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.4 required=3.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_HELO_NONE, SPF_PASS,USER_AGENT_SANE_1 autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0D4E8C3F2D3 for ; Fri, 28 Feb 2020 14:48:48 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id AEDA02469F for ; Fri, 28 Feb 2020 14:48:47 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=ziepe.ca header.i=@ziepe.ca header.b="CV3X+nQK" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org AEDA02469F Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=ziepe.ca Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id F306E6B0005; Fri, 28 Feb 2020 09:48:46 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id EE1236B0006; Fri, 28 Feb 2020 09:48:46 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id DCFCA6B0007; Fri, 28 Feb 2020 09:48:46 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0156.hostedemail.com [216.40.44.156]) by kanga.kvack.org (Postfix) with ESMTP id C44E46B0005 for ; Fri, 28 Feb 2020 09:48:46 -0500 (EST) Received: from smtpin19.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id 61577181AEF1F for ; Fri, 28 Feb 2020 14:48:46 +0000 (UTC) X-FDA: 76539817452.19.beast94_2f70b7d4f8338 X-HE-Tag: beast94_2f70b7d4f8338 X-Filterd-Recvd-Size: 5913 Received: from mail-qt1-f195.google.com (mail-qt1-f195.google.com [209.85.160.195]) by imf10.hostedemail.com (Postfix) with ESMTP for ; Fri, 28 Feb 2020 14:48:45 +0000 (UTC) Received: by mail-qt1-f195.google.com with SMTP id x8so24878qts.0 for ; Fri, 28 Feb 2020 06:48:45 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ziepe.ca; s=google; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to:user-agent; bh=rsM6P0iWQtYbTIZfDH4GZGs0dcL9Uy2VcIbqhq4DVAs=; b=CV3X+nQKX4SjkWXOCXtPcfUWPbWcnGZIr5HXT0td8jCm8UnHLWOJGbCDin+fnDMVh6 HJjdT1BDInHrt1jgg7+HKPKmxJ+UWR8YMidtypOgu4QTRFY9Wu0jLvzuIF2MK/mIyaDX U6540tjncMTP9HxIc2pcrk8RD8kEm4WNC+qgrATMR0y1E2OgtHFcKDmfecdvihyCtKzE 5z/LqZfYlvAK7Ftc6n4Me1dMh0Y1wkVX2SLDydwWUd9444Zrts13ZQcXeFvt1tJDmWyx pVirruhQetio5UlZFpFM8fniSZUI6ZOWFVd0j2FrnRrxsI3M0zlRlNvFarECiOVYnx29 GpFw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to:user-agent; bh=rsM6P0iWQtYbTIZfDH4GZGs0dcL9Uy2VcIbqhq4DVAs=; b=hEqjj0GHTz/mI9r6l9QrsIYnyTnkVeO9YnVHFWRio0shGBPCyQDf3yDgv5kMKf1ro7 ci9Jy8wCDJiKdLqiFjbHAi8fTteN7hrFCF3xUKq7gfhC9zlvPv5yDLWQ6dB/NAZgdXQ+ X0eavR87ejRp8RAUsnInz4tu3Dl5GizeOnFnpZLyPfCl88KIbBtnv3qlJ6zB5XZMsWAF /8wF8BK73dq3rCBun8eFWoOXot0DifibvtI6zz7NhBv+GOXIGt7yRfV7ACggEgWtS9tH /gfarylgvozvQ7MbfWWnpLuZqbxL3rFVijszG/ppzqkBLDmPRSHHmLfhLRmipOCad6Ub QA7A== X-Gm-Message-State: APjAAAVnMkoK/Ys/GTLnftuXPETDS9bDJc3TYFP4I/JB2wilT6qOgbBV GpeC88vImnegDQTrKUYY3vFGfg== X-Google-Smtp-Source: APXvYqx4if0FgrogA1eRtXiNBtPg1pCTbepMjiVN00txoqKmRdlRgAbN74CUZqB71Vhu4mXSAdk1Rg== X-Received: by 2002:ac8:425a:: with SMTP id r26mr4520637qtm.138.1582901325295; Fri, 28 Feb 2020 06:48:45 -0800 (PST) Received: from ziepe.ca (hlfxns017vw-142-68-57-212.dhcp-dynamic.fibreop.ns.bellaliant.net. [142.68.57.212]) by smtp.gmail.com with ESMTPSA id t29sm5422548qtt.20.2020.02.28.06.48.44 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Fri, 28 Feb 2020 06:48:44 -0800 (PST) Received: from jgg by mlx.ziepe.ca with local (Exim 4.90_1) (envelope-from ) id 1j7gwW-0002z7-9g; Fri, 28 Feb 2020 10:48:44 -0400 Date: Fri, 28 Feb 2020 10:48:44 -0400 From: Jason Gunthorpe To: Jean-Philippe Brucker Cc: mark.rutland@arm.com, linux-pci@vger.kernel.org, linux-mm@kvack.org, will@kernel.org, Dimitri Sivanich , catalin.marinas@arm.com, zhangfei.gao@linaro.org, devicetree@vger.kernel.org, kevin.tian@intel.com, Arnd Bergmann , robh+dt@kernel.org, linux-arm-kernel@lists.infradead.org, Greg Kroah-Hartman , iommu@lists.linux-foundation.org, Andrew Morton , robin.murphy@arm.com, christian.koenig@amd.com Subject: Re: [PATCH v4 01/26] mm/mmu_notifiers: pass private data down to alloc_notifier() Message-ID: <20200228144844.GQ31668@ziepe.ca> References: <20200224182401.353359-1-jean-philippe@linaro.org> <20200224182401.353359-2-jean-philippe@linaro.org> <20200224190056.GT31668@ziepe.ca> <20200225092439.GB375953@myrica> <20200225140814.GW31668@ziepe.ca> <20200228143935.GA2156@myrica> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20200228143935.GA2156@myrica> User-Agent: Mutt/1.9.4 (2018-02-28) X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Fri, Feb 28, 2020 at 03:39:35PM +0100, Jean-Philippe Brucker wrote: > > > + list_for_each_entry_rcu(bond, &io_mm->devices, mm_head) { > > > + /* > > > + * To ensure that we observe the initialization of io_mm fields > > > + * by io_mm_finalize() before the registration of this bond to > > > + * the list by io_mm_attach(), introduce an address dependency > > > + * between bond and io_mm. It pairs with the smp_store_release() > > > + * from list_add_rcu(). > > > + */ > > > + io_mm = rcu_dereference(bond->io_mm); > > > > A rcu_dereference isn't need here, just a normal derference is fine. > > bond->io_mm is annotated with __rcu (for iommu_sva_get_pasid_generic(), > which does bond->io_mm under rcu_read_lock()) I'm surprised the bond->io_mm can change over the lifetime of the bond memory.. > > > If io_mm->ctx and io_mm->ops are already valid before the > > > mmu notifier is published, then we don't need that stuff. > > > > So, this trickyness with RCU is not a bad reason to introduce the priv > > scheme, maybe explain it in the commit message? > > Ok, I've added this to the commit message: > > The IOMMU SVA module, which attaches an mm to multiple devices, > exemplifies this situation. In essence it does: > > mmu_notifier_get() > alloc_notifier() > A = kzalloc() > /* MMU notifier is published */ > A->ctx = ctx; // (1) > device->A = A; > list_add_rcu(device, A->devices); // (2) > > The invalidate notifier, which may start running before A is fully > initialized at (1), does the following: > > io_mm_invalidate(A) > list_for_each_entry_rcu(device, A->devices) > A = device->A; // (3) I would drop the work around from the decription, it is enough to say that the line below needs to observe (1) after (2) and this is trivially achieved by moving (1) to before publishing the notifier so the core MM locking can be used. Regards, Jason